Neural network processor

Abstract

In one example, a neural network processor includes a memory interface (534), an instruction buffer (520), a weight buffer (526), an input data register (528a), a weight register (528b), an output data register (528a), a compute engine (524), and a controller (522). The controller is configured to: receive a first instruction from the instruction buffer; in response to the first instruction, an input data element is fetched from the memory interface to the input data register and a weight element is fetched from the weight buffer to the weight register. The controller is further configured to: receive a second instruction from the instruction buffer; and in response to the second instruction: retrieve the input data element and the weight element from the input data register and the weight register, respectively, to the compute engine; and performing a compute operation between the input data element and the weight element using the compute engine to generate an output data element.

Description

Background Art

[0001] An artificial neural network is a computing system whose architecture is based on biological neural networks. Artificial neural networks can be trained using training data through a training process to learn how to perform specific computing tasks. Artificial neural networks can be implemented on a neural network processor, which can include memory and computing resources to support the computational operations of the artificial neural network. Certain applications may impose limitations on the amount of memory and computing resources available on a neural network hardware accelerator. Summary of the invention

[0002] In one example, a neural network processor is provided. The neural network processor includes a memory interface, an instruction buffer, a weight buffer, an input data register, a weight register, an output data register, a computing engine, and a controller. The controller is configured to receive a first instruction from the instruction buffer, and in response to the first instruction: obtain an input data element from the memory interface to the input data register, and obtain a weight element from the weight buffer to the weight register. The controller is also configured to receive a second instruction from the instruction buffer, and in response to the second instruction: obtain the input data element and the weight element from the input data register and the weight register to the computing engine, respectively, perform a computing operation between the input data element and the weight element using the computing engine to generate an output data element, and store the output data element at the output data register.

[0003] In one example, a method is provided. The method includes: receiving a first instruction from an instruction buffer of a neural network processor, and in response to the first instruction: acquiring input data elements from a memory external to the neural network processor to input data registers of the neural network processor; and acquiring weight elements from a weight buffer of the neural network processor to weight registers of the neural network processor. The method further includes: receiving a second instruction from the instruction buffer, and in response to the second instruction: acquiring the input data elements and the weight elements from the input data registers and the weight registers to a computing engine of the neural network processor, respectively, using the computing engine to perform computing operations between the input data elements and the weight elements to generate output data elements, and storing the output data elements at output data registers of the neural network processor.

[0004] In one example, a neural network processor includes a memory interface, an instruction buffer, a weight buffer, an input data register, a weight register, an output data register, an address register, an address generation engine, a data load / store engine, a calculation engine, and a controller. The address generation engine may be configured to set the input data address, the output data address, and the weight address in the address register. The data load / store engine may be configured to: based on the input data address, obtain input data from the memory interface to the input data register; based on the output data address, obtain output data from the output data register to the memory interface; and based on the weight address, obtain weights from the weight buffer to the weight register. The calculation engine may be configured to perform calculations based on the input data and the weights to generate the output data. The controller is configured to, in response to one or more instructions from the instruction buffer: extract a first sub-instruction for the address generation engine to set one of the input data address or the output data address, extract a second sub-instruction for the address generation engine to set the weight address, extract a third sub-instruction for the calculation engine to perform the calculation, extract a fourth sub-instruction for the data load / store engine to obtain the weight at the weight address of the weight buffer, extract a fifth sub-instruction for the data load / store engine to obtain the input data at the input data address via the memory interface, and in parallel configure the address generation with the first and second sub-instructions, configure the calculation engine with the third sub-instruction, and configure the data load / store engine with the fourth and fifth instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 is a schematic diagram illustrating a system in which inference operations may be performed, according to some examples.

[0006] Figure 2 , Figure 3 and Figure 4 is shown according to some examples by Figure 1 Schematic diagrams and charts of examples of data processing operations performed by electronic devices of the system.

[0007] Figure 5 is a schematic diagram illustrating a neural network processor according to some examples.

[0008] Fig. 6A , 6B and 6C are examples of Figure 5 A diagram of examples of instructions executed by a neural network processor.

[0009] Figure 7 Including according to some examples of the illustration can be made Figure 5A diagram of a set of instructions executed by a neural network processor.

[0010] Fig. 8A , 8B and 9 are shown according to some examples by Figure 5 Diagram of the memory management scheme provided by the neural network processor.

[0011] Fig. 10A According to some examples Figure 5 Schematic diagram of the loop buffer circuit of a neural network processor.

[0012] Fig. 10B According to some examples Fig. 10A A diagram of the operation of the circular buffer circuit.

[0013] Fig.11 and Fig.12 According to some examples Fig. 6A and 6B The flow control element of the instruction syntax.

[0014] Fig.13 According to some examples Figure 5 A schematic diagram of the internal components of the neural network processor's computational engine.

[0015] Fig.14 Contains some examples of Fig.13 Diagram of the convolution operation performed by the compute engine.

[0016] Fig.15A , 15B , 15C and 15D include diagrams according to some examples. Fig.13 A diagram of the post-processing operations performed by the computation engine for different weight precisions.

[0017] Fig.16 is shown according to some examples by Fig.13 Schematic diagram of the arithmetic operations performed by the calculation engine.

[0018] Fig.17 According to some examples Fig.13 Schematic diagram of the multiplier circuit of the computation engine.

[0019] Fig.18 According to some examples Fig.13 Schematic diagram of the computing unit of the computing engine.

[0020] Fig.19 According to some examples Fig.18 Schematic diagram of the accumulator of the computational unit.

[0021] Fig. 20A and 20B According to some examples Fig.13 The calculation engine includes Fig.18 Schematic diagram of the internal components of the computational unit.

[0022] Figure 21A-1 , 21A-2 , 21B-1, 21B-2, 21C-1, 21C-2, 21D-1, 21D-2, 21E-1, 21E-2, 21F-1, 21F-2, 21G-1, 21G-2, 21H-1 and 21H-2 are diagrams according to some examples. Fig.13 Schematic diagram of different configurations of the computation engine for different input and weight precisions.

[0023] Fig.22A , 22B and 22C are shown according to some examples Fig.13 Schematic diagram of the internal multiplexer of the computation engine.

[0024] Fig.23 According to some examples Fig.13 Schematic diagram of the internal components of the post-processing engine.

[0025] Fig.24 According to some examples Fig.13 Schematic diagram of the internal components of the post-processing engine.

[0026] Fig.25A , 25B and 25C are shown according to some examples Fig.13 Schematic diagram of the configuration of the post-processing engine.

[0027] Figure 26A-1 , 26A-2 and 26B are diagrams showing some examples of Fig.13 A diagram and chart of the internal components of the max pooling engine and its operation.

[0028] Fig. 27 , 28 , 29A and 29B are flow charts illustrating the operation of a neural network processor according to some examples.

[0029] The same reference numbers are used throughout the drawings to indicate the same (or similar) features. DETAILED DESCRIPTION

[0030] Figure 11 is a schematic diagram showing a system 100. The system 100 may include a plurality of electronic devices 102, including electronic devices 102a, 102b, and 102c, and a cloud network 103. Each electronic device 102 may include a sensor 104 and a data processor 106. For example, the electronic device 102a includes a sensor 104a and a data processor 106a, the electronic device 102b includes a sensor 104b and a data processor 106b, and the electronic device 102c includes a sensor 104c and a data processor 106c. The sensor 104 may be of various types, such as an audio / acoustic sensor, a motion sensor, an image sensor, and the like. In some examples, each electronic device 102 may include a plurality of different types of sensors (e.g., an acoustic sensor and a motion sensor), or multiple instances of the same type of sensor (e.g., multiple microphones). Each sensor system may receive a stimulus 108 (e.g., an acoustic signal, a light signal, a motion, and the like), and generate a decision 110 based on the received stimulus. For example, the decision may indicate whether an event of interest is detected. For example, electronic device 102a may generate decision 110a based on stimulus 108a, electronic device 102b may generate decision 110b based on stimulus 108b, and electronic device 102c may generate decision 110c based on stimulus 108c. In some examples, each electronic device 102 may be an Internet of Things (IoT) end node and may be an edge device of network 103. Each electronic device 102 may transmit a corresponding decision 110 to cloud network 103, which may then perform an action based on the decision (e.g., transmit an alert about an unusual event, contact law enforcement, etc.).

[0031] The data processor 106 of a specific electronic device 102 may perform data processing operations on the data collected by the sensor 104 on the specific electronic device to generate a decision 110. For example, in an example where the sensor 104 includes an audio / acoustic sensor, the data processor 106 may perform data processing operations such as keyword location, voice activity detection, and detection of specific acoustic features (e.g., glass breakage, gunshots). In addition, in an example where the sensor 104 includes a motion sensor, the data processor 106 may perform data processing operations such as vibration detection, activity recognition, and anomaly detection (e.g., when no one is home or at night when the window / door is hit or opened). In addition, in an example where the sensor 104 includes an image sensor, the data processor 106 may perform data processing operations such as face recognition, gesture recognition, and visual wake-up word detection (e.g., determining whether there is a person in the environment). The data processor 106 may also generate and output a decision 110 based on the results of the data processing operations, including, for example, keyword detection in voice, specific acoustic features, specific activities, detection of specific gestures, and the like. In situations where the electronic device 102 includes multiple sensors, the data processor 106 may perform sensor fusion operations on the different types of sensor data to generate the decision 110 .

[0032] The data processor 106 may include various circuit systems for processing sensor signals generated by the sensor 104. For example, the data processor 106 may include a sample-and-hold (S / H) circuit for sampling the sensor signal. The data processor 106 may also include an analog-to-digital converter (ADC) for quantizing the sample into a digital signal. The data processor 106 may also include a neural network processor for implementing an artificial neural network to process the sample. An artificial neural network (hereinafter referred to as a "neural network") may include a plurality of processing nodes. The neural network may perform an inference operation or a classification operation on the sensor data to generate the aforementioned decision. The inference operation may be performed by combining the sensor data with a set of weight elements obtained from the neural network training operation to generate a decision. Examples of neural networks may include deep neural networks (DNNs), convolutional neural networks (CNNs), and the like.

[0033] The processing nodes of the neural network can be divided into layers, including, for example, an input layer, several intermediate layers (e.g., hidden layers), and an output layer. The input layer and the intermediate layer can each be a convolutional layer forming a CNN, and the output layer can be a fully connected layer, and the input layer, the intermediate layer, and the output layer together form a DNN. Each processing node of the input layer receives an element of an input set, and scales the element with a weight element to indicate the degree of influence of the element on the output. The input set may include, for example, acoustic data, motion data, image data, a combination of different types of data, or a set of input features extracted from those data. In addition, the processing nodes in the intermediate layer can combine the scaled elements received from each processing node of the input layer to calculate a set of intermediate outputs. For example, each processing node in the intermediate layer can calculate the sum of the element-weight product, and then generate the intermediate output by applying an activation function to the sum. The intermediate output of each processing node from an intermediate layer can be regarded as an activated vote (or no vote) associated with a weight indicating the voting influence to determine the intermediate output of the next intermediate layer. The intermediate output can represent the output feature of a specific intermediate layer. The output layer can generate the sum of the scaled intermediate outputs from the final intermediate layer. In some examples, the output layer can generate a binary output (e.g., "yes" or "no") based on whether the sum of the scaled intermediate outputs exceeds a threshold, which can indicate a decision for a data processing operation (e.g., keyword detection in speech, detection of specific acoustic features, specific activities, specific gestures).

[0034] The neural network processor of the data processor 106 can be programmed to perform calculations based on an artificial neural network model. The neural network processor can be programmed based on a sequence of instructions that include computational operations associated with the model (e.g., addition, multiplication, processing of activation functions, etc.). The instructions can also access internal and external memory devices to obtain and store data. The compiler can receive information about the neural network model, input data, and available memory and computing resources, and generate the set of instructions to indicate, for example, when to access internal and external memory devices to obtain data, which component of the neural network processor performs calculations on data based on the neural network model, etc., thereby performing neural network processing. Figure 1 In some examples, the data processor 106 of each electronic device 102 may receive data 120 (e.g., 120a, 120b, and 120c) from the cloud network 103, including, for example, a set of weight elements, compiled instructions representing each layer of the neural network model, etc. In some examples, the neural network processor of each electronic device 102 may perform a training operation to adjust the initial weight elements. In some examples, the training operation may also be performed at the cloud network 103, and each electronic device 102 may receive an updated weight element from the cloud network 103 and perform an inference / classification operation using the updated weights.

[0035] Figure 2 , Figure 3 and Figure 4 is a schematic diagram illustrating an example of data processing operations performed by the electronic device 102 when processing an acoustic signal. Figure 2 Example data processing operations 200 for recognizing words in speech are shown. The example data processing operations include feature extraction operations 202, DNN layer processing operations 204, and post-processing operations 206. As part of feature extraction operations 202, data processor 106 may receive samples of acoustic signals 212 from sensor 104 and extract features 214 from the samples. The acoustic signal may represent a speech signal having an amplitude / power level that varies over time. In addition, features 214 may include distinguishing characteristics of the acoustic signal from which subsequent processing operations (e.g., DNN layer processing operations 204 and post-processing operations 206) may recognize words in speech. Figure 2 In the example of , the feature 214 may include a time-frequency distribution of the signal power. In some examples, the feature extraction operation 202 may include a convolution operation, which may be performed by a neural network processor implementing a CNN in the data processor 106, a general-purpose processor (e.g., a microcontroller), a computing in memory (CIM) circuit, etc.

[0036] In addition, as part of the DNN layer processing operation 204, the data processor 106 can process the features 214 using a multi-layer DNN and a set of weight elements 216 and calculate a set of outputs 218. The post-processing operation 206 can post-process and quantize the outputs 218 to generate inference outputs 220. The post-processing operation may include, for example, activation function processing for mapping the outputs 218 to the set of inference outputs, and other post-processing operations, such as batch normalization (batch norm or BNorm) and residual layer processing for facilitating convergence in training. The inference output 220 may include a set of probabilities for a set of candidate words. Figure 2 In the example of , the inference output 220 can map the candidate word "yes" with a probability of 0.91, map "no" with a probability of 0.02, and map other words with a probability of 0.01. In some examples, as part of the post-processing operation 206, the decision that the speech signal 212 includes the word "yes" can also be made based on the candidate word "yes" being mapped to the highest probability among the candidate words. In some examples, such as Figure 2 As shown, the output of the post-processing operation 206 can represent the output of the neural network layer, which can be fed back into the DNN layer processing operation 204 as the input of another neural network layer.

[0037] Figure 3 It can be shown that Figure 2 An example of an example processing operation 300 of a portion of a DNN layer processing operation 204. Figure 3, processing operation 300 may be a convolution operation performed by a neural network layer on a set of input data elements 302 to generate a set of output data elements 304. Input data elements 302 and output data elements 304 may each be in the form of a multi-dimensional array, including a first dimension (input height), a second dimension (input width), and a third dimension (input channels). The array of input data elements 302 may have a height F h 、Width F w , and has N in Input channels. For example, input data element 302 can represent feature 214, where the first dimension can represent time, the second dimension can represent frequency, and the third dimension can represent a data channel (or channels). A channel can represent a data source. For example, in the case where input data element 302 is audio data generated by multiple microphones, data generated from a particular microphone can be associated with a particular data channel. As another example, in the case where input data element 302 is image data generated by an image sensor of different color channels (e.g., red, green, and blue), data generated for a particular color channel can be associated with a particular data channel.

[0038] To perform a convolution operation to compute an output data element (e.g., output data element 304a), data processor 106 may compute a dot product between a set of weight elements 306 and a subset of input data elements 302. Figure 3 In the example of h , weight width K w And also has N in The set of weight elements 306 may be moved relative to the set of input data elements 302 by a stride D to overlap different subsets of the input data elements 302. Different dot products (and different output data elements 304) may be calculated between the set of weight elements 306 and different overlapping subsets of the input data elements 302 at different strides, as follows:

[0039]

[0040] In Equation 1, r represents the index along the width dimension, s represents the index along the height dimension, and c represents the index along the input channel dimension. In addition, X c eD+r,fD+s represents an input data element 302 of input channel c and has an index eD+r along the width dimension and fD+s along the height dimension, where e and f are multiples of the stride D, F c r,s represents a weight data element 306 having an index r along the width dimension and s along the height dimension, and Ye,f Denotes an output data element 304 having an index e along the width dimension and f along the height dimension. In some examples, the data processor 106 may perform a convolution operation between the input data element 302 and the plurality of sets of weight elements 306 to generate an output data element 304 having an index e along the width dimension and f along the height dimension. out The convolution operation may be performed on an input data element 304 and an output data element 304 for each output channel, wherein the convolution operation between the input data element and a set of weight elements 306 may generate the output data element 304 for a particular output channel.

[0041] Figure 4 400. In some examples, loop instructions 400 may represent a set of loop instructions 400 executed by a neural network layer. Figure 3 For example, the set of loop instructions 400 may represent a portion of a convolution operation at a particular stride D. The neural network processor of the data processor 106 may execute the set of loop instructions to calculate a convolution operation having N from the input data elements 302 and the plurality of sets of weight elements 306. out The loop instruction 400 may represent a nested loop structure, including scanning the output channels (n is between 0 and N). out -1) and input channels (m between 0 and N in -1) dimension, sweeping through the input height (i in the range of 0 to F h -1) and the input width (j is between 0 and F w -1) dimension, and sweeping through the weight filter / kernel height (h between 0 and K h -1) and weight width (w is between 0 and K w -1) dimension, and computes the output data element Y as the dot product between the input data element associated with the input height, input width, and input channel index range and the weight element associated with the weight height, weight width, input channel, and output channel index range.

[0042] The parameters of the set of loop instructions 400 can be configured based on the specific type / topology of the neural network layer to be represented by the set of loop instructions. For example, to implement a deep convolutional layer where each output data element is generated from the dot product of the input data element and the weight element of the same input channel, the outermost loops 402 and 404 can be combined into one loop, or the variable m can be set to a constant. In addition, to implement an average pooling layer, all weight elements provided to the neural network layer can be set to 1 / (K w *K h). In addition, to implement a point-wise convolutional layer, each output data element is derived from an input data element and the depth is equal to the number of input channels N in The dot product of the 1x1 kernel is generated, K w and K h (width and height of the weight element array) can be set to 1 respectively. In addition, to implement a fully connected layer, K w , Kh, F w and F h Can be set to 1 individually.

[0043] Reference again Figure 1 , having the electronic device 102 perform reasoning operations on the sensor data locally, rather than sending the sensor data to the cloud network 103 for reasoning, can bring a variety of advantages. For example, by retaining and processing the sensor data locally, the need to transmit the sensor data from the electronic device 102 to the cloud network 103 can be reduced or even completely eliminated. Such an arrangement can reduce the power consumption and bandwidth required to transmit sensor data, especially when the electronic device 102 continuously generates a large amount of sensor data (for example, as part of a continuous online system). In addition, the security and privacy risks associated with the transmission of sensor data (for example, images, voice data, etc.) can also be reduced. In addition, the speed and reliability of providing reasoning decisions can also be improved, or at least less affected by factors external to the electronic device 102, such as network bottlenecks in sensor data transmission, unavailability of the cloud network 103, etc.

[0044] Although it is advantageous to have the electronic device 102 perform reasoning operations on sensor data locally, there are various challenges. Specifically, reasoning operations, even if performed by dedicated hardware such as a neural network processor, can be power-intensive and can use a large amount of memory and computing resources. On the other hand, the electronic device 102 can be a low-power device and can also have a small form factor, especially if they are IoT devices. Therefore, the neural network processor on the electronic device 102 may have very limited memory and computing resources to perform reasoning operations. In addition, different applications may have different and conflicting requirements for reasoning operations. For example, some applications may require high precision for reasoning calculations, while other applications may not require such high precision. In addition, the neural network processor can support a variety of neural network topologies, layer types, kernel / filter sizes, and dimensions for filters, input data, and output data to support different applications. All of this poses a challenge for the neural network processor to perform various reasoning operations with limited memory and computing resources to support a wide range of application scenarios.

[0045] Figure 5is a schematic diagram illustrating an example of internal components of an electronic device 102 having a neural network processor 502 that can address at least some of the above challenges. Figure 5 , the electronic device 102 may include a neural network processor 502, which is coupled to a memory 512, a direct memory access (DMA) controller 516, a processor 514, and a sensor interface circuit 517 via an interconnect 518. The sensor interface circuit 517 may include an interface circuit system (e.g., an analog-to-digital converter (ADC)) that interfaces with the sensor 104. In some examples, the neural network processor 502, the memory 512, the DMA controller 516, the interconnect 518, and the sensor 104 may be part of an integrated circuit (IC) such as a system on a chip (SoC). In some examples, the neural network processor 502 may be a stand-alone application-specific integrated circuit (ASIC). The interconnect 518 may be a SoC bus interconnect or other type of interconnect. In addition, the processor 514 may be a general-purpose processor and include, for example, a central processing unit (CPU), one or more processor cores, etc., capable of executing instructions to perform various tasks.

[0046] The memory 512 is shared by the neural network processor 502 and the processor 514. The memory 512 may store instructions, input data and weights for each neural network layer to be provided to the neural network processor 502 to perform reasoning operations, and output data generated by the neural network processor 502 from reasoning operations. The input data may include, for example, sensor data provided by the sensor 104, feature data extracted from the sensor data, or intermediate output data generated by a previous neural network layer. The memory 512 may also store other data, such as program instructions and data for the processor 514. The memory 512 may include any suitable on-chip memory, such as a flash memory device, a static random access memory (SRAM), a resistive random access memory (ReRAM), etc.

[0047] By having the processor 514 and the neural network processor 502 share the memory 512, rather than providing dedicated memory devices for the neural network processor 502 and the processor 514 separately, the total memory size of the electronic device 102 can be reduced, which can reduce the power and footprint of the electronic device 102. As will be described in more detail below, the neural network processor 502 can implement various memory management schemes, such as in-place computing, where output data overwrites input data using circular addressing of output data and circular addressing of input data to support continuous online applications, thereby reducing memory resource usage and memory data movement, which allows the neural network processor 502 to operate with limited memory resources. In addition, the neural network processor 502 is configured to handle variable latency in accessing data from the memory 512, thereby allowing the processors 514 to operate in parallel with minimal impact on the performance of the neural network processor 502.

[0048] The processor 514 may execute a software program that uses the neural network processor 502 to perform reasoning operations and then perform additional operations based on the results of the reasoning operations. For example, the processor 514 may execute a software program for a home security system. The software program may include instructions (e.g., an application programming interface (API)) for the neural network processor 502. One API may be associated with a calculation at one neural network layer, and the software program may include multiple APIs for multiple neural network layers. After calling / executing the API, the processor 514 may provide the neural network processor 502 with the memory address of the neural network layer instruction that can be executed by the neural network processor 502, as well as the weights, parameters, and input data, which the neural network processor may then obtain at the memory address of the memory 512. The processor 514 may also transmit a control signal to the neural network processor 502 to start the calculation of the neural network layer. After the neural network layer calculation is completed and the output data is stored in the memory 512, the neural network processor 502 may transmit the memory address of the output data and the control signal back to the processor 514 to signal completion, and the processor 514 may call another API for the next neural network layer. The processor 514 may also execute software programs to perform other functions, such as transmitting inference decisions to the cloud network 103 (or other device / system), providing a graphical user interface, etc. The processor 514 may perform those other functions in parallel with the neural network processor 502 .

[0049] The DMA controller 516 may be configured to perform DMA operations to transfer certain data between the memory 512 and the neural network processor 502. For example, after calling the API, the processor 514 may provide the neural network processor 502 with a memory address (e.g., in the form of a memory descriptor) for the stored instructions, weights, and parameters. The neural network processor 502 may then obtain the stored instructions, weights, and parameters based on the memory address provided by the processor 514. As will be described below, in some examples, the neural network processor 502 may directly obtain input data from the memory 512 as needed, rather than using the DMA controller 516 to obtain input data in batches. In addition, the neural network processor 502 may store the newly generated output data directly to the memory 512 in relatively small chunks (e.g., a chunk size of 32 bits) when the output data is generated and reaches a chunk size, rather than using the DMA controller 516 to obtain the output data in batches. Such an arrangement may avoid having an input / output data buffer on the neural network processor 502 (or at least reduce its size), which may reduce the power and footprint of the neural network processor 502.

[0050] The neural network processor 502 may be a neural network hardware accelerator and may provide hardware resources including computing resources and memory resources for neural network layer calculations to support reasoning operations. The neural network processor 502 may include an instruction buffer 520, a computing controller 522, and a computing engine 524 with configurable computing precision. The neural network processor 502 also includes a weight and parameter buffer 526, a register 528, a load / store controller 530, and an address generator 532. The load / store controller 530 further includes a memory interface 534. Each component of the neural network processor 502 may include a combinational logic circuit (e.g., a logic gate), a sequential logic circuit (e.g., a latch, a flip-flop, etc.), and / or a memory device (e.g., SRAM) to support various operations of the neural network processor 502, as will be described below.

[0051] Instruction buffer 520 may retrieve and store computation instructions for a neural network layer from memory 512 in response to a control signal from processor 514. For example, in response to an API being called to start neural network layer computation, processor 514 may control instruction buffer 520 to transfer instructions (e.g., microcode) for neural network layer computation from memory 512, or receive instructions from other sources (e.g., another processor) and store the instructions at instruction buffer 520.

[0052] The compute controller 522 may decode each compute instruction stored in the instruction buffer 520 and control the compute engine 524, the load / store controller 530, and the address generator 532 to perform operations based on the instructions. For example, in response to an instruction to perform a convolution operation, the compute controller 522 may control the compute engine 524 to perform calculations for the convolution operation after the data and weights for the convolution operation are acquired and stored in the data register 528a and the weight / parameter register 528b, respectively. The compute controller 522 may maintain a program counter (PC) that tracks / points to the next instruction to be executed. In some examples, the compute instructions may include flow control elements, such as loop elements, macro elements, etc., that may be extracted by the compute controller 522. The compute controller 522 may then change the PC value in response to the flow control element and change the flow / order of instruction execution based on the flow control element. Such an arrangement allows a neural network layer instruction to include a loop instruction that reflects a convolution layer calculation, such as Figure 4 those shown, and can be made more compact, which can reduce microcode size and facilitate the conversion of neural network layer topology to instructions.

[0053] The computation engine 524 may include circuitry for performing convolution operations to support CNN network layer computations, and circuitry for performing post-processing operations (e.g., BNorm and residual layer processing) on ​​neural network output data. As will be discussed below, the computation engine 524 may be configured to perform MAC (e.g., convolution) and post-processing operations on weights and input data across a range of bit precisions (e.g., binary precision, ternary precision, 4-bit precision, 8-bit precision, etc.) based on parameters provided by the processor 514 and stored in configuration registers 528c. Parameters may be provided for specific neural network layers and may be updated between executions of different neural network layers. This allows the processor 514 to dynamically configure the bit precision of convolution and post-processing operations at the computation engine 524 based on, for example, the need for inference operations to be performed, the application using the inference results, the available power of the electronic device 102, etc. In some examples, the computation engine 524 allows weights and input data for different neural network layers to have different bit precisions, which may support various accuracy and computational trade-offs. This also allows the neural network processor 502 to be used as a domain-specific or dedicated instruction set processor, where the neural network processor 502 can be configured / customized to efficiently execute certain applications (e.g., deep neural network-based machine learning).

[0054] The weight and parameter buffer 526 can store weights for multiple neural network layers, as well as parameters for different internal components of the neural network processor 502 to support post-processing operations. The neural network processor 502 can obtain the weights and parameters from the memory 512 via (for example) the DMA controller 516. In some examples, the weight and parameter buffer 526 may include an SRAM device. Having the weight and parameter buffer 526 store weights and parameters that are static for a particular neural network layer can reduce the movement of such static data between the neural network processor 502 and the memory 512 during the calculation of the neural network layer. In addition, the size of the weight and parameter data can be relatively small compared to the size of the input and output data used for the neural network layer calculation, which allows the weight and parameter buffer 526 to have a smaller footprint.

[0055] In addition, registers 528 may include data registers 528a, weight and parameter registers 528b, address registers 528c, and configuration registers 528d. Data registers 528a may store a subset of input data and a subset of output data for calculation of a neural network layer, and weight and parameter registers 528b may store a subset of weights for neural network layer calculations and store parameters for post-processing operations. Address registers 528c may store memory addresses to be accessed by load / store controller 530 at memory 512 to obtain weights and input / output data. Address registers 528c may also store addresses to be accessed by load / store controller 530 in weight and parameter buffers 526 to obtain weights and parameters. Configuration registers 528d may store configuration parameters common to various components of neural network processor 502. For example, configuration registers 528d may store parameters to set the bit precision of input data elements and weight elements for convolution calculations and post-processing operations at compute engine 524, specific memory management schemes of neural network processor 502 (e.g., in-place calculations, circular addressing of input data, etc.), etc. As described above, some of these parameters may be provided / updated by processor 514 between executions of a neural network or between executions of two neural network layers.

[0056] The read / write operations of the data register 528a and the weight register 528b may be performed by the load / store controller 530 based on instructions executed by the compute controller 522. For example, in response to an instruction instructing to obtain input data to the data register 528a, the load / store controller 530 may directly obtain the input data from the memory 512 via the memory interface 534 (e.g., without passing through the DMA controller 516) and store the obtained data at the data register 528a. In addition, in response to an instruction instructing to store the output data back to the memory 512, the load / store controller 530 may obtain the output data from the data register 528a and store the output data at the memory 512 directly via the memory interface 534 (e.g., without passing through the DMA controller 516). As discussed above, such an arrangement may avoid having an input / output data buffer (or reduce its size) on the neural network processor 502, which may reduce the power and footprint of the neural network processor 502.

[0057] In addition, as will be described below, the load / store controller 530 can implement various memory management schemes by setting the memory addresses stored in the address register 528c, such as in-situ calculations in which output data overwrites input data, and circular addressing of input data to support continuous online applications. Such an arrangement can reduce the occupied area of ​​input / output data in the memory 512 and reduce the movement of input / output data in the memory 512, thereby facilitating shared access to the memory 512 between the neural network processor 502 and the processor 514. On the other hand, the load / store controller 530 can also obtain weights and parameters from the weight and parameter buffer 526 to the weight register 528b based on instructions executed by the calculation controller 522. As described above, the processor 514 can control the neural network processor 502 to access the weight and parameter buffer 526 via a separate memory interface (in Figure 5 ) and uses DMA controller 516 to obtain weights and parameters from memory 512 or from another processor and store the weights and parameters in weight and parameter buffer 526.

[0058] Fig. 6A , 6B 6C are diagrams showing examples of instructions that may be executed by the neural network processor 502. Fig. 6A and 6B , instruction 600 may include multiple sub-instructions, including sub-instructions 602, 604, 606, 608, 610, 612, and 614. Each sub-instruction may be directed to a specific component of the neural network processor 502, and different components may be programmed by sub-instructions and perform operations represented by sub-instructions in parallel and / or independently. For example, sub-instruction 602 is directed to the computation engine 524. Sub-instruction 604 is directed to a portion of the address generator 532 that generates memory addresses to access weights, input data, output data, and post-processing parameters such as bias, scaling, and shift. Sub-instruction 604 may also include a start loop indicator, and may also be directed to the computation controller 522. Sub-instruction 606 is directed to a portion of the address generator 532 that generates memory addresses to access input data and output data. In addition, sub-instruction 608 is directed to a load / store controller 530 for obtaining weights and post-processing parameters (e.g., bias, scaling, and shift) from the weight and parameter buffer 526 to the weight / parameter register 528b. In addition, sub-instruction 610 is directed to load / store controller 530 for fetching input data from memory 512 to data register 528a and storing output data from data register 528a at memory 512. In addition, sub-instruction 612 is directed to computation controller 522 and provides an end indicator of a loop instruction or macroinstruction.

[0059] In addition, sub-instruction 614 indicates the type of instruction 600. Neural network processor 502 may support instructions of different types and bit lengths. For example, in Fig. 6A and 6B In the example, instruction 600 is a 48-bit (48b) instruction with five sub-instructions (602, 604, 606, 608, and 610) for the compute engine 524, the address generator 532, and the load / store controller 530. The neural network processor 502 may also support 24-bit (24b) instructions with a subset of sub-instructions 602-610, as well as other sub-instructions. Based on the sub-instructions 614, the compute controller 522 may determine the type and bit length of the instruction, and decode and extract the sub-instructions from the instruction based on the type and bit length.

[0060] The computation controller 522 may extract the sub-instructions from the predetermined bit positions of the instruction and generate control signals for target components of the neural network processor 502 based on the extracted sub-instructions. The computation controller 522 may control the computation engine 524, the address generator 532, and the load / store controller 530 to execute the corresponding sub-instructions in parallel, thereby allowing the neural network processor 502 to provide N-way (e.g., 5-way when executing sub-instructions 602-610) parallel programmability.

[0061] Each of the sub-instructions 602-610 includes fields that identify operations to be performed by the target component and / or registers to be accessed. For example, sub-instruction 602 includes fields 602a, 602b, 602c, and 602d. Field 602a may identify a calculation to be performed by the computation engine 524, such as a multiply-accumulate (MAC) calculation operation, a BNorm calculation operation, or a maximum pooling calculation operation. Field 602b may identify a destination data register (labeled MACreg0-MACreg7) where the output of the calculation is stored. Fields 602c and 602d, respectively, identify source input data registers (in MACreg0-MACreg7) from which input data elements and weight elements are obtained for calculation. Fig. 6A and 6B Din0 and Din1) and the weight registers (in Fig. 6A and 6B As will be described below, in some examples, field 602c may also indicate whether a max pooling operation will be performed by the computation engine 524, or whether the max pooling operation will provide a zero output.

[0062] In addition, sub-instruction 604 includes fields 604a, 604b, and 604c. Field 604a may identify an operation to be performed to update the address stored in the source address register identified by field 604c, and the updated address is to be stored in the destination address register identified by field 604b. Operations may include, for example, increment by one (ADD), decrement by one (SUB), and move operations (MOV) to replace an address in a destination address register with an address in a source address register. The address may be a memory address in memory 512 (for input data / output data) or an alias / reference / address of a location in weight and parameter buffer 526 (for weights or parameters). The source / destination address registers may include address registers for input data and output data (in Fig. 6A and 6B ARin0, ARin1, ARout0, and ARout1 in the Fig. 6A and 6B ARwt0 and ARwt1 in the ), address registers for shifting and scaling (in Fig. 6A and 6B ARss0 and ARss1 in the Fig. 6A and 6B In some examples, field 604a may also identify a setup loop operation (SETUP_LP) that may be a start indicator of a loop instruction, field 604c may identify a loop count register LC-reg (which may be part of configuration register 528d) that stores a loop count value for the loop, and sub-instruction 604 may be processed by compute controller 522.

[0063] In addition, sub-instruction 606 includes fields 606a, 606b, and 606c. Field 606a may identify an operation to be performed to update the address stored in the source address register identified by field 606c, and the updated address is to be stored in the destination address register identified by field 606b. The operation may include, for example, increment by one (ADD), decrement by one (SUB), and move operations (MOV) to replace the address in the destination address register with the address in the source address register. The address is a memory address in memory 512 for inputting or outputting data, and the source / destination address register may include address registers for inputting data and outputting data (in Fig. 6A and 6BAs will be described below, in situations where in-situ computation or circular addressing of input data is to be performed, address generator 532 may use a circular buffer to process the adjusted addresses to provide circular addressing of input / output data.

[0064] In addition, sub-instruction 608 includes fields 608a, 608b, and 608c. Field 608a may indicate a load instruction to obtain weights / biases / scaling / shifts from addresses stored in address registers identified by field 608c (e.g., ARwt0, ARwt1, ARss0, ARss1, ARbias0, ARbias1) and store the obtained weights / biases / scaling / shifts in registers identified by field 608b (weight-0, weight-1, scale-shift-0, scale-shift-1, MACreg0-MACreg7).

[0065] In addition, sub-instruction 610 includes fields 608a, 608b, and 608c. Field 608a may indicate whether sub-instruction 610 is a load instruction or a store instruction. For a load instruction, field 610c may identify an address register (ARin0, Arin1) storing a memory address from which input data will be loaded, and field 610b may identify an input data register (Din0 or Din1) storing input data. For a store instruction, field 610c may identify an address register (ARout0, ARout1) storing a memory address for storing output data, and field 610b may identify an output data register (Dout) from which output data will be obtained.

[0066] In some examples, each of the Din0, Din1, and Dout registers has 32 bits, each of the weight and scale-shift registers has 64 bits, and the MACreg register has 72 bits.

[0067] Figure 6CExamples of 24-bit instructions 630, 632, 634, and 636 are shown. Instruction 630 may be a 24b type 1 instruction and include sub-instructions 602, 606, 610. Instruction 632 may be a 24b type 2 instruction and include sub-instruction 602 and sub-instruction 608', which includes the functionality of sub-instruction 608 (for getting weights and post-processing parameters from weight and parameter buffer 526 to weight / parameter register 528b) and automatic increment of buffer address. Instruction 634 may be a 24b type 3 instruction and include sub-instruction 640 for flow control (macros, stop) and debugging (breakpoints). In addition, instruction 636 may be a 24b type 4 instruction and include sub-instructions 604 and 606. Each of the 24b instructions 630, 632, 634, and 634 also includes sub-instructions 614 (for indicating the type / bit length of the instruction) and 612 (for supporting flow control).

[0068] Configuring the neural network processor 502 to execute instructions of various bit lengths and various numbers of sub-instructions can reduce code size and power while providing flexibility to maximize the parallelism of sub-instruction execution. Specifically, the neural network processor 502 does not always execute the five sub-instructions 602, 604, 606, 608, and 610 to support neural network calculations, and some combinations of sub-instructions (e.g., sub-instructions that read from memory and increment address registers) are executed together more frequently than other combinations. Therefore, by supporting 24b instructions with sub-instructions that are executed more frequently together, code size can be reduced. The power consumed in fetching and decoding shortened instructions can also be reduced. On the other hand, by supporting 48b instructions containing five sub-instructions 602-610, the parallelism of sub-instruction execution can also be maximized. All of this can improve the flexibility of the neural network processor 502 in supporting different applications with different power, code size, and execution parallelism requirements.

[0069] Figure 7 A diagram 700 is included showing a set of instructions / microcode stored in instruction buffer 520. Figure 7 In , the values ​​in column 702 may represent the address of each instruction in instruction buffer 520. Figure 7In the example of , the neural network processor 502 can execute instructions in sequence according to the values ​​in column 702, and each value in column 702 can also represent an execution time slot. In addition, column 706 represents a sub-instruction 610 executed by the load / store controller 530 when acquiring weights, column 704 represents a sub-instruction 608 executed by the load / store controller 530 when acquiring input data, column 708 represents a sub-instruction 606 executed by the address generator 532 when updating a memory address from which input data is acquired, column 710 represents a sub-instruction 604 executed by the address generator 532 when updating a memory address from which weights are acquired, and column 712 represents a sub-instruction 602 executed by the computing engine 524 when performing MAC calculations.

[0070] In time slot 0, the load / store controller 530 begins to obtain input data at the first memory address of the memory 512 specified in the ARin0 register. The input data will be stored at the input data register Din0. In addition, the load / store controller 530 obtains the first set of weight elements at the first buffer address (of the weight and parameter buffer 526) specified in the ARwt0 register and stores the weights at the weight register Wt0. The address generator 532 also increments the first memory address in the ARin0 register to generate a second memory address, and increments the first buffer address in the Wt0 register to generate a second buffer address. Input data can continue to be obtained from the memory 512 in time slot 1.

[0071] In time slot 2, the acquisition of input data from memory 512 to input data register Din0 is completed. In addition, the load / store controller 530 acquires a second set of weight elements at the second buffer address specified in the ARwt0 register and stores the weights at the weight register Wt1. The address generator 532 also increments the second buffer address in the Wt0 register to generate a third buffer address. The computation engine 524 also executes subinstruction 602 to perform a MAC calculation between the input data stored in Din0 and the subset of weight elements in the weight register Wt0 (Wt0L), while the load / store controller 530 acquires the weights to the weight register Wt1.

[0072] At time slot 3, the load / store controller 530 begins to obtain input data at the second memory address of the memory 512 specified in the ARin0 register. The input data will be stored at the input data register Din1. The address generator 532 also increments the second memory address in the ARin0 register to generate a third memory address. The computation engine 524 also executes subinstruction 602 to perform a MAC calculation between the input data stored in Din0 and the remaining weight elements in the weight register Wt0 (Wt0H). The load / store controller 530 pauses obtaining weights from the third buffer address at time slot 3 because the weights in the weight registers Wt0 and Wt1 are either in use (at time slot 3) or have not yet been used.

[0073] In time slot 4, the input data continues to be obtained at the second memory address of the memory 512. The MAC calculation between the input data stored in Din0 and the remaining weight elements in the weight register Wt0 is completed. Therefore, the load / store controller 530 obtains the third set of weight elements from the third buffer address to the weight register Wt0. The address generator 532 also increments the third buffer address to the fourth buffer address. The computing engine 524 also executes the sub-instruction 602 to perform a MAC calculation between the input data stored in Din0 and the subset of weight elements in the weight register Wt1 (Wt1L).

[0074] The fetching of input data to Din1 is completed at time slot 5. However, the load / store controller 530 pauses fetching input data from the third memory address to the input data register Din0 because the compute engine 524 is still operating on the input data in Din0. At time slot 5, the compute engine 524 also executes subinstruction 602 to perform a MAC calculation between the input data stored in Din0 and the remaining weight elements in the weight register Wt1 (Wt1H).

[0075] At time slot 6, the load / store controller 530 retrieves the fourth set of weight elements from the fourth buffer address to the weight register Wt1, while the compute engine 524 executes sub-instruction 602 to perform a MAC calculation between the input data stored in Din0 and the first set of weights stored in the weight register Wt0 (Wt0L) and updated by the load / store controller 530 at time slot 4. The address generator 532 also increments the fourth buffer address to the fifth buffer address. At time slot 7, the compute engine 524 also executes sub-instruction 602 to perform a MAC calculation between the input data stored in Din0 and the remaining weight elements in the weight register Wt0 (Wt0H).

[0076] At time slot 8, when the computation engine 524 executes the sub-instruction 602 to perform a MAC calculation between the input data stored in Din0 and the weight elements in the weight registers Wt1 (Wt1L and Wt1H), the load / store controller 530 obtains the weight to the weight register Wt0 at the fifth buffer address. The weight obtained to the weight register Wt0 at time slot 8 will be used by the computation engine 524 for the MAC operation with the input data stored in Din1. The address generator 532 also increments the fifth buffer address to generate a sixth buffer address.

[0077] In time slot 9, the computation engine 524 also executes subinstruction 602 to perform MAC calculations between the input data stored in Din0 and the remaining weight elements in the weight register Wt1 (Wt1H), and completes the first set of MAC calculations between Din0 and the weights. The results of the MAC calculations are stored in data registers MACreg0-MACreg7.

[0078] At time slot 10, the load / store controller 530 fetches the weights at the sixth buffer address into the weight register Wt1, while the compute engine 524 executes subinstruction 602 to perform a MAC calculation between the input data stored in Din1 and a subset of the weight elements in the weight register Wt0 (Wt0L).

[0079] In time slot 11, the load / store controller 530 obtains input data to register Din0 at the third memory address of the memory 512. The address generator 532 also increments the third memory address to generate a fourth memory address, and the computation engine 524 executes subinstruction 602 to perform a MAC calculation between the input data stored in Din1 and the remaining weight elements in the weight register Wt0 (Wt0H).

[0080] like Figure 7As shown, the neural network processor 502 can execute instructions to maximize (or at least increase) the parallelism between the acquisition of weights and input data by the load / store controller 530, the update of the memory and buffer addresses for the address generator 532 to acquire weights and input data, and the computational operations at the computation engine 524. The stalling of the computation engine 524 caused by the delay of the load / store controller 530 to acquire weights and input data can also be minimized (or at least reduced). For example, in time slots 2 and 3, while the computation engine 524 performs calculations on the weights in register Wt0, the load / store controller 530 updates the weights in register Wt1, thereby allowing the computation engine 524 to perform calculations on the updated weights in register Wt1 in subsequent time slots 4 and 5 without additional delay, and the computation engine 524 does not need to stop waiting for the weights in register Wt1 to be updated. In addition, input data is acquired from the memory 512 to one register (e.g., Din1) while the computation engine 524 operates on input data stored in other registers (e.g., Din0), so that the computation engine 524 does not need to stop waiting for the updated input data. In addition, by spacing memory 512 accesses for relatively long durations (e.g., Figure 7 8 time slots are separated in the middle), memory access conflicts between the processor 514 and the neural network processor 502 can also be avoided or at least reduced.

[0081] Fig. 8A and Figure 8B is a diagram illustrating an example memory management scheme that may be implemented by address generator 532 . Fig. 8A and 8B The example memory management scheme in can represent an in-situ calculation in which the output data overwrites the input data, and a circular addressing implementation of the output data. With such an arrangement, the input and output data do not need to be stored in separate spaces in the memory 512, which can reduce the memory resource usage of the neural network processor 502.

[0082] Referring to diagram 800, memory 512 can store 16 groups of input data elements (fm0), where each group of input data elements includes four input data elements for four different channels (c0, c1, c2, and c3). Each row in diagram 800 can be associated with a memory address, and rows can be associated with consecutive addresses. However, space for storing 17 groups of input data elements can be allocated, but row 16 (memory space 802) is empty and can be used to store a group of output data elements (fm1).

[0083] At time T0, the initial read address is at row 0, as indicated by pointer 804. The first set of input data elements in row 0 are retrieved to an input data register (e.g., Din0) and then retrieved to the computation engine 524 for computational operations. The initial write address of the first set of output data elements of the computational operation is associated with memory space 802, as indicated by pointer 806. Such an arrangement can prevent the output data from overwriting the first set of input data elements, which may still be in use. The initial read and write addresses can be set using sub-instructions 606. After the first set of input data elements are retrieved, the address generator 532 can increment the read address one by one in response to another sub-instruction 606 so that the pointer 804 can then point to row 1.

[0084] At time T1, a second set of input data elements (fm0_01c0-fm0_01c3) in row 1 are retrieved into the input data register, as indicated by pointer 804. After retrieving the second set of input data elements, the address generator 532 may increment the read address by one in response to sub-instruction 606, so that pointer 804 may then point to row 2. A first set of output data elements (fm1_00c0-fm1_00c3) are stored in row 16, as indicated by pointer 806. After storing the first set of output data elements in row 16, the address generator 532 may increment the write address by one in response to sub-instruction 606. However, to support in-situ computation, circular addressing may be provided for output data, so that the incremented write address may wrap around and point to row 0, so that the next set of output data elements may overwrite the first set of input data elements.

[0085] refer to Figure 8B , at time T2, the third set of input data elements in row 2 (fm0_02c0-fm0_02c3) are retrieved into the input data register, as indicated by pointer 804. After retrieving the third set of input data elements, the address generator 532 may increment the read address by one in response to sub-instruction 606, so that pointer 804 may then point to row 3. The second set of output data elements (fm1_00c0-fm1_00c3) are stored in row 0, as indicated by pointer 806. After storing the second set of output data elements in row 0, the address generator 532 may increment the write address by one in response to sub-instruction 606, so that pointer 806 may then point to row 1. Finally, at time T3, all of the output data elements are in rows 0-14 and 16, and pointer 806 points to row 14, which stores output data elements fm1_15_c0-fm1_15_c3.

[0086] Fig. 9 A diagram illustrating another example memory management scheme that may be implemented by address generator 532 is included. Fig. 9The example memory management scheme in may represent circular addressing of input data to support continuously online applications, such as voice command recognition applications. For such applications, reasoning operations may be executed more frequently than the command length. Therefore, there may be significant overlap of input data between two reasoning operations.

[0087] For example, for voice command recognition, the input speech samples may be grouped into 40 millisecond (ms) frames, and 8 frequency domain features may be extracted for each frame. Thus, for a command length of 1 second, features of 25 frames are extracted, resulting in input features having 25 positions and 8 channels at each position. In a case where an inference operation is performed for each 120 ms, the new input features to be processed by the inference operation may have three sets of new features (at three positions, 8 channels at each position) relative to the previous set of input features. The overlapping 22 sets of input features may be moved in the memory so that the input features fit within the same set of allocated memory addresses, but such data movement increases the latency of memory access and the power consumption of the memory.

[0088] Rather than moving overlapping input data in memory 512, new input data may be stored in memory 512 (e.g., by sensor 104, by processor 514, etc.) according to a circular addressing scheme, and address generator 532 may also update the input data address based on the same circular addressing scheme to obtain the input data. Fig. 9 900 in shows an example of a circular addressing scheme. In diagram 900, memory 512 can store 25 groups of input data elements, where each group of input data elements includes eight input data elements for eight different channels (c0-c7). Each row in diagram 900 can be associated with a memory address, and rows can be associated with consecutive addresses.

[0089] At time T0, the memory 512 stores 25 initial sets of input data elements (fm0_00*-fm0_24*). The first address for retrieving the 25 sets of input data elements to perform the first speculative operation is at row 0, indicated by pointer 902. The address generator 532 may provide the initial input data address as an address associated with row 0 and increment the input data address in response to the sub-instruction 606, and the load / store controller 530 may retrieve the 25 sets of input data elements from the memory 512 at the input data address.

[0090] At time T1, new three groups of input data elements (fm0_25*-fm0_27*) are stored in memory 512. To avoid moving the remainder of the initial group of input data elements, the new group of input data elements is stored in memory addresses associated with rows 0, 1, and 2, overwriting input data elements fm0_00*-fm0_02*. The first address for retrieving the 25 groups of input data elements to perform the second inference operation is at row 3, indicated by pointer 902. The address generator 532 may provide the initial input data address as the address associated with row 3, and increment the input data address in response to the sub-instruction 606, and the load / store controller 530 may retrieve the 25 groups of input data elements from memory 512 at the input data address. However, to read the new group of input data elements, the address generator 532 also implements a circular addressing scheme for the input data, in which the input data address wraps around after incrementing beyond the address associated with row 24 (storing input data element fm0_24*), and restarts at row 0 and ends at row 2.

[0091] At time T2, three additional new groups of input data elements (fm0_28*-fm0_30*) are stored in memory 512. To avoid moving the remainder of the initial groups of input data elements, these new groups of input data elements are stored in memory addresses associated with rows 3, 4, and 5, overwriting input data elements fm0_03*-fm0_05*. The first address for retrieving the 25 groups of input data elements to perform the third inference operation is at row 6, indicated by pointer 902. The address generator 532 may provide the initial input data address as the address associated with row 6, and increment the input data address in response to the sub-instruction 606, and the load / store controller 530 may retrieve the 25 groups of input data elements from memory 512 at the input data address. However, in order to read a new group of input data elements, address generator 532 also implements a circular addressing scheme for the input data, in which the input data address wraps around after incrementing beyond the address associated with row 24 (storing input data element fm0_24*) and restarts at row 0 and ends at row 5.

[0092] Fig. 10A An example of a circular buffer circuit 1000 that may be part of the address generator 532 is shown. Fig. 10B A diagram illustrating an example operation of the circular buffer 1000 is provided. Fig. 10B , the circular buffer circuit 1000 can receive the input / output data address ARw0. If the circular addressing is disabled, the circular buffer 1000 can forward ARw0 as the output address, or ARw0 at the starting address of the memory area 1001 allocated for input / output data ( Fig. 10A and 10BThe start-addr) and end address ( Fig. 10A and Fig. 10B . Referring to the left side figure, if circular addressing is enabled, and if ARw0 becomes a certain offset (offset1) lower than the starting address (by decrementing), then the circular buffer circuit 1000 can perform a wraparound and provide a circularized version of ARw0 (ARw0_cir0) by subtracting offset1 from the end address. In addition, referring to the right side figure, if circular addressing is enabled, and if ARw0 becomes a certain offset (offset 2) higher than the starting address (by incrementing), then the circular buffer circuit 1000 can perform a wraparound and provide a circularized version of ARw0 (ARw0_cir1) by subtracting offset2 from the starting address. On the other hand, if circular addressing is disabled, or if ARw0 is within the memory area 1001, then the circular buffer circuit 1000 can forward ARw0 as the output address.

[0093] refer to Fig. 10A , the circular buffer circuit 1000 may include multiplexers 1002, 1004, 1006, and 1008, difference circuits 1012, 1014, and 1016, a summing circuit 1020, an incrementing circuit 1022, a decrementing circuit 1024, and multiplexer control circuits 1030 and 1032. The circular buffer circuit 1000 has an address input 1040 coupled to an input / output data address register (e.g., ARin0 or ARout0) to receive an input / output data address (ARw0).

[0094] Multiplexer 1002 may receive a starting input data address (“start-in” in FIG. 10 ) and a starting output data address (“start-out” in FIG. 10 ), and multiplexer 1004 may receive an ending input data address (“end-in” in FIG. 10 ) and an ending output data address (“end-out” in FIG. 10 ). The starting and ending input data addresses may define a memory region allocated to input data, and the starting and ending output data addresses may define a memory region allocated to output data, wherein the starting address is lower / less than the ending address. Each of multiplexers 1002 and 1004 may also receive an indication M0 indicating whether the circular addressing mode is provided for updating input data addresses (e.g., for continuous online applications) or for updating output data addresses (e.g., for in-situ calculations). Indication M0 may also be received from configuration register 528 c. If the circular addressing mode provides for updating the input data address, then the multiplexer 1002 may provide the starting input data address as the starting address ("start-addr" in FIG. 10 ), and the multiplexer 1004 may provide the ending input data address as the ending address ("end-addr" in FIG. 10 ), otherwise the multiplexer 1002 may provide the starting output data address as the starting address, and the multiplexer 1004 may provide the ending output data address as the ending address. The increment circuit 1022 has an input coupled to the output of the multiplexer 1004 and may generate an incrementing end address 1060, and the decrement circuit 1024 has an input coupled to the output of the multiplexer 1002 and may generate a decrementing start address 1062.

[0095] In addition, a difference circuit 1012 has a first input coupled to the address input 1040 and a second input coupled to the output of the multiplexer 1002. A difference circuit 1014 has a first input coupled to the address input 1040 and a second input coupled to the output of the multiplexer 1004. The difference circuit 1012 can generate an offset 1050 between the starting address and ARw0, which also indicates whether ARw0 is above or below the starting address. In addition, the difference circuit 1014 can generate an offset 1052 between the ending address and ARw0, which also indicates whether ARw0 is above or below the starting address. In addition, the difference circuit 1016 can generate an address 1070 by subtracting the offset 1050 from the incrementing ending address 1060, and the summing circuit 1020 can generate an address 1072 by adding the offset 1052 to the decrementing ending address 1062. If ARw0 is lower than the start address, then address 1070 may be the wraparound address ARw0_cir0, and if ARw0 is higher than the end address, then address 1072 may be the wraparound address ARw0_cir1.

[0096] Multiplexers 1006 and 1008 may selectively forward one of input address ARw0, address 1070 (ARw0_cir0), or address 1072 (ARw0_cir1) to address output 1042. The selection is performed by multiplexer control circuits 1030 and 1032. Specifically, multiplexer control circuits 1030 and 1032 may each receive an indication of whether circular addressing is enabled ("circ enable" in FIG. 10) from, for example, configuration register 528c. If circular addressing is disabled, or ARw0 is above the start address, or ARw0 is below the end address, then multiplexers 1006 and 1008 may connect address input 1040 to address output 1042 and provide the address stored in the input / output data address register as an output. On the other hand, if circular addressing is enabled and ARw0 is below the start address, then multiplexers 1006 and 1008 may provide ARw0_cir0 (address 1070) to address output 1042. Additionally, if loop addressing is enabled and ARw0 is above the ending address, multiplexers 1006 and 1008 may provide ARw0_cir1 (address 1072 ) to address output 1042 .

[0097] As discussed above, the neural network processor 502 supports sub-instructions that allow for compact representation of convolutional layer computations, including flow control elements, such as loop elements and macro elements, such as Figure 4 Those shown. Fig.11 An example program including a loop element that can be executed by the neural network processor 502 is shown. Fig.11At the top, program 1100 may include instructions 1102, 1104, 1106, and 1108, each of which includes one or more sub-instructions. Instruction 1102 may include sub-instruction 604 having field 604a identifying a setup loop operation (SETUP_LP) and field 604c identifying a loop count register (LC-reg). Field 604a may include an opcode indicating a setup loop operation. Instruction 1102 is followed by instructions 1104 and 1106, each of which has sub-instruction 612 indicating a zero / deasserted end marker. Program 1100 also includes instruction 1108, which has sub-instruction 612 indicating an asserted end marker. Instructions 1102, 1104, 1106, and 1108 may define a loop operation that starts at instruction 1104 and ends at instruction 1108, and the loop operation will be repeated the number of times defined in the loop count register identified by field 604c of instruction 1102. To perform the loop operation, the compute controller 522 may identify one or more pairs of sub-instructions containing the opcode SETUP_LP and the asserted end marker by tracking the order / sequence of extracting the opcode SETUP_LP and the asserted end marker when the compute controller 522 parses the instruction, and store the program counter value of the sub-instruction pair. After extracting the asserted end marker, the compute controller 522 may reset the program counter back to the count value of the sub-instruction with the paired SETUP_LP opcode, and repeat the execution of the instruction between the pair of SETUP_LP sub-instructions and the asserted end marker.

[0098] Fig.11Also shown is a program 1120 representing a nested loop operation. Program 1120 includes instructions 1122, 1124, 1126, 1128, 1130, and 1132, each of which includes one or more sub-instructions. Instruction 1122 includes an opcode SETUP_LP and identifies a first loop register LC-reg0 storing a first loop count. Instruction 1124 includes an opcode SETUP_LP and identifies a second loop register LC-reg1. Instruction 1130 includes an asserted end tag, and sub-instruction 1132 also includes an asserted end tag. The compute controller 522 may pair the SETUP_LP of sub-instruction 1122 with the asserted end tag of instruction 1132, and pair the SETUP_LP of sub-instruction 1124 with the asserted end tag of instruction 1130, based on the order in which the compute controller 522 extracts the SETUP_LP opcode and the asserted end tag. The compute controller 522 may repeat the execution of instructions 1124, 1126, and 1128 the number of times specified in LC-reg0. In addition, after extracting the asserted end marker of the sub-instruction 1130, the compute controller 522 may reset the program counter to restart the execution of the sub-instruction 1124, and repeat the execution of instructions 1126 and 1128 the number of times specified in LC-reg1, thereby performing a nested loop operation.

[0099] Fig.12 An example program 1200 is shown that includes a macro element that can be executed by the neural network processor 502. Program 1200 includes instructions 1201, 1202, 1204, 1206, and 1208, each of which includes one or more sub-instructions. Each instruction is also associated with a program counter value and has an end marker that is asserted or deasserted. Instruction 1206 includes fields 1206a, 1206b, and 1206c. Field 1206a may include a macro element operation code (MACRO) that identifies a macro, which includes a group of instructions. Field 1206b may identify the program counter value of the first instruction of the macro (e.g., PC#0), and the last instruction of the macro may include an asserted end marker sub-instruction 612. Field 1206c may indicate the number of times the macro will be executed. After executing instruction 1206, compute controller 522 may reset the program counter to the count value of field 1206b and execute all subsequent instructions until instruction 1204 with an asserted end marker is received. Instruction 1204 may also include Fig.12 The calculation controller 522 may repeat the execution of the macro the number of times specified in field 1206c. After repeating the execution of the macro the number of times, the calculation controller 522 may stop resetting the program counter after executing instruction 1206 and continue to execute subsequent instructions 1208.

[0100] Fig.13is a schematic diagram showing an example of the internal components of the computing engine 524. Fig.13 , the computation engine 524 includes a MAC engine 1300 and a post-processing engine 1302. The MAC engine 1300 may receive input data elements and weight elements from input data registers (e.g., DIN0, DIN1) of the data register 528a, weight registers (e.g., weight-0, weight-1) of the weight / parameter register 528b, and scale-shift registers (e.g., scale-shift-0, scale-shift-1) also belonging to the weight / parameter register 528b, respectively. In response to a control signal 1301 from the computation controller 522 that generates a control signal in response to the sub-instruction 602 indicating a MAC operation, the MAC engine 1300 may perform MAC calculations on the input data elements and weight elements. The MAC engine 1300 includes arithmetic circuits to perform MAC calculations of neural network layers, such as CNN layers, DNN layers, fully connected layers, etc., and store intermediate output data at MAC registers (e.g., MACreg0-7) of the data register 528a. The MAC engine 1300 may also retrieve the old partial sum from the MAC register, update the old partial sum by adding the result of the MAC calculation to the old partial sum, and store the updated partial sum back into the MAC register. The calculation engine 524 also includes a MAC register multiplexer 1303 for selecting which MAC register to access by the MAC engine 1300 to perform the accumulation.

[0101] In addition, the post-processing engine 1302 can receive intermediate output data elements from the MAC register. In response to the control signal 1305 from the compute controller 522 that generates the control signal in response to the sub-instruction 602 indicating the BNorm operation, the post-processing engine 1302 can perform post-processing operations (e.g., BNorm and residual layer processing) on ​​the intermediate output data to generate output data, and store the output data at the output data register (e.g., DOUT) of the data register 528a. As will be described below, the processing engine 1302 can also perform a data packing operation at the output data register and transmit a signal to the load / store controller 530 to store the output data from the output data register back to the memory 512 after the data packing operation is completed. Such an arrangement can reduce the access of the neural network process 502 to the memory 512 when writing back the output data, which can reduce memory usage and power consumption. In addition, the post-processing engine 1302 can also receive input data elements from the input data register and perform residual mode operations based on the input data elements.

[0102] As described above, the computation engine 524 may be configured to perform MAC and post-processing operations on weights and input data across a range of bit precision (e.g., binary precision, ternary precision, 4-bit precision, 8-bit precision, etc.) based on parameters provided by the processor 514. The computation engine 524 includes a weight multiplexer 1304 and an input data multiplexer 1306 to support precision configurability. Specifically, depending on the input data and weight configuration 1310, the weight multiplexer 1304 may obtain all weight elements stored in a weight register (e.g., one of the weight-0 or weight-1 registers), or copy half of the stored weight elements as packed data with a predetermined number of bits (e.g., 32 bits). In addition, depending on the configuration 1311, the weight multiplexer 1304 may perform processing on the weight elements to support various operations, such as depthwise convolution operations and average pooling. For depth convolution, the weight multiplexer 1304 may select an 8-bit weight stored in a weight register, divide the 8-bit weight into weight element groups (e.g., four groups), and fill each weight element group with zeros so that the MAC engine 1300 may multiply the input data elements of a particular channel by zeros. Such an arrangement may ensure that the intermediate output of a particular channel is a MAC operation based on the input data of only the channel. As another example, for average pooling operations, the weight multiplexer 1304 may selectively forward zeros and ones as weight elements, wherein the input data elements paired with the weight element one are represented in the intermediate output data elements, and the input data elements paired with the zero weight element are zeroed and not represented in the intermediate output data elements. Configuration 1311 may indicate a layer type, which may also indicate whether a depth convolution operation will be performed. Configuration 1311 may also indicate whether an average pooling operation will be performed. Configuration 1311 may be based on configuration data stored in configuration register 528d and / or a control signal from the computation controller 522 in response to an instruction.

[0103] In addition, the input data multiplexer 1306 can retrieve all input data elements stored in the input data register (e.g., one of Din0 or Din1), or copy half of the stored input data elements as packed data with a predetermined number of bits (e.g., 32 bits). In some examples, the input data and weight configuration 1310 may include an 8-bit mode (D8) or a 4-bit mode (D4) that indicates whether the compute engine 524 retrieves the input data in 8-bit form or in 4-bit form. The input data and weight configuration 1310 can be stored in and received from the configuration register 528D. Whether the compute engine 524 operates in D8 or D4 mode can depend on the input and weight precision, which can also determine the number of input data elements and the number of weight elements retrieved to the MAC engine 1300 at one time (e.g., in one clock cycle).

[0104] In addition, the MAC engine 1300 and the post-processing engine 1302 can receive input and weight precision configuration 1312. Depending on the input data precision and weight precision, the arithmetic circuits and logic circuits of the MAC engine 1300 and the post-processing engine 1302 can handle calculations in different ways. The post-processing engine 1302 also receives post-processing parameters 1314 that can define, for example, parameters for post-processing operations, some or all of which may also depend on input and / or weight precision. The input and weight precision configuration 1312 and some post-processing parameters 1314 can be received from the configuration register 528d. Some post-processing parameters 1314 (e.g., shifting and scaling) can vary between different internal components of the post-processing engine 1302. These parameters can be obtained from the weight / parameter buffer 526 to the weight / parameter register 528b, and the post-processing engine 1302 can receive those parameters from the weight / parameter register 528b.

[0105] In addition, the computing engine 524 may also include a maximum pooling engine 1320 for performing a maximum pooling operation on the input data elements stored in the input data registers (DIN0, DIN1) and the output data elements stored in the output data register (DOUT), and storing the maximum pooling result back to the output data register (DOUT). Specifically, if the input data element at DIN0 / DIN1 has a higher value than the output data element in DOUT, the maximum pooling engine 1320 may overwrite the output data element with the input data element. The computing controller 522 may provide a control signal 1322 to the maximum pooling engine 1320 in response to, for example, field 602c of the sub-instruction 602 indicating that a maximum pooling operation will be performed. Then, the maximum pooling engine 1320 may perform the maximum pooling operation in response to the control signal 1322. The maximum pooling engine 1320 also receives post-processing parameters 1314, and the maximum pooling operation may be configured based on the parameters. In some examples, max pooling engine 1320 may operate independently or in parallel with MAC engine 1300 and post-processing engine 1302, which may minimize interference of the max pooling operation with the operation of the rest of computation engine 524 and improve efficiency.

[0106] Fig.14 An example convolution operation performed by MAC engine 1300 is shown. Fig.14, the MAC engine 1300 may perform convolution operations in a lane-major sequence. The MAC engine 1300 may first perform a convolution operation 1402 on an input data element 1404 at a particular height and width position and associated with a first set of input lanes (using a set of weights associated with a set of output lanes) to generate a first partial sum of an intermediate output data element 1406. The input data element 1404 may be stored in an input data register (e.g., Din0), and the intermediate output data element 1406 may be stored in an intermediate output data register (e.g., MACreg0). Next, the MAC engine 1300 may perform a convolution operation 1412 on an input data element 1414 at the same height and width position as the input data element 1404 but associated with a second set of input lanes to generate a second partial sum of the intermediate output data element 1406, and add the second partial sum to the first partial sum. The input data element 1414 may be stored in another input data register (e.g., Din1). The MAC engine 1300 may perform additional convolutions on input data elements at the same height and width positions of other input lanes, including input data element 1424, and accumulate partial sums at the intermediate output data registers to generate intermediate output data element 1406. After the input data elements at a particular height and width position of all input lanes have been processed, the MAC engine 1300 may restart the convolution operations on the input data elements 1404, 1414, and 1424 using another set of weights associated with a different set of output lanes to calculate another set of intermediate output data elements, such as intermediate output data elements 1430 and 1432 associated with different output lanes.

[0107] Fig.15A , Fig. 15B , Fig. 15C and Fig.15D 1 shows example convolution operations and post-processing operations performed by MAC engine 1300 and post-processing engine 1302 using weight elements and input data elements of different bit precisions. Fig.15A In the embodiment, the MAC engine 1300 can perform convolution operations between 256 8-bit input data elements (labeled as in0, in1, in2, in3, ..., in254 and in255) and 8-bit weight elements to generate 24-bit intermediate output data elements, wherein each intermediate output data element is generated by the sum of the products of the input data elements and the weight elements. The post-processing engine 1302 performs post-processing operations on each intermediate output data element, including BNorm and clamp / activation function operations (e.g., Fig.14 to produce 8-bit output data elements.

[0108] exist Fig. 15B, the MAC engine 1300 can perform convolution operations between 256 8-bit input data elements (labeled as in0, in1, in2, in3, in254, and in255) and 1-bit (binary) weight elements to generate 16-bit intermediate output data elements, where each intermediate output data element is generated by the sum of the multiplication products between the input output data elements and the weight elements, followed by post-processing operations (e.g., BNorm and ReLU) of each intermediate output data element to generate 8-bit output data elements. With binary weights (+1 or -1), the MAC engine 1300 can generate multiplication products between input data elements and weight elements simply by forwarding the input data element when the weight element is one and forwarding the negation of the input data element when the weight element is -1, which can reduce complexity and power and increase computational speed. Weight storage can also have an 8x reduction compared to 8-bit weights. Although the precision of intermediate output data elements is reduced (from 24 bits to 16 bits), this reduced precision may still be acceptable to some applications, and the impact of the reduced precision can be further mitigated by having the neural network topology have mixed precision between neural network layers.

[0109] exist Fig. 15C , the MAC engine 1300 may perform a convolution operation between 8-bit input data elements and 2-bit ternary weight elements (-1, 0, +1) to generate 16-bit intermediate output data elements, wherein each intermediate output data element is generated by the sum of multiplication products between the input output data elements and the weight elements, followed by post-processing operations (e.g., BNorm and ReLU) of each intermediate output data element to generate the 8-bit output data elements. In representing weights, 2-bit ternary weight elements (providing three alternative levels of representing weight elements) may be more accurate than binary weights (providing two alternative levels). In addition, the MAC engine 1300 may generate multiplication products between input data elements and weight elements simply by forwarding the input data element when the weight element is one, forwarding the negation of the input data element when the weight element is -1, and forwarding zero when the weight element is zero, which may reduce complexity and power and increase computational speed.

[0110] exist Fig.15DIn the embodiment of the present invention, the MAC engine 1300 can perform a convolution operation between an 8-bit input data element and two sets of 2-bit ternary weight elements (-1, 0, +1) to generate a 16-bit intermediate output data element, wherein each intermediate output data element is generated by the sum of the multiplication products between the input output data element and the two sets of weight elements, and the intermediate output data elements generated from the two sets of weight elements are summed. The summed intermediate output data elements can then be post-processed (e.g., BNorm and ReLU) to generate an 8-bit output data element. When representing weights, using two sets of 2-bit ternary weight elements (providing nine alternative levels) can be more accurate than using one set of 2-bit ternary weight elements (providing three alternative levels), but the calculations in the MAC engine 1300 using ternary weight elements have similar simplified benefits.

[0111] Fig.16 Executed by MAC engine 1300 to support, for example Fig.14 Example of arithmetic operations for an example convolution operation in a channel-major sequence of . Fig.16 The arithmetic operations shown in can be performed by the MAC engine 1300 in one clock cycle. Fig.16 , in each clock cycle, the MAC engine 1300 may receive four input values ​​X[0], X[1], X[2], and X[3], each associated with a different input channel. The MAC engine 1300 may also receive 16 weight values ​​associated with different input and output channels, such as W[0,0] (associated with output channel 0 and input channel 0), W[1,3] (associated with output channel 1 and input channel 3), etc. Fig.16 The example operation of may be performed on 8-bit input data elements and 2-bit weight elements or 4-bit input data elements and 4-bit weight elements. As will be described below, depending on the input and weight precision, the calculation engine 524 may receive 8-bit input data elements or 4-bit input data elements. In addition, the calculation engine 524 may also receive 4-bit weight elements, 2-bit weight elements, 8-bit weight elements, etc. The calculation engine 524 may be configured to split the weights / data to perform the calculation by the input data and weight configuration 1310 and the precision configuration 1312. Fig.16 For example, the calculation engine 524 can split the 4-bit weight into two 2-bit weights and perform 8-bit data and 2-bit weight calculations on the two sets of weights, and each weight value can contain a single 4-bit weight element, two 2-bit weight elements, or can be half of an 8-bit weight element.

[0112] The MAC engine 1300 may generate partial sums for each of the intermediate output data elements Y[0], Y[1], Y[2], and Y[3] associated with different output channels, based on the multiplication of the input data X[0]-X[3] associated with the particular output channel and the weights with the input channels of X[0]-X[3]. For example, for the intermediate output data element Y[0], the MAC engine 1300 may calculate the multiplication product between W[0,3] and X[3], the multiplication product between W[0,2] and X[2], the multiplication product between W[0,1] and X[1], and the multiplication product between W[0,0] and X[0], and perform an accumulation operation by summing the multiplication product with the previous partial sum Y0' from one of the intermediate output data registers MACreg0-MACreg3 to generate a new partial sum Y0, and the new partial sum Y0 may be stored in MACreg0-MACreg3 in place of the previous partial sum Y0'. The summation may be a saturated summation. In the case where the first instruction is for a convolution operation, it can be accessed via a multiplexer (in Fig.16 The multiplexer (marked as MUX in the figure) obtains a bias value (e.g., Bias0) from another set of intermediate output data registers MACreg4-MACreg7 to the MAC engine 1300, and the multiplication product can be added with the bias value to generate a partial sum. Such an arrangement can avoid performing addition in subsequent post-processing operations, which can speed up the post-processing operations. After the first instruction is executed, for subsequent instructions of the convolution operation, the multiplexer can route the output of MACreg0-MACreg3 to the MAC engine 1300 to add the multiplication product to the previous partial sum.

[0113] MAC engine 1330 may perform 16 multiplication operations, such as multiplication operation 1602, to generate a multiplication product in one clock cycle. In some examples, MAC engine 1330 may include 16 multiplier circuits to perform the 16 multiplication operations in parallel. MAC engine 1330 may also update a previous partial sum Y'[0] stored in a MAC register by adding a new partial sum to Y'[0].

[0114] Table 1 below shows a set of input precisions and weight precisions supported by the compute engine 524. Each row also indicates, for a given input precision and weight precision, the number of input data elements processed by the compute engine 524 per clock cycle, the number of output data elements provided by the compute engine 524 per clock cycle, and the number of multiply-accumulate (MAC) operations performed by the compute engine 524 per clock cycle. The number of MACs per cycle may be equal to the product between the number of input data elements processed per clock cycle and the number of output data elements generated.

[0115] In some examples, as will be described below, the MAC engine 1330 may include an array of 32 4-bit by 2-bit multiplier circuits, where each multiplier circuit may perform a multiplication operation (or bitwise operation) between a 4-bit number and a 2-bit number in each clock cycle, and the 32 multiplier circuits may perform 32 multiplications between a 4-bit number and a 2-bit number.

[0116] For 8-bit input data elements and 2-bit weight elements (binary or ternary), each input data element can be split into two 4-bit data values, where two multiplier circuits perform an operation of one 8-bit data element and one 2-bit weight element. Therefore, an array of 32 multiplier circuits can perform 32 / 2 computation operations for 8-bit input data and 2-bit weights per clock cycle.

[0117] For 8-bit input data elements and 4-bit weight elements, each input data element can be split into two 4-bit data values, and each 4-bit weight element can be split into two 2-bit weight values, where four multiplier circuits perform operations for one 8-bit data element and one 4-bit weight element. Therefore, an array of 32 multiplier circuits can perform 32 / (2*2) computation operations for 8-bit input data and 4-bit weights per clock cycle.

[0118] For 8-bit input data elements and 8-bit weight elements, each input data element can be split into two 4-bit data values, and each weight element can be split into four 2-bit weight values, where eight multiplier circuits perform the operation of one 8-bit data element and one 8-bit weight element. Therefore, an array of 32 multiplier circuits can perform 32 / (2*4) calculation operations of 8-bit input data and 4-bit weights in each clock cycle.

[0119] For a 4-bit input data element and a 4-bit weight element, each weight element can be split into two 2-bit weight values, where two multiplier circuits perform an operation of one 4-bit data element and one 2-bit weight element. Therefore, an array of 32 multiplier circuits can perform 32 / 2 computation operations for 4-bit input data and 2-bit weights per clock cycle.

[0120] For binary weights and data, the MAC engine 1330 (or the computation engine 524) can internally convert the binary weights into 2-bit weights. As will be described below, each multiplier circuit can perform four bitwise computation operations (e.g., XNOR) between the binary weights and data, and an array of 32 multiplier circuits can perform 128 computation operations on the binary weights and data per clock cycle.

[0121] Table 1

[0122] Input precision Weight accuracy Input number Number of outputs MAC / cycle number Signed / unsigned 8-bit Binary (-1,1) 4 4 16 Signed / unsigned 8-bit Ternary (-1,0,1) 4 4 16 Signed / unsigned 8-bit Signed 4 bits 4 2 8 Signed / unsigned 8-bit Signed 8-bit 4 1 4 Signed / unsigned 4-bit Signed 4 bits 4 4 16 Signed / unsigned 4-bit Ternary (-1,0,1) 8 4 32 Binary (-1,1) Binary (-1,1) 16 8 128

[0123] Fig.17 1 shows the operation of an example multiplier circuit 1700 that may be part of the MAC engine 1300. The example multiplier circuit 1700 may perform Fig.16 , such as multiplication operation 1602. Multiplication circuit 1700 may receive an 8-bit input value D[7:0], which may be Fig.16 One of X[0]-X[3], and a 4-bit weight value W[3:0], which can be Fig.16 The input value D and the weight value W may each be signed or unsigned. The multiplier circuit 1700 may include two 4-bit x 2-bit multiplier circuits 1702 and 1704. Fig.17 , the logical operations of multiplier circuits 1702 and 1704 are represented in the form of instructions, which can be synthesized into (and represented by) logic circuits representing multiplier circuits 1702 and 1704.

[0124] The multiplier circuit 1700 may receive a multiplier configuration 1710 to configure each of the multiplier circuits 1702 and 1704 to perform multiplication operations in various modes. The multiplier configuration 1710 may be part of the precision configuration 1312 and may include a first flag indicating whether D[7:4] is signed (e.g., if the flag is asserted or indicates a logic, then D[7:4] is signed, if the flag is deasserted or indicates a logic zero, then unsigned), a second flag indicating whether D[3:0] is signed (both based on the input precision), a third flag indicating whether W[3:2] is signed, a fourth flag indicating whether W[1:0] is signed (both based on the weight precision), and a fifth flag indicating whether to operate in binary mode. If the precision configuration 1312 indicates that the input data and weights each have one bit (binary) precision, then the multiplier circuits 1702 and 1704 may operate in binary mode.

[0125] If the multiplier configuration 1710 indicates that the binary mode is disabled, the multiplier circuit 1702 can generate the output N0 as a 7-bit signed number by performing a multiplication operation between the four LSBs D[3:0] and the two LSBs W[1:0], and the multiplier circuit 1704 can generate the output N1 as a 7-bit signed number by performing a multiplication operation between the four MSBs D[7:4] and the two MSBs W[3:2]. Based on the multiplier configuration 1710, if at least one of D[3:0] or W[1:0] is signed, the multiplication operation of N0 can be a signed multiplication operation, and if at least one of D[7:4] or W[3:2] is signed, the multiplication operation of N1 can be a signed multiplication operation.

[0126] On the other hand, if multiplier configuration 1710 indicates binary mode is enabled, then multiplier circuit 1702 may perform a bitwise XNOR operation between D[3:0] and W[3:0] (e.g., Fig.17 In The bitwise XNOR outputs are summed to generate an 8-bit output N0. In addition, the multiplier circuit 1704 can generate an 8-bit output N1 by performing a bitwise XNOR operation between D[7:4] and W[3:0] and summing the bitwise XNOR outputs. A truth table for the XNOR operation is provided below, where -1 is represented by a logical zero and +1 is represented by a logical one. As can be seen from the table, the multiplication of two binary numbers represented in this way can be calculated using the XNOR operation.

[0127] Table 2

[0128] XNOR Input 0 XNOR Input 1 XNOR Output +1 +1 +1 +1 -1 -1 -1 +1 -1 -1 -1 +1

[0129] The MAC engine 1300 may include multiple computational units, each of which includes a set of multiplier circuits 1700 and other logic circuits for performing MAC operations for the input precision and weight precision ranges described in Table 1 to generate intermediate output data elements. Fig.18 An example of a computing unit 1800 is shown. Fig.18 The computing unit 1800 may have four computing data inputs, receiving Fig.18 The calculation unit 1800 also has four calculation weight inputs, which receive Fig.18 1700a, 1700b, 1700c, and 1700d. The input of multiplier circuit 1700a is coupled to a first calculation data input and a first calculation weight input. The input of multiplier circuit 1700b is coupled to a second calculation data input and a second calculation weight input. The input of multiplier circuit 1700c is coupled to a third calculation data input and a third calculation weight input. The input of multiplier circuit 1700d is coupled to a fourth calculation data input and a fourth calculation weight input. Each multiplier circuit 1700 generates two 7-bit outputs N0 and N1, as described above. Multiplier circuit 1700a generates 7-bit outputs N0a and N1a, multiplier circuit 1700b generates 7-bit outputs N0b and N1b, multiplier circuit 1700c generates 7-bit outputs N0c and N1c, and multiplier circuit 1700d generates 7-bit outputs N0d and N1d. Each multiplier circuit 1700a-d also receives a multiplier configuration 1710, as described above.

[0130] The computing unit 1800 also includes adders 1802a, 1802b, and 1802c, and adders 1804a, 1804b, and 1804c. Adders 1802a, 1802b, and 1802c generate the sum of N1a, N1b, N10c, and N1d as MAC_L, which has 9 bits and is represented as a signed number when operating in non-binary mode. In addition, adders 1804a, 1804b, and 1804c generate the sum of N0a, N0b, N0c, and N0d as MAC_R, which also has 9 bits and is represented as a signed number in non-binary mode. The computing unit 1800 also includes a bit shifter circuit 1806, which can shift MAC_L to the left by the number of bits specified in the shift control 1808 to generate MAC_L'. The computing unit 1800 may receive a shift control 1808 from the configuration register 528c. As will be described below, the amount of the left shift is based on the input and weight precision. In addition, the computing unit 1800 includes an adder 1809 that generates the sum of MAC_R and MAC_L' as MAC_out, which has 13 bits and can be represented as a signed number in non-binary mode, and as a MAC output. MAC_L and MAC_R can also be MAC outputs.

[0131] The calculation unit 1800 also includes an accumulator 1810, which can receive MACreg_in (18 bits) as the old partial sum and update the old partial sum by adding it to the MAC output to generate a new partial sum MACreg_out. Fig.19 Example internal components of accumulator 1810 and its operation are shown. Fig.19 In FIG. 1 , the logical operation of the accumulator 1810 is represented in the form of instructions, which can be synthesized into (and represented by) a logic circuit representing the accumulator 1810 .

[0132] refer to Fig.19 , the accumulator 1810 may include two adders 1902 and 1904, each of which may be a 9-bit adder. If the MAC engine 1300 is not operating in binary mode, then the adders 1902 and 1904 may form an 18-bit adder, and the 18-bit adder updates the old partial sum (MACreg_in) by adding the 13-bit MAC_out to the 18-bit old partial sum. The adder 1904 may first perform a 9-bit addition between the 9-bit LSBs of MAC_out and MACreg_in, and then propagate the carry (if any) to the adder 1902 for addition between the MSB of MAC_out and the MSB of MACreg_in to generate the 18-bit MACreg_out. The addition / summation may be a saturated addition, and may be represented as:

[0133] MACreg_out[17:0]=Saturate((MACreg_in[17:0]+MAC_out[12:0]), 18 bits) (Equation 2)

[0134] On the other hand, if the MAC engine 1300 operates in binary mode, each of N0a, N0b, N0c, and N0d and each of N1a, N1b, N1c, and N1d can obtain a maximum value of 4, and MAC_L and MAC_R can each obtain a maximum value of 16. Adders 1902 and 1904 can be two separate 9-bit adders. Adder 1902 can generate 9-bit MSBs of MACreg_out by summing 9-bit MSBs of MACreg_in and 6-bit LSBs of MAC_L. Adder 1904 can generate 9-bit LSBs of MACreg_out by summing 9-bit LSBs of MACreg_in and 6-bit LSBs of MAC_R. Because there is no carry from adder 1904 to adder 1902, adders 1902 and 1904 can perform addition in parallel and speed up the update of partial sums.

[0135] Fig. 20A and 20B It shows that Fig.18 Schematic diagram of the MAC engine 1300 of the computing unit 1800. Fig. 20A and 20B, the MAC engine 1300 includes calculation units 1800a, 1800b, 1800c, and 1800d. The MAC engine 1300 has first, second, third, and fourth MAC data inputs for receiving 8-bit input data IN0, IN1, IN2, and IN3 from the input data multiplexer 1306, respectively. The first calculation data input of each calculation unit is coupled to the first MAC data input (to receive IN0). The second calculation data input of each calculation unit is coupled to the second MAC data input (to receive IN1). The third calculation data input of each calculation unit is coupled to the third MAC data input (to receive IN2). The fourth calculation data input of each calculation unit is coupled to the fourth MAC data input (to receive IN3). The input data multiplexer 1306 selectively forwards input data from one of the Din0 or Din1 input data registers, and according to the D4 or D8 mode, all bits (e.g., 32 bits, including four 8-bit input data) stored in the input data register can be forwarded or half of the bits (16 bits) can be copied. For example, in D8 mode, the input data multiplexer 1306 may forward X0 (associated with input channel 0) as IN0, forward X1 (associated with input channel 1) as IN1, forward X2 (associated with input channel 2) as IN2, and forward X3 (associated with input channel 3) as IN3. In D4 mode, the input data multiplexer 1306 may forward a copy of the lower half (LSB) of the X2 bits as IN0, forward a copy of the upper half (MSB) of the X2 bits as IN1, forward a copy of the lower half of the X3 bits as IN2, and forward a copy of the upper half of the X3 bits as IN3. It may also forward a copy of the lower half of the X0 bits as IN0, forward a copy of the upper half of the X0 bits as IN1, forward a copy of the lower half of the X1 bits as IN2, and forward a copy of the upper half of the X1 bits as IN3.

[0136] In addition, the MAC engine 1300 includes a group of first, second, third, and fourth MAC weight inputs. Each computing unit is coupled to a respective group of first, second, third, and fourth MAC weight inputs, wherein a first calculation weight input is coupled to a first MAC weight input of the group, a second calculation weight input is coupled to a second MAC weight input of the group, a third calculation weight input is coupled to a third MAC weight input of the group, and a fourth calculation weight input is coupled to a fourth MAC weight input of the group. Each computing unit can receive four 4-bit weights from the weight multiplexer 1304. For example, computing unit 1800a receives 4-bit weights W0a, W1a, W2a, and W3a at the first, second, third, and fourth calculation weight inputs of computing unit 1800a, respectively. Computing unit 1800b receives 4-bit weights W0b, W1b, W2b, and W3b at the first, second, third, and fourth calculation weight inputs of computing unit 1800b, respectively. Calculation unit 1800c receives 4-bit weights W0c, W1c, W2c, and W3c at the first, second, third, and fourth calculation weight inputs of calculation unit 1800c, respectively. In addition, calculation unit 1800d receives 4-bit weights W0d, W1d, W2d, and W3d at the first, second, third, and fourth calculation weight inputs of calculation unit 1800d, respectively. In FIG. 20, the first group of the first, second, third, and fourth MAC weight inputs may receive W0a, W1a, W2a, and W3a, respectively. The second group of the first, second, third, and fourth MAC weight inputs may receive W0b, W1b, W2b, and W3b, respectively. The third group of the first, second, third, and fourth MAC weight inputs may receive W0c, W1c, W2c, and W3c, respectively. The fourth group of the first, second, third, and fourth MAC weight inputs may receive W0d, W1d, W2d, and W3d, respectively.

[0137] The weight reuser 1304 may selectively forward weights from one of the weight-0 register or the weight-1 register. In D4 mode, computation units 1800a and 1800b may receive 32 upper half bits of the selected weight register, and computation units 1800c and 1800d may receive 32 lower half bits of the selected register. In D8 mode, computation units 1800a and 1800b may receive a copy of half of the upper half bits (16 bits) of the selected weight register, and computation units 1800c and 1800d may receive a copy of half of the lower half bits (16 bits) of the selected weight register. Computation units 1800a-d may store intermediate output data elements in one MAC register, wherein each intermediate output data element may have 18 bits.

[0138] Fig. 20A and 20BAlso shown is a data merge circuit 2002, which can be part of the MAC engine 1300 or the post-processing engine 1302. As will be described below, for certain input and weight precisions, the outputs of some or all of the computation units 1800a-d can be merged to generate a particular intermediate output data element. The data merge circuit 2002 includes bit shifters 2004a, 2004b, and 2006. The bit shifter 2004a is configured to perform a two-bit left shift on the 18-bit output of the computation unit 1800a, and the bit shifter 2004b is configured to perform a two-bit left shift on the 18-bit output of the computation unit 1800c. The data merge circuit 2002 also includes adders 2008a, 2008b, and 2030. The adder 2008a can add a left-shifted version of the output of the computation unit 1800a to the output of the computation unit 1800b to generate a partial sum 2020a. The adder 2008b may add a left-shifted version of the output of the computation unit 1800c to the output of the computation unit 1800d to generate the partial sum 2020b. The bit shifter 2006 is configured to perform a four-bit left shift on the partial sum 2020a. The data merge circuit 2002 further includes an adder 2030 for generating a partial sum 2032 by adding a left-shifted version of the partial sum 2020a to the partial sum 2020b. The MAC register multiplexer 1303 may receive the 18-bit outputs of the computation units 1800a-1800d, the partial sums 2020a, 2020b, and 2032, and select one of them to be stored in the MAC register based on the input weight precision, as will be described below.

[0139] 21A-21F also illustrate different configurations of the computation engines 1800a-d, data merging circuit 2002, weight multiplexer 1304, input data multiplexer 1303, and MAC register multiplexer 1303 to support MAC operations involving different input and weight precisions.

[0140] Figure 21A-1 and 21A-2An example configuration is shown in which each input data element is an 8-bit number and each weight element is a 2-bit number. The input data multiplexer 1306 can operate in D8 mode and provide 32-bit input data containing four 8-bit input data elements in each cycle. The 2-bit weight element can be a ternary weight (-1, 0, 1), or extended from a 1-bit binary weight (e.g., +1 is mapped to bit 01, -1 is mapped to bit 11). In addition, the weight multiplexer 1304 can operate in D8 mode, where it copies half of the stored weight elements in the weight register to provide 32-bit weights (including 16x2-bit weight elements). The computing unit 1800a can receive four 8-bit input data elements at the first, second, third and fourth calculation data inputs, and four copies of the 2-bit weight elements at the first, second, third and fourth weight inputs. The computing unit 1800a can calculate a partial sum of one of the intermediate output data elements (e.g., Y0), as follows:

[0141] Y0=(W0*X0 H +W1*X1 H +W2*X2 H +W3*X3 H )<<4+(W0*X0 L +W1*X1 L +W2*X2 L +W3*X3 L )(Equation 3)

[0142] In Equation 3, each of W0, W1, W2, and W3 is a 2-bit weight element. X0, X1, X2, and X3 are each 8-bit input data elements. H 、X1 H , X2 H and X3 H The 4 MSBs of X0, X1, X2, and X3 (also represented as D[7:4] bits), respectively, while X0 L 、X1 L , X2 L and X3 L The 4 LSBs (also represented as D[3:0] bits) of X0, X1, X2, and X3, respectively. Multiplier circuit 1700a (of computation unit 1800a) generates W0*X0 H As N0a and generate W0*X0 L As N0b, multiplier circuit 1700b generates W1*X1 H As N1a and generate W1*X1 L As N1b, multiplier circuit 1700c generates W2*X2 H As N2a and generate W2*X2 LAs N2b, and multiplier circuit 1700d generates W3*X3 H As N3a and generate W3*X3 L As N3b. Shift control 1808 can control bit shifter circuit 1806a to shift MAC_L to the left by 4 bits. MAC register multiplexer 1303 can selectively connect the output of accumulator 1810a to the MAC register and bypass data merge circuit 2002.

[0143] Figure 21B-1 and 21B-2 An example configuration is shown in which each input data element is a 4-bit number and each weight element is a 4-bit number. The input data multiplexer 1306 can operate in D4 mode and provide 16-bit input data containing four 4-bit input data elements (e.g., by copying half of the bits in the Din0 / Din1 register) in each cycle. In addition, the weight multiplexer 1304 can also operate in D4 mode, where it provides all stored weight elements in the weight register to provide 64-bit weights (including 16x4-bit weight elements). The computing unit 1800a can receive a copy of 4-bit X0, X1, X2 X3 at the first, second, third and fourth calculation data inputs (each represented as D[3:0] bits), and four 4-bit weight elements at the first, second, third and fourth calculation weight inputs. The computing unit 1800a can calculate the partial sum of Y0, as follows:

[0144] Y0=(W0 H *X0+W1 H *X1+W2 H *X2+W3 H *X3)<<2+(W0 L *X0+W1 L *X1+W2 L *X2+W3 L *X3)(Equation 4)

[0145] In Equation 4, each of W0, W1, W2, and W3 is a 4-bit weight element. X0, X1, X2, and X3 are each 4-bit input data elements. H 、W1 H 、W2 H and W3 H They are the 2 MSBs of W0, W1, W2, and W3, and W0 L 、W1 L 、W2 L and W3 LThe 2-bit LSBs of W0, W1, W2, and W3 are respectively. Multiplier circuit 1700a receives X0 and X4 (from input data multiplexer 1306 operating in D4 mode) as 8-bit inputs and W0 as 4-bit input, and generates X0*W0 H As N0a and generate X4*W0 L As N0b. In addition, multiplier circuit 1700b generates X1*W1 H As Nla and generate X5*W1 L As N1b, multiplier circuit 1700c generates X2*W2 H As N2a and generate X6*W2 L As N2b, and multiplier circuit 1700d generates X3*W3 H As N3a and generate X7*W3 L As N3b. Shift control 1808 can control bit shifter circuit 1806a to shift MAC_L to the left by 2 bits. MAC register multiplexer 1303 can selectively connect the output of accumulator 1810a to the MAC register and bypass data merge circuit 2002.

[0146] In subsequent loops (in Figure 21B-1 and 21B-2 ), computing unit 1800a may receive copies of 4-bit X4, X5, X6, X7 and four 4-bit weight elements W4, W5, W6 and W7 at the first, second, third and fourth computational weight inputs, compute a partial sum based on equation 2, and add the partial sum to Y0.

[0147] Figure 21C-1 and 21C-2 An example configuration is shown in which each input data element is a 4-bit number and each weight element is a 2-bit number. Input data multiplexer 1306 can operate in D8 mode and provide 32-bit input data comprising eight 4-bit input data elements in each cycle. In addition, weight multiplexer 1304 also can operate in D4 mode, wherein it provides all stored weight elements in weight register to provide 64-bit weight (comprising 32x2-bit weight elements). Computing unit 1800a can receive eight 2-bit weight elements, wherein the first, second, third and fourth weight input places have two 2-bit weight elements respectively. Computing unit 1800a can also receive eight 4-bit input data elements, wherein the first, second, third and fourth input data input places have two 4-bit input data elements respectively. Computing unit 1800a can calculate the partial sum of Y0, as follows:

[0148] Y0=(W0*X0+W2*X2+W4*X4+W6*X6)+(W1*X1+W3*X3+W5*X5+W7*X7) (Equation 5)

[0149] In Equation 5, each of W0, W1, W2, and W3 is a 2-bit weight element. X0, X1, X2, X3, X4, X5, X6, and X7 are each 4-bit input data elements (and represented as D[3:0] bits). Multiplier circuit 1700a receives X0 and X1 as 8-bit inputs and receives A0 and A1 as 4-bit inputs, and generates A0*X0 as N0a and generates A1*X1 as N0b. In addition, multiplier circuit 1700b generates A2*X2 as N1a and generates A3*X3 as N1b, multiplier circuit 1700c generates A4*X4 as N2a and generates A5*X5 as N2b, and multiplier circuit 1700d generates A6*X6 as N3a and generates A7*X7 as N3b. Shift control 1808 can control bit shifter circuit 1806a not to shift MAC_L to the left. Figure 21C-1 and 21C-2 Omitted. The MAC register multiplexer 1303 may selectively connect the output of the accumulator 1810a to the MAC register and bypass the data merging circuit 2002.

[0150] Figure 21D-1 and 21D-2 An example configuration is shown in which each input data element is an 8-bit number and each weight element is a 4-bit number. Input data multiplexer 1306 can operate in D8 mode and provide 32-bit input data containing four 8-bit input data elements in each cycle. In addition, weight multiplexer 1304 can also operate in D8 mode, where it copies half of the stored weight elements of the weight register to provide 64-bit weights (including replicas of 8x4-bit weight elements). Each of computing units 1800 can receive replicas of 2 bits in four groups of 4-bit weight elements and four 8-bit input data elements. Computing units 1800a, 1800b, 1800c, and 1800d can calculate the partial sum of Y0 and Y1, as follows:

[0151] Y0 H =(W0 H *X0 H +W1 H *X1 H +W2 H *X2 H +W3 H *X3 H )<<4+(W0 H *X0 L +W1 H *X1 L +W2 H *X2 L +W3 H*X3 L )(Equation 6)

[0152] Y0 L = (W0 L *X0 H +W1 L *X1 H +W2 L *X2 H +W3 L *X3 H )<<4+(W0 L *X0 L +W1 L *X1 L +W2 L *X2 L +W3 L *X3 L )(Equation 7)

[0153] Y0 = Y0 H <<2+Y0 L (Equation 8)

[0154] Y1 H = (W4 H *X0 H +W5 H *X1 H +W6 H *X2 H +W7 H *X3 H )<<4+(W4 H *X0 L +W5 H *

[0155] X1 L +W6 H *X2 L +W7 H *X3 L )(Equation 9)

[0156] Y1 L = (W4 L *X0 H +W5 L *X1 H +W6 L *X2 H +W7 L *X3 H )<<4+(W4 L *X0 L +W5 L *X1 L +W6 L *X2L +W7 L *X3 L )(Equation 10)

[0157] Y1=Y1 H <<2+Y1 L (Equation 11)

[0158] In equations 6-11, each of W0, W1, W2, W3, W4, W5, W6, and W7 is a 4-bit weight element. X0, X1, X2, and X3 are each 8-bit input data elements. H 、W1 H 、W2 H 、W3 H 、W4 H 、W5 H 、W6 H 、W7 H They are the 2 MSBs of W0, W1, W2, W3, W4, W5, W6, and W7, where W0 L 、W1 L 、W2 L 、W3 L 、W4 L 、W5 L 、W6 L 、W7 L These are the 2 LSBs of W0, W1, W2, W3, W4, W5, W6, and W7. In addition, X0 H 、X1 H , X2 H and X3 H The 4 MSBs of X0, X1, X2, and X3 (also represented as D[7:4] bits) are respectively L 、X1 L , X2 L and X3 L These are the 4 LSBs of X0, X1, X2, and X3 (also represented as D[3:0] bits).

[0159] Calculate Y0 H and Y0 L Multiple computing units 1800 may be involved. For example, a first computing weight input of computing unit 1800a may receive W0 H The second calculation weight input of the calculation unit 1800a can receive W1 H The third calculation weight input of the calculation unit 1800a can receive W2 H A copy of , and the fourth calculation weight input of the calculation unit 1800a can receive W3 HIn addition, the first calculation weight input of the calculation unit 1800b can receive W0 L The second calculation weight input of the calculation unit 1800b can receive W1 L The third calculation weight input of the calculation unit 1800b can receive W2 L A copy of , and the fourth calculation weight input of the calculation unit 1800b can receive W3 L A replica of .

[0160] In addition, the first calculation weight input of the calculation unit 1800c can receive W4 H The second calculation weight input of the calculation unit 1800c can receive W5 H The third calculation weight input of the calculation unit 1800c can receive W6 H A copy of the calculation unit 1800c, and the fourth calculation weight input of the calculation unit 1800c can receive W7 H The first calculation weight input of the calculation unit 1800d can receive W0 L The second calculation weight input of the calculation unit 1800d can receive W1 L The third calculation weight input of the calculation unit 1800d can receive W2 L A copy of the calculation unit 1800d, and the fourth calculation weight input of the calculation unit 1800d can receive W3 L A first calculation data input of each of the calculation units 1800a-1800d may receive X0, a second calculation data input of each of the calculation units 1800a-1800d may receive X1, a third calculation data input of each of the calculation units 1800a-1800d may receive X2, and a fourth calculation data input of each of the calculation units 1800a-1800d may receive X3.

[0161] The calculation unit 1800a can calculate W0 H *X0 H +W1 H *X1 H +W2 H *X2 H +W3 H *X3 H and W0 H *X0 L +W1 H *X1 L , and the computing unit 1800b can calculate W0 L *X0 H +W1 L *X1 Hand W0 L *X0 L +W1 L *X1 L +W2 H *X2 L +W3 H *X3 L The bit shifters 1806a and 1806b of each computational unit can perform a left shift of four bits. H The partial sum of Y0 is stored in the first MAC register (eg, MACreg0), and Y0 L The partial sum of may be stored in a second MAC register (eg, MACreg1). In addition, in the same cycle, the computing unit 1800c may calculate W4 H *X0 H +W5 H *X1 H +W6 H *X2 H +W7 H *X3 H and W4 H *X0 L +W5 H *X1 L +W6 H *X2 L +W7 H *X3 L , and the computing unit 1800d can calculate W4 L *X0 H +W5 L *X1 H +W6 L *X2 H +W7 L *X3 H and W4 L *X0 L +W5 L *X1 L +W6 L *X2 L +W7 L *X3 L , to generate Y1 H and Y1 L Partial sum. Y1 H and Y1 L The partial sum of Y0 and Y1 can be stored in the third MAC register (eg, MACreg2) and the fourth MAC register (eg, MACreg3). Therefore, two partial sums of outputs (Y0 and Y1) can be generated in each cycle.

[0162] Figure 21E-1 and21E-2 An example configuration is shown in which each input data element is 8 bits and each weight element is 8 bits. The input data multiplexer 1306 can operate in D8 mode and provide 32-bit input data including four 8-bit input data elements in each cycle. In addition, the weight multiplexer 1304 can also operate in D8 mode, where it copies half of the stored weight elements in the weight register to provide 64-bit weights (including copies of 4x8-bit weight elements). Each computing unit 1800 can receive a copy of 2 bits of four groups of 8-bit weight elements and four 8-bit input data elements. The computing units 1800a-1800d can calculate the partial sum of Y0 as follows:

[0163] Y0 HH =(W0 HH *X0 H +W1 HH *X1 H +W2 HH *X2 H +W3 HH *X3 H )<<4+(W0 HH *X0 L +W1 HH *X1 L +W2 HH *X2 L +W3 HH *X3 L )(Equation 12)

[0164] Y0 HL =(W0 HL *X0 H +W1 HL *X1 H +W2 HL *X2 H +W3 HL *X3 H )<<4+(W0 HL *X0 L +W1 HL *X1 L +W2 HL *X2 L +W3 HL *X3 L )(Equation 13)

[0165] Y0 LH =(W0 LH *X0 H +W1 LH *X1 H +W2 LH *X2 H +W3LH *X3 H )<<4+(W0 LH *X0 L +W1 LH *X1 L +W2 LH *X2 L +W3 LH *X3 L )(Equation 14)

[0166] Y0 LL =(W0 LL *X0 H +W1 LL *X1 H +W2 LL *X2 H +W3 LL *X3 H )<<4+(W0 LL *X0 L +W1 LL *X1 L +W2 LL *X2 L +W3 LL *X3 L )(Equation 15)

[0167] Y0=Y0 HH <<6+Y0 HL <<4+Y0 LH <<2+Y0 LL (Equation 16)

[0168] In equations 12-16, W0 HH 、W1 HH 、W2 HH and W3 HH W0, W1, W2, and W3 bits [7:6], W0 HL 、W1 HL 、W2 HL and W3 HL W0, W1, W2, and W3 bits [5:4], W0 LH 、W1 LH 、W2 LH and W3 LH are bits [3:2] of W0, W1, W2, and W3, and W0 LL 、W1 LL 、W2 LL and W3 LL These are bits [1:0] of W0, W1, W2, and W3. In addition, X0 H 、X1 H , X2H and X3 H The 4 MSBs of X0, X1, X2, and X3 (also represented as D[7:4] bits), respectively, while X0 L 、X1 L , X2 L and X3 L The 4 LSBs of X0, X1, X2, and X3 (also represented as D[3:0] bits). In addition, Y0 HH 、Y0 HL 、Y0 LH and Y0 LL They are Bits [7:6], [5:4], [3:2], and [1:0] of Y0 respectively.

[0169] Calculate Y0 HH 、Y0 HL 、Y0 LH and Y0 LL Multiple computing units 1800 may be involved. For example, the first computing weight input of computing unit 1800a may receive W0 HH The second calculation weight input of the calculation unit 1800a can receive W1 HH The third calculation weight input of the calculation unit 1800a can receive W2 HH A copy of , and the fourth calculation weight input of the calculation unit 1800a can receive W3 HH In addition, the first calculation weight input of the calculation unit 1800b can receive W0 HL The second calculation weight input of the calculation unit 1800b can receive W1 HL The third calculation weight input of the calculation unit 1800b can receive W2 HL A copy of , and the fourth calculation weight input of the calculation unit 1800b can receive W3 HL A replica of .

[0170] In addition, the first calculation weight input of the calculation unit 1800c can receive W0 LH The second calculation weight input of the calculation unit 1800c can receive W1 LH The third calculation weight input of the calculation unit 1800c can receive W2 LH A copy of , and the fourth calculation weight input of the calculation unit 1800c can receive W3 LH The first calculation weight input of the calculation unit 1800d can receive W0 LL The second calculation weight input of the calculation unit 1800d can receive W1 LLThe third calculation weight input of the calculation unit 1800d can receive W2 LL A copy of the calculation unit 1800d, and the fourth calculation weight input of the calculation unit 1800d can receive W3 LL A first calculation data input of each of the calculation units 1800a-1800d may receive X0, a second calculation data input of each of the calculation units 1800a-1800d may receive X1, a third calculation data input of each of the calculation units 1800a-1800d may receive X2, and a fourth calculation data input of each of the calculation units 1800a-1800d may receive X3.

[0171] The calculation unit 1800a can calculate W0 HH *X0 H +W1 HH *X1 H +W2 HH *X2 H +W3 HH *X3 H and W0 HH *X0 L +W1 HH *X1 L +W2 HH *X2 L +W3 HH *X3 L , the computing unit 1800b can calculate W0 HL *X0 H +W1 HL *X1 H +W2 HL *X2 H +W3 HL *X3 H and

[0172] W0 HL *X0 L +W1 HL *X1 L +W2 HL *X2 L +W3 HL *X3 L , the computing unit 1800c can calculate

[0173] W0 LH *X0 H +W1 LH *X1 H +W2 LH *X2 H +W3 LH *X3 H and

[0174] W0 LH *X0 L +W1 LH *X1 L +W2 LH *X2 L +W3 LH *X3 L , and the computing unit 1800d can calculate

[0175] W0 LL *X0 H +W1 LL *X1 H +W2 LL *X2 H +W3 LL *X3 H and

[0176] W0 LL *X0 L +W1 LL *X1 L +W2 LL *X2 L +W3 LL *X3 L The bit shifters 1806a-d of each computational unit can perform a left shift of four bits. HH The first partial sum may be stored in a first MAC register (eg, MACreg0), Y0 HL The partial sum may be stored in a second MAC register (eg, MACreg1), Y0 LH The partial sum of can be stored in a third MAC register (eg, MACreg2), and Y0 LL The partial sum of can be stored in the fourth MAC register (eg, MACreg3). The calculation unit 1800 can calculate Y0 HH is a signed number, and calculates Y0 HL 、Y0 LH and Y0 LL An unsigned number.

[0177] Figures 21F, 21G, and 21H illustrate example configurations that support depthwise convolution operations. As described above, for depthwise convolution operations, intermediate output data elements may be generated from input data elements of the same channel. To support depthwise convolution operations, weight multiplexer 1304 may selectively forward zeros to certain multipliers assigned to perform multiplication operations on input data elements of different channels, such that the output of such multipliers is zero and does not affect the intermediate output data elements.

[0178] Figure 21F-1 and 21F-2 An example configuration supporting a deep convolution operation between 8-bit input data elements and 8-bit weight elements is shown. To support the deep convolution operation, the computation units 1800a-d may receive non-zero weights (e.g., W0 for a particular channel) from the weight multiplexer 1304. HH 、W0 HL 、W0 LH 、W0 LL ) and zero weights for other channels. With the configuration of FIG. 21F , the computing units 1800a-d and the data merging circuit 2002 can calculate Y0 as follows:

[0179] Y0 HH =(W0 HH *X0 H )<<4+(W0 HH *X0 L )(Equation 17)

[0180] Y0 HL =(W0 HL *X0 H )<<4+(W0 HL *X0 L )(Equation 18)

[0181] Y0 LH =(W0 LH *X0 H )<<4+(W0 LH *X0 L )(Equation 19)

[0182] Y0 LL =(W0 LL *X0 H )<<4+(W0 LL *X0 L )(Equation 20)

[0183] Y0=Y0 HH <<6+Y0 HL <<4+Y0 LH <<2+Y0 LL (Equation 21)

[0184] Figure 21G-1 and 21G-2 An example configuration supporting a depthwise convolution operation between 8-bit input data elements and 2-bit weight elements is shown. Figure 21G-1 and 21G-221G , computing unit 1800a can be assigned to calculate (first channel) Y0, and receive the non-zero weight (W0) of only the first channel under the first calculation weight input and the zero weight of other channels under other calculation weight inputs. Computing unit 1800b can be assigned to calculate (second channel) Y1, and receive the non-zero weight (W1) of only the second channel under the second calculation weight input and the zero weight of other channels under other calculation weight inputs. In addition, computing unit 1800c can be assigned to calculate (third channel) Y2, and receive the non-zero weight (W2) of only the third channel under the third calculation weight input and the zero weight of other channels under other calculation weight inputs. In addition, computing unit 1800d can be assigned to calculate (fourth channel) Y3, and receive the non-zero weight (W3) of only the fourth channel under the fourth calculation weight input and the zero weight of other channels under other calculation weight inputs. Through the configuration of Figure 21G, computing units 1800a-d can calculate Y0, Y1, Y2 and Y3 as follows:

[0185] Y0=(W0*X0 H )<<4+(W0*X0 L )(Equation 22)

[0186] Y1=(W1*X1 H )<<4+(W1*X1 L )(Equation 23)

[0187] Y2=(W2*X2 H )<<4+(W2*X2 L ) (Equation 24)

[0188] Y3=(W3*X2 H )<<4+(W3*X3 L ) (Equation 25)

[0189] Figure 21H-1 and 21H-2 An example configuration supporting a deep convolution operation between 8-bit input data elements and 4-bit weight elements is shown. In Figure 21H, computing units 1800a and 1800b can be assigned to calculate Y0 (of the first channel), and receive the non-zero weight (W0) of only the first channel under the first calculation weight input and the zero weight of other channels under other calculation weight inputs. Computing units 1800c and 1800d can be assigned to calculate Y1 (of the second channel), and receive the non-zero weight (W1) of only the second channel under the second calculation weight input and the zero weight of other channels under other calculation weight inputs. Through the configuration of Figure 21H, computing units 1800a-d and data merging circuit 2002 can calculate Y0 and Y1 as follows:

[0190] Y0 H =(W0H *X0 H )<<4+(W0 H *X0 L ) (Equation 26)

[0191] Y0 L =(W0 L *X0 H )<<4+(W0 L *X0 L ) (Equation 27)

[0192] Y1 H =(W1 H *X1 H )<<4+(W1 H *X1 L ) (Equation 28)

[0193] Y1 L =(W1 L *X1 H )<<4+(W1 L *X1 L ) (Equation 29)

[0194] Y0=Y0 H <<2+Y0 L (Equation 30)

[0195] Y1=Y1 H <<2+Y1 L (Equation 31)

[0196] In a subsequent loop (not shown in the figure), the calculation units 1800a and 1800b may receive the non-zero weight (W2) of the third channel under the third calculation weight input and the zero weights of the other channels under the other calculation weight inputs, and the calculation units 1800c and 1800d may receive the non-zero weight (W3) of the fourth channel under the fourth calculation weight input and the zero weights of the other channels under the other calculation weight inputs, and calculate Y2 and Y3 as follows:

[0197] Y2 H =(W2 H *X2 H )<<4+(W2 H *X2 L ) (Equation 32)

[0198] Y2 L =(W2 L *X2 H )<<4+(W2 L *X2 L ) (Equation 33)

[0199] Y3 H =(W3 H *X3 H )<<4+(W3 H *X3 L ) (Equation 34)

[0200] Y3 L =(W3 L *X3 H )<<4+(W3 L *X3 L ) (Equation 35)

[0201] Y2=Y2 H <<2+Y2 L (Equation 36)

[0202] Y3=Y3 H <<2+Y3 L (Equation 37)

[0203] Fig.22A , 22B 22C show examples of internal components of the MAC register multiplexer 1303, input data multiplexer 1306, and weight multiplexer 1304. Fig.22A 13 shows example internal components of the MAC register multiplexer 1303. Fig.22A, the MAC register multiplexer 1303 may include a demultiplexer circuit 2200 and a multiplexer circuit 2202. The demultiplexer circuit 2200 is coupled between the outputs of a group of computing units 1800 (a plurality of MACreg_out[17:0], for example, four MACreg_outs, eight MACreg_outs, etc.) and the inputs of the MAC registers, and the multiplexer circuit 2202 is coupled between the outputs of the MAC registers and the inputs of the group of computing units 1800 (a plurality of MACreg_in[17:0]). The demultiplexer circuit 2200 may be controlled by the computing controller 522 (e.g., based on a sub-instruction) to connect the outputs of the computing units 1800 to the inputs of the MAC registers so that the computing units 1800 store the partial sums generated by the MAC registers, or store the bias values ​​(Bias0, Bias1, Bias2, and Bias3) in some of the MAC registers. The bias values ​​may be added to the partial sums as part of a post-processing operation. The computation controller 522 may control the demultiplexer circuit 2200 to store the bias value in the MAC register in response to the first sub-instruction to perform the neural network layer computation. By introducing the bias value into the MAC register as part of the initialization operation, the delay caused by the post-processing engine 1302 adding the bias as part of the post-processing operation may be avoided. In addition, the multiplexer 2202 may be controlled by the computation controller 522 based on the sub-instruction that selects which of the MAC registers provides the previous partial sum for updating by a particular computation unit 1800.

[0204] Fig. 22B Example internal components of the input data multiplexer 1306 are shown. Fig. 22B, the input data multiplexer 1306 may have inputs coupled to a first input data register (Din0) and a second input data register (Din1), and an output coupled to the MAC engine 1300 to provide four input data IN0, IN1, IN2, and IN3. The input data multiplexer 1306 further includes multiplexers 2210, 2212, a replica circuit 2214, and a multiplexer 2216. The multiplexers 2210, 2212, and 2216 are controlled by control signals M0, M1, and M2, some of which may be derived from the input data and weight configuration 1310, and some of which may be provided by the computation controller 522. For example, the multiplexer 2210 may be controlled by the control signal M0 to selectively provide data from one of the registers Din0 or Din1 as IN0, IN1, IN2, and IN3. In D4 mode, multiplexer 2212 can select 8 bits from the upper 16 bits (bits 31-16) or the lower 16 bits (bits 15-0) of the selected register for bit replication. Replication circuit 2214 splits the selected 8-bit value into two 4-bit values ​​and generates two 8-bit values ​​by replicating for each 4-bit value. Multiplexer 2216 can be controlled by control signal M2 set based on D4 or D8 mode to forward the selected 8-bit value as IN0, IN1, IN2 and IN3 (in D8 mode), or forward 8-bit values ​​each containing a corresponding replicated 4-bit value as IN0, IN1, IN2 and IN3.

[0205] Fig. 22C Example internal components of the weight multiplexer 1304 are shown. Fig. 22C, the weight multiplexer 1304 may have inputs coupled to a first weight register (weight-0 register) and a second weight register (weight-1 register), and four weight input groups coupled to the MAC engine 1300 to provide outputs of a first group weight W0a, W1a, W2a, and W3a, a second group weight W0b, W1b, W2b, and W3b, a third group weight W0c, W1c, W2c, and W3c, and a fourth group weight W0d, W1d, W2d, and W3d. The weight multiplexer 1304 also includes multiplexers 2220, 2222, a replica circuit 2224, and a multiplexer 2226. The multiplexers 2220, 2222, and 2226 are controlled by control signals M0, M1, and M2, some of which may be derived from the input data and the weight configuration 1310, and some of which may be provided by the computation controller 522. For example, multiplexer 2210 can be controlled by control signal M0 to selectively provide 64-bit data from one of weight-0 register or weight-1 register as weight group. In D8 mode, multiplexer 2222 can select 16 bits for bit replication from the upper 32 bits (one of Wt1H or Wt0H) or the lower 32 bits (one of Wt1L or Wt0L) of the selected register. Copy circuit 2224 divides the selected 32-bit value into two 16-bit values ​​and generates two 32-bit values ​​by replicating for each 16-bit value. Multiplexer 2226 can be controlled by control signal M2 set based on D4 or D8 mode to forward 64-bit value from the selected register as weight group in D4 mode, or forward two 32-bit values ​​from the replica of 16-bit value as weight group in D8 mode.

[0206] In addition, the weight multiplexer 1304 also includes depth and average pooling logic 2228. When enabled by a depth convolution enable signal (e.g., from configuration 1311), logic 2228 can select one of the 8 bits from one of the 64-bit weight-0 or weight-1 registers, split the 8 bits into 4 groups of 2 bits, fill the weight groups with zeros, and send the weight groups to calculation units 1800a, 1800b, 1800c, and 1800d. In addition, depending on the weight accuracy from the input data and weight configuration 1310, the weight multiplexer 1304 can fill the different weights in each group with zeros, as shown in Figures 21F-21H. In addition, when enabled by an average pooling enable signal (e.g., from configuration 1311), the weight multiplexer 1304 can provide copies of weight elements W0, W1, W2, and W3 in the groups of the first, second, third, and fourth weight elements, as shown in Figures 21F-21H. Figure 21G-1 and Figure 21G-2 , where each of W0, W1, W2 and W3 is equal to logic one.

[0207] Fig.23Example internal components of post-processing engine 1302 are shown. Fig.23 , the post-processing engine 1302 includes a post-processing circuit 2302 and a data packing circuit 2304. The post-processing circuit 2302 can receive intermediate output data of different channels, such as Y0, Y1, Y2 and Y3, from the MAC register. The post-processing circuit 2302 includes circuits, such as arithmetic circuits and various logic circuits, for performing post-processing operations (e.g., BatchNorm, residual layer processing, etc.) on the intermediate output data in response to a control signal 1305 from the computing controller 522, and the computing controller generates a control signal in response to the sub-instruction 602 indicating the post-processing operation. The processing circuit 2302 also receives a precision configuration 1312 indicating the precision of the input, output and weight, and a post-processing parameter 1314 defining various parameters of the post-processing operation including the output data bit size / precision, and configures the post-processing operation based on the precision configuration 1312 and the post-processing parameter 1314. In some examples, the processing circuit 2302 may perform multiple post-processing operations on the intermediate output data Y0, Y1, Y2, and Y3 in parallel in one clock cycle to generate output data Out0, Out1, Out2, and Out3. In some examples, the processing circuit 2302 may include Fig. 20A and 20B The data merging circuit 2002 is used to provide intermediate output data elements of different input and weight precisions, as described above.

[0208] The data packing circuit 2304 may store output data, such as Out0, Out1, Out2, and Out3, at the DOUT register. In some examples, the DOUT register has 32 bits. The data packing circuit 2304 is configured to store the output data at a specific bit position of the DOUT register, and in response to the DOUT register being filled with the output data (e.g., storing 32 bits of output data), transmit a control signal 2305 to the load / store controller 530 to retrieve the output data from the DOUT register back to the memory 512, which enables the DOUT register to be overwritten with the new output data.

[0209] Specifically, for a certain set of input, output and weight precision, the data packing circuit 2304 can generate four 8-bit output data elements Out0, Out1, Out2 and Out3 in one clock cycle. The data packing circuit 2304 can store the four 8-bit output data elements in the DOUT register after one clock cycle, and then transmit the control signal 2305 to the load / store controller 530. For different sets of input, output and weight precision, the processing circuit 2302 can generate 16-bit output data per clock cycle, such as four 4-bit output data elements or two 8-bit output data elements. In such cases, the processing circuit 2302 can store the first group of 16-bit output data in the first position of the DOUT register after the first clock cycle, and then store the second group of 16-bit output data in the second position of the DOUT register after the second clock cycle. Such an arrangement allows the load / store controller 530 to transmit output data in blocks of a specific number of bits (e.g., 32 bits) that can be optimized for write operations of the memory 512. Such an arrangement may reduce accesses to memory 512 by the neural network process 502 when writing back output data, which may reduce memory usage and power consumption.

[0210] In some examples, data packing circuit 2304 may also receive control signal 2306 from compute controller 522 and transmit control signal 2305 in response to control signal 2306. Compute controller 522 may generate control signal 2306 in response to an instruction from instruction buffer 520.

[0211] Fig.24 An example of the internal components of the processing circuit 2302 is shown. Fig.24, the processing circuit 2302 may include a series of processing circuits coupled between a data input for receiving intermediate output data elements and a data output for providing output data elements. The processing circuit may include a multiplier circuit 2402, a bit shifter 2404, a saturation circuit 2406, an adder 2408, and a clamp circuit 2410, each of which may include arithmetic circuits (e.g., adders and multipliers) and / or logic circuits (e.g., bit shifter circuits, mapping tables implemented using multiplexers and demultiplexers for saturation and clamp circuits, etc.) for processing the intermediate output data elements and providing output data elements. For example, the processing circuit 2302 may include: a multiplier circuit 2402a, a bit shifter 2404a, a saturation circuit 2406a, an adder 2408a, and a clamp circuit 2410a, which are coupled between a first input of the processing circuit 2302 (e.g., for receiving Y0) and a first output of the processing circuit 2302 (e.g., for providing Out0); a multiplier circuit 2402b, a bit shifter 2404b, a saturation circuit 2406b, an adder 2408b, and a clamp circuit 2410b, which are coupled between a second input of the processing circuit 2302 (e.g., for receiving Y1) and a second output of the processing circuit 2302 (e.g., for providing Out0); ut1); multiplier circuit 2402c, bit shifter 2404c, saturation circuit 2406c, adder 2408c and clamp circuit 2410c, which are coupled between a third input of processing circuit 2302 (e.g., for receiving Y2) and a third output of processing circuit 2302 (e.g., for providing Out2); and multiplier circuit 2402a, bit shifter 2404a, saturation circuit 2406a, adder 2408a and clamp circuit 2410a, which are coupled between a fourth input of processing circuit 2302 (e.g., for receiving Y3) and a fourth output of processing circuit 2302 (e.g., for providing Out3).

[0212] The processing circuit 2302 also includes a data routing circuit 2412 coupled between an input of the processing circuit 2302 and an input of the multiplier circuits 2402a-d, and a data routing circuit 2414 coupled between an output of the clamp circuits 2410a-d and an output of the processing circuit 2302. As will be described below, based on the input, output, and weight precision, the data routing circuit 2412 can route intermediate output data elements to the multiplier circuits, and the data routing circuit 2414 can route the output of the clamp circuits to the output of the processing circuit 2302. In addition, the processing circuit 2302 includes a multiplexer circuit 2416a coupled between the output of the adder 2408a and the input of the adder 2408b, and a multiplexer circuit 2416c coupled between the output of the adder 2408c and the input of the adder 2404d. The multiplexer circuit 2416 can also be controlled based on the weight precision.

[0213] Each multiplier circuit 2402 is coupled to a corresponding input of 2302 with a scaling factor (in Fig.24 The input of the bit shifter 2404 is coupled to the output of the multiplier to perform a right shift of the scaled intermediate output data elements by a shift parameter (in Fig.24 The input of saturation circuit 2406 is coupled to the bit shifter output. If the data element is within -2 15 and (2 15 -1), then the saturation circuit 2406 may forward the right-shifted and scaled intermediate output data element, otherwise if the data element is below -2 15 , then the output lower limit is -2 15 , and if the data element exceeds (2 15 -1), then the output upper limit (2 15 -1). In some examples, the upper and lower limits of the saturation circuit 2406 can be fixed / built-in and non-programmable. The adder 2408 can add a value to the restricted, right-shifted and scaled intermediate output data element. The added value can come from the residual layer output (labeled as in [0], in [1], in [2], and in [3]) that can be obtained from the input data register DIN0 / DIN1 to provide residual layer processing. In some examples, based on the control of the multiplexer circuit 2416, the added value can also come from the output of another adding circuit 2408 in the processing circuit 2302. The output of the adder 2408a is coupled to the input of the clamp circuit 2410. The clamp circuit 2410 can implement a clamp function, which can be based on an activation function rectified linear unit (ReLU). The clamp function can also clamp / limit the output between an upper value (labeled as clamp_high) and a lower value (labeled as clamp_low). Scaling, shifting, clamp_high and clamp_low values, and connections may be defined in the precision configuration 1312 and / or post-processing parameters 1314. For example, different clamp_high and clamp_low values ​​may be set based on the output data precision, as follows:

[0214] Table 3

[0215]

[0216]

[0217] For each intermediate output data element Y_n (e.g., Y0, Y1, Y2, and Y3), the processing circuit 2302 may generate an output data element Out_n (e.g., Out0, Out1, Out2, and Out3) based on the following equation. The scaling and shifting may be different for different channels. In addition, a bias value bias[n] may be introduced in the MAC register to initialize the intermediate output data elements, as described above in FIG. 22 .

[0218] Out_n=clamp(Scale[n]*(Y_n+bias[n])>>shift[n], clamp_high, clamp_low) (Equation 38)

[0219] exist Fig.24 , the scaling values ​​(e.g., scale[0]-scale[3]) and shift values ​​(e.g., shift[0]-shift[3]) may vary between different neural network layers and may be received from weight / parameter registers 529b (and weight and parameter buffers 526). The clamp_high and clamp_low values ​​may be relatively static between different neural network layers and may be received from configuration registers 528d.

[0220] Fig.25A , Fig.25B and Fig.25C 2 is a schematic diagram showing different configurations of processing circuit 2302 for post-processing operations supporting different input and weight precisions. For simplicity, data routing circuits 2412 and 2414 and multiplexer circuits 2416a-c are omitted, and their data routing operations are described in detail in FIG. Fig.25A - shown in C.

[0221] Fig.25A is an example configuration of processing circuit 2302 supporting 8-bit data and 2-ternary (2T) weights, which is also Fig.15D In Fig.25A, data routing circuit 2412 may connect a first input to multiplier circuit 2402a, a second input to multiplier circuit 2402b, a third input to multiplier circuit 2402c, and a fourth input to multiplier circuit 2402d. In addition, to support 2T weight operation, multiplexer circuit 2416a may route the output of adder 2408a to the input of adder 2408b, so that adder 2408b adds the output of saturation circuit 2406a (for Y0) and the output of saturation circuit 2406b (for Y1), and provides the sum to clamp circuit 2410b. In addition, multiplexer circuit 2416c can route the output of adder 2408c to the input of adder 2408d, so that adder 2408d adds the output of saturation circuit 2406c (for Y2) and the output of saturation circuit 2406d (for Y3) and provides the sum to clamp circuit 2410d. Data routing circuit 2414 can route the output of clamp circuit 2410b to the first and third outputs to provide replicas Out0 and Out2, and route the output of clamp circuit 2410d to the second and fourth outputs to provide replicas Out1 and Out3. In some examples, the data packing circuit 2304 may store one of Out0 or Out2 and one of Out1 or Out3 generated by the processing circuit 2302 in a first clock cycle at a first bit position of the DOUT register, then store one of Out0 or Out2 and one of Out1 or Out3 generated by the processing circuit 2302 in a second clock cycle at a second bit position of the DOUT register, and then transmit a control signal 2305 that initiates the acquisition of output data from the DOUT register to the memory 512. In some examples, the data packing circuit 2304 may also store Out0, Out2, Out1, and Out3 generated by the processing circuit 2302 in a clock cycle, and transmit a control signal 2305 that initiates the acquisition of output data from the DOUT register to the memory 512.

[0222] Fig.25B and Fig.25C An example configuration of a processing circuit 2302 including a data merging circuit 2002 that may be part of a data routing circuit 2412 is shown. Fig.25B In FIG. 2 , the data merging circuit 2002 is configured to support 8-bit input data and 4-bit weights, such as Figure 21D-1 and 21D-2The data routing circuit 2412 may route the output of the adder 2008a to the inputs of the multiplier circuits 2402a and 2402c, and route the output of the adder 2008b to the inputs of the multiplier circuits 2402b and 2402d. The processing circuit 2302 may generate Out0 and Out2 as duplicates, and generate Out1 and Out3 as duplicates. The data packing circuit 2304 may store Out0 and Out2 generated in one clock cycle as duplicate data, and store Out1 and Out3 as duplicate data in the DOUT register, or store different groups Out0 / Out2 and Out1 / Out3 from two different clock cycles in the DOUT register, as described above.

[0223] In addition, Fig.25C 21E . Data routing circuit 2412 may route the output of adder 2008a to the inputs of multiplier circuits 2402a and 2402c, and route the output of adder 2030 to the inputs of multiplier circuits 2402a-d. Processing circuit 2302 may generate Out0, Out1, Out2, and Out4 as duplicates. Data packing circuit 2304 may store Out0-Out3 generated in one clock cycle as duplicate data in the DOUT register, or store different Out0 / Out1 / Out2 / Out3 from four different clock cycles in the DOUT register, as described above.

[0224] Figure 26A-1 , 26A-2 26B show an example of the internal components of the max pooling engine 1320 and its operation. The max pooling engine 1320 can compare the input data elements of one of the DIN0 or DIN1 input data registers with the output data elements of the DOUT register, and if a particular output data element in the DOUT register has a lower value than the compared input data element, then replace it. The max pooling engine 1320 can also be bypassed to avoid overwriting the DOUT register, or output zero as the result of the max pooling operation.

[0225] refer to Figure 26A-1 and 26A-2, the maximum pooling engine 1320 may include a multiplexer 2602, a comparison circuit system 2604, and a multiplexer 2606. The multiplexer 2602 may be controlled (e.g., based on the post-processing parameters 1314) to select which of the input data registers DIN0 or DIN1 performs the maximum pooling operation. The comparison circuit system 2604 may receive configuration data via terminals cnfg-hi[1:0] and cnfg-lo[1:0], indicating, for example, whether to perform a 4-bit or 8-bit comparison, a signed or unsigned comparison, etc., and compare the input data (from one of DIN0 or DIN1) and the output data (in DOUT) in 8-bit blocks based on the configuration data (e.g., between DIN[31:24] and DOUT[31:24], between DIN[23:16] and DOUT[23:16], between DIN[15:8] and DOUT[15:8], and between DIN[7:0] and DOUT[7:0]). The configuration data may be part of the post-processing parameters 1314 from the configuration registers 528d.

[0226] For each 8-bit block, comparisons between the 4-bit MSBs may be performed by an MSB comparison circuit, such as comparison circuit 2604a (also labeled CMP-hi in FIGS. 26A and 26B ), and comparisons between the 4-bit LSBs may be performed by an LSB comparison circuit, such as comparison circuit 2604b (also labeled CMP-lo in FIGS. 26A and 26B ). The CMP-hi circuit may also transmit the 4-bit MSB comparison result to the CMP-lo circuit to ensure an 8-bit comparison, and the CMP-lo circuit does not indicate that DOUT is less than DIN when the MSB of DOUT has the same or greater value than the MSB of DIN but the LSB of DOUT has a smaller value than the LSB of DIN. In addition, depending on the comparison result, the multiplexer (e.g., multiplexers 2604c and 2604d) may overwrite the 4-bit MSBs and / or the 4-bit LSBs of the 8-bit block of output data with the input data. In the case of an 8-bit comparison, the multiplexer overwrites the 8-bit DOUT data with the 8-bit DIN data only when the entire 8-bit DOUT data has a lower value than the 8-bit DIN data. In addition, the multiplexer 2606 can provide 8-bit chunks of DOUT data or zeros. On the other hand, in the case of a 4-bit comparison, the multiplexer compares between the 4-bit DOUT data and the 4-bit DIN data, and if the 4-bit DIN data exceeds the 4-bit DOUT data and regardless of the comparison results between the other 4-bit DIN / DOUT data, the 4-bit DOUT data can be replaced.

[0227] Fig.26B 2610 and 2612 are diagrams illustrating example operations of the CMP-hi and CMP-lo circuits. Fig.26B, diagrams 2610 and 2612 illustrate the logical operations of the CMP-hi and CMP-lo circuits, respectively, expressed in the form of instructions that can be synthesized into (and expressed as) logic circuits representing the CMP-hi and CMP-lo circuits.

[0228] Referring to diagram 2610, a CMP-hi circuit (e.g., comparison circuit 2604a) may receive configuration data, such as whether the 4 MSBs of the 8-bit blocks of DIN and DOUT data being compared are signed or unsigned data. If the data is signed, then CMP-hi may sign-extend the 4 MSBs of the DIN and DOUT data, otherwise prefix them with zeros (by adding zeros before the MSBs).

[0229] If the 4-bit MSB of the DIN data has a greater value than the 4-bit MSB of the DOUT data, the CMP-hi circuit can send a control signal to a multiplexer (e.g., multiplexer circuit 2604c) to overwrite the 4-bit MSB of the DOUT data with the 4-bit LSB of the DIN data. The CMP-hi circuit can also provide a first comparison signal to the CMP-lo circuit (setting cmp-res = "GT"). On the other hand, if the 4-bit MSB of the DIN and DOUT data are the same, the CMP-hi circuit can provide a second comparison signal (setting cmp-res = "EQ"). The first and second comparison signals are provided to the CMP-lo circuit to avoid making an erroneous comparison decision in which the MSB of DOUT has a higher value than DIN, but the LSB of DOUT has a lower value than DIN. In both cases where the 4-bit MSB of the DIN data has the same value or a lower value than the DOUT data, the CMP-hi circuit can maintain the 4-bit MSB of the DOUT data in the DOUT register.

[0230] In addition, referring to diagram 2612, the CMP-lo circuit (e.g., comparison circuit 2604b) can receive configuration data, such as whether the 4 LSBs of the 8-bit blocks of DIN and DOUT data being compared are signed or unsigned data, and whether an 8-bit comparison or a 4-bit comparison is being performed. If the data is signed, then the CMP-lo can sign-extend both the 4 LSBs of the DIN and DOUT data, and prefix them with zeros otherwise. In addition, if a 4-bit comparison is being performed, then the CMP-lo circuit can ignore the control signal from the CMP-hi circuit and overwrite the 4 LSBs of the DOUT data with the 4 LSBs of the DIN data if it has a higher value. In addition, if an 8-bit comparison is performed, the CMP-lo circuit may overwrite the 4 LSBs of the DOUT data with the 4 LSBs of the DIN data (using multiplexer circuit 2604d) only if the first control signal indicates that the 4 MSBs of the DOUT data have a higher value than the 4 MSBs of the DIN data, or if the second control signal indicates that the 4 MSBs are equal and the 4 LSBs of the DOUT data have a lower value than the 4 LSBs of the DIN data.

[0231] Fig. 27 Flowchart showing method 2700 of operation of a neural network processor. Method 2700 may be performed by various components of neural network processor 502, such as compute controller 522, load / store controller 530, compute engine 524, etc.

[0232] In operation 2702, the compute controller 522 receives a first instruction from the instruction buffer 520. An example syntax of the first instruction is Fig. 6A and 6B The first instruction may include a sub-instruction 610 instructing to obtain input data elements into input data registers and a sub-instruction 608 instructing to obtain weight elements into weight registers.

[0233] In operation 2704, in response to the first instruction (e.g., sub-instruction 610), the load / store controller 530 obtains the input data element from the memory external to the neural network processor (e.g., memory 512) to the input data register of the neural network processor (e.g., Din0 or Din1 of data register 528a). The load / store controller 530 may perform a read operation via the memory interface 534 to obtain the input data element. The read address may be generated by the address generator 532 and may be based on a circular addressing scheme, such as Figures 8A-10B Described in .

[0234] In addition, in operation 2706, in response to the first instruction (e.g., sub-instruction 608), the load / store controller 530 retrieves the weight element from the weight buffer of the neural network processor (e.g., weight and parameter buffer 526) to the weight register of the neural network processor (e.g., weight-0 or weight-1 register of weight / parameter register 528b). The retrieval of the weight element and the input data element can be performed in parallel.

[0235] In operation 2708, the compute controller 522 receives a second instruction from the instruction buffer 520. The second instruction may include a sub-instruction 602 indicating that a computation operation (e.g., a MAC operation, a post-processing operation, etc.) is to be performed by the compute engine 524 on the weight elements stored in the weight registers and the input data elements stored in the input data registers.

[0236] In operation 2710 , in response to the second instruction, the computing engine may obtain input data elements and weight elements from input data registers and weight registers, respectively.

[0237] In operation 2712, in response to the second instruction, the computation engine may perform computation operations, including MAC and post-processing operations, on the input data elements and the weight elements to generate output data elements.

[0238] In operation 2714, the compute engine may store the output data element in an output data register (eg, Dout).

[0239] Fig.28 Flowchart showing method 2800 of operation of a neural network processor. Method 2800 may be performed by various components of neural network processor 502, such as compute controller 522, load / store controller 530, compute engine 524 including MAC engine 1300, etc.

[0240] In operation 2802, the computation controller 522 receives a first indication of a particular input precision and a second indication of a particular weight precision. The first and second indications may be received from the configuration register 528c.

[0241] In operation 2804, the computation controller 522 configures the computation engine of the neural network processor based on the first and second indications. The configuration may be based on, for example, setting a D4 / D8 mode for the weight multiplexer 1304 and the input data multiplexer 1306, setting a configuration 1710 including a binary mode (or no binary mode) for the computation unit 1800 of the MAC engine 1300, and so on.

[0242] In operation 2806, the compute controller 522 receives instructions from the instruction buffer. The instructions may include sub-instructions 602 indicating that a set of MAC operations are to be performed by the MAC engine 1300 on the weight elements stored in the weight registers and the input data elements stored in the input data registers.

[0243] In operation 2808, in response to the instruction, the computing engine configured based on the first and second indications can obtain input data elements and weight elements from the input data register and the weight register, respectively. As described above, depending on whether the weight is 4-bit or 8-bit precision and whether the input data element is 4-bit or 8-bit precision (this can set the D8 / D4 mode), the weight remultiplexer 1304 and the data multiplexer 1306 can obtain weight elements of different bit precisions and input data elements of different bit precisions to the computing unit 1800, as shown in Figures 21A-21H. Depending on the bit precision, the weight remultiplexer 1304 can provide different bits of weight elements to different computing units, or provide different weight elements to different computing units. In the D8 mode, the weight remultiplexer 1304 can also copy some bits. In addition, in the D4 mode, the data multiplexer 1306 can provide a replica of the input data element.

[0244] In operation 2810, in response to the instruction, the computing engine configured based on the first and second indications can perform a multiply-accumulate (MAC) operation between the input data elements at a specific input precision and the weight elements at a specific weight precision to generate an intermediate output data element. For example, for a binary mode, the computing engine can perform a bitwise XNOR operation between the input data elements and the weight elements. In addition, for certain data and weight precisions (e.g., 8-bit input precision and 2-bit weight precision, 4-bit input and weight precision, etc.), each computing unit can generate an intermediate output data element. For other data and weight precisions (e.g., 8-bit input precision and 4-bit weight precision, 8-bit input precision and 8-bit weight precision), the computing engine can use a data merging circuit (e.g., data merging circuit 2002) to merge the outputs of different computing units to generate intermediate output data elements, as shown in Figures 21D and 21E.

[0245] In operation 2812, the compute engine may store the intermediate output data element at an intermediate output data register (eg, a MAC register).

[0246] Fig.29A and 29B Flowchart showing method 2900 of operation of a neural network processor. Method 2900 may be performed by various components of neural network processor 502, such as compute controller 522, load / store controller 530, compute engine 524 including MAC engine 1300, etc.

[0247] In operation 2902, the computation controller 522 receives a first indication of a particular output precision and a second indication of a particular weight precision. The computation controller 522 may also receive a third indication of a particular input precision. The first, second, and third indications may be received from the configuration register 528c.

[0248] In operation 2904, the computation controller 522 configures the post-processing engine 1302 based on the first and second indications (and the third indication). The configuration may be based on, for example, setting the clamp high and clamp low values ​​based on the output precision (8 bits or 4 bits), setting the multiplexer circuit 2416a to route the adder 2408a output to the adder 2408b input and setting the multiplexer circuit 2416b to route the adder 2408c output to the adder 2408d input to support 2-ternary weights, setting the data routing circuit 2412 to route the output of the data merge circuit 2002 to the multipliers 2402a-d to support 8-bit input precision and 4-bit weight precision or 8-bit input precision and 8-bit weight precision, as Figures 25A-25C as shown in .

[0249] In operation 2906, the compute controller 522 receives a first instruction from the instruction buffer. The first instruction may include a sub-instruction 602 indicating that a set of MAC operations are to be performed by the MAC engine 1300 on weight elements stored in weight registers and input data elements stored in input data registers.

[0250] In operation 2908, in response to the first instruction, the MAC engine 1300 of the computing engine 524 can obtain input data elements and weight elements from the input data register and the weight register, respectively. In some examples, the computing engine 524 can also be configured based on input and weight precision. As described above, depending on whether the weight is 4-bit or 8-bit precision, and whether the input data element is 4-bit or 8-bit precision (this can set the D8 / D4 mode), the weight remultiplexer 1304 and the data multiplexer 1306 can obtain weight elements of different bit precision and input data elements of different bit precision to the computing unit 1800, as shown in Figures 21A-21H. Depending on the bit precision, the weight remultiplexer 1304 can provide different bits of weight elements to different computing units, or provide different weight elements to different computing units. In the D8 mode, the weight remultiplexer 1304 can also copy some bits. In addition, in the D4 mode, the data multiplexer 1306 can provide a copy of the input data element.

[0251] In operation 2910, in response to the instruction, the MAC engine 1300 can perform a multiply-accumulate (MAC) operation between the input data elements at a specific input precision and the weight elements at a specific weight precision to generate an intermediate output data element. For example, for a binary mode, the computing engine can perform a bitwise XNOR operation between the input data elements and the weight elements. In addition, for certain data and weight precisions (e.g., 8-bit input precision and 2-bit weight precision, 4-bit input and weight precision, etc.), each computing unit can generate an intermediate output data element. For other data and weight precisions (e.g., 8-bit input precision and 4-bit weight precision, 8-bit input precision and 8-bit weight precision), the computing engine can use a data merging circuit (e.g., data merging circuit 2002) to merge the outputs from different computing units to generate intermediate output data elements, as shown in Figures 21D and 21E.

[0252] In operation 2912, the MAC engine 1300 may store the intermediate output data element at an intermediate output data register (eg, a MAC register).

[0253] refer to Fig.29B In operation 2914, the compute controller 522 receives a second instruction from the instruction buffer. The second instruction may include a sub-instruction 602 indicating that a post-processing operation (eg, BNorm) is to be performed on the intermediate output data by the post-processing engine 1302.

[0254] In operation 2916 , in response to the second instruction, the post-processing engine 1302 configured based on the first and second (and third) indications retrieves the intermediate output data element from the intermediate output data register.

[0255] In operation 2918, in response to the second instruction, the post-processing engine 1302 configured based on the first and second (and third) instructions performs post-processing operations on the intermediate data elements, such as BNorm operations, residual layer processing, etc., to generate output data elements, such as Fig.24 and 25A -C. Post-processing operations, including scaling, bit shifting, clamping, etc., can be performed in one clock cycle.

[0256] In operation 2920, in response to the second instruction, the post-processing engine 1302 stores the output data element at the output data register (e.g., Dout) of the data register 528a. The storage of the output data element may be performed by the data packing circuit 2304. After storing the threshold size of data (e.g., 32 bits) at Dout, the data packing circuit 2304 may transmit a control signal 2305 to the load / store controller 530 to obtain the output data to the memory 512.

[0257] In this specification, the term "coupled" may encompass connections, communications, or signal paths that support a functional relationship consistent with this specification. For example, if device A provides a signal to control device B to perform an action, then: (a) in the first example, device A is coupled to device B; or (b) in the second example, if the intermediate component C does not substantially change the functional relationship between device A and device B, so that device A controls device B via the control signal provided by device A, then device A is coupled to device B through the intermediate component C. In addition, in this specification, a device "configured to" perform a task or function may be configured (e.g., programmed and / or hardwired) by the manufacturer at the time of manufacture to perform the function, and / or may be configured (or reconfigured) by the user after manufacture to perform the function and / or other additional or alternative functions. Configuration may be by firmware and / or software programming of the device, by the construction and / or layout of the hardware components and interconnects of the device, or a combination thereof. In addition, in this specification, a circuit or device containing certain components may actually be used to couple to those components to form the circuit system or device. For example, a structure described as including one or more semiconductor elements (e.g., transistors), one or more passive elements (e.g., resistors, capacitors, and / or inductors), and / or one or more sources (e.g., voltage and / or current sources) may actually include only semiconductor elements within a single physical device (e.g., a semiconductor die and / or integrated circuit (IC) package) and may be used to couple to at least some of the passive elements and / or sources to form the structure during or after manufacturing, such as by an end user and / or a third party.

[0258] Although specific transistor structures are mentioned above, other transistors or device structures may be used. For example, a p-type MOSFET may be used in place of an n-type MOSFET with little or no additional changes. In addition, other types of transistors (e.g., bipolar transistors) may be used in place of the transistors shown. The capacitor may be implemented using different device structures (e.g., metal structures stacked on top of each other to form a parallel plate capacitor), or may be formed on a layer (metal or doped semiconductor) that is closer to or further from the surface of the semiconductor substrate.

[0259] As used above, the terms "terminal", "node", "interconnection" and "pin" are used interchangeably. Unless explicitly stated to the contrary, these terms are generally used to refer to the interconnection between device elements, circuit elements, integrated circuits, devices or other electronic or semiconductor components or their ends.

[0260] Although some components may be described as components of a specific process technology in this article, these components may be exchanged for components of other process technologies. The circuits described herein may be reconfigured to include replaced components to provide functions that are at least partially similar to the functions available before the component replacement. Unless otherwise specified, components shown as resistors generally represent any one or more elements that are coupled in series and / or in parallel to provide the amount of impedance represented as the resistor shown. For example, a resistor or capacitor shown and described as a single component in this article may actually be a plurality of resistors or capacitors that are coupled in series or in parallel between two nodes that are the same as the single resistor or capacitor, respectively. In addition, the use of the phrase "ground terminal" in this specification includes chassis grounding, earth grounding, floating grounding, virtual grounding, digital grounding, common grounding and / or any other form of ground connection that is applicable to or suitable for the teaching of this specification. Unless otherwise specified, the "approximately", "roughly" or "substantially" in front of a value represents + / -10% of the value.

[0261] The described examples can be modified, and other examples are possible, within the scope of the claims.

Claims

1. A neural network processor, comprising: Memory interface; Instruction buffer; Weight buffer; Input data register; Weight register; Output data register; Computation engine; as well as A controller configured to: receiving a first instruction from the instruction buffer; In response to the first instruction, fetching input data elements from the memory interface to the input data register, and fetching weight elements from the weight buffer to the weight register; receiving a second instruction from the instruction buffer; as well as In response to the second instruction: Retrieving the input data element and the weight element from the input data register and the weight register to the computing engine respectively; Using the computation engine, performing computation operations between the input data elements and the weight elements to generate output data elements; and The output data element is stored at the output data register.

2. The neural network processor of claim 1, wherein the controller is configured to: fetching a first sub-instruction from the first instruction, wherein the first sub-instruction instructs to obtain the input data element into the input data register; and fetching a second sub-instruction from the first instruction, wherein the second sub-instruction instructs to fetch the weight element from the weight buffer to the weight register; and The controller is configured to fetch the input data elements from the memory interface to the input data register in response to the first sub-instruction, and to fetch the weight elements from the weight buffer to the weight register in response to the second sub-instruction.

3. The neural network processor of claim 2 , wherein the controller is configured to fetch the input data elements from the memory interface to the input data registers and fetch the weight elements from the weight buffer to the weight registers in parallel.

4. The neural network processor of claim 2, wherein the input data element is a first input data element, the input data register is a first input data register, the neural network processor further comprises a second input data register, and the second instruction comprises a third sub-instruction instructing to obtain the second input data element into the second input data register; and Wherein the controller is configured to fetch the second input data element from the memory interface to the second input data register in response to the third sub-instruction.

5. The neural network processor of claim 4, wherein the controller is configured to perform the computational operation using the computational engine and to fetch the second input data element from the memory interface in parallel.

6. The neural network processor of claim 2, further comprising an input data address register configured to store an input data memory address, wherein the first sub-instruction includes a reference to the input data address register; and Wherein the controller is configured to, in response to the first sub-instruction, perform a memory read operation at the input data memory address via the memory interface to retrieve the input data element.

7. The neural network processor of claim 6, wherein the input data memory address is a first input data memory address, the first instruction comprises a third sub-instruction, wherein the third sub-instruction indicates that the first input data memory address is to be incremented or decremented, and the controller is configured to: extracting the third sub-instruction from the first instruction; In response to the third sub-instruction, generating a second input data memory address by incrementing or decrementing the first input data memory address; and The second input data memory address is stored in the input data address register to overwrite the first input data memory address.

8. The neural network processor of claim 7, wherein the controller is configured to generate the second input data memory address based on a starting input data memory address and an ending input data memory address, wherein the starting input data memory address and the ending input data memory address define an input data memory region.

9. The neural network processor of claim 8, further comprising a configuration register configured to store an indication of an address management mode and the starting and ending input data memory addresses, and wherein the controller is configured to indicate a cyclic address mode in response to the indication: If the second input data memory address is lower than the starting input data memory address: determining a first offset between the second input data memory address and the ending input data memory address; updating the second input data memory address based on the ending input data memory address and the first offset; and storing the updated second input data memory address in the input data address register to overwrite the first input data memory address; If the second input data memory address is higher than the end input data memory address: determining a second offset between the second input data memory address and the starting input data memory address; updating the second input data memory address based on the starting input data memory address and the second offset; and storing the updated second input data memory address in the input data address register to overwrite the first input data memory address; and If the second input data memory address is between the starting input data memory address and the ending input data memory address, the second input data memory address is stored in the input data address register to overwrite the first input data memory address.

10. The neural network processor of claim 6, further comprising an output data address register configured to store an output data memory address; The controller is configured as follows: receiving a third instruction from the instruction buffer, wherein the third instruction instructs the output data element to be written from the output data register to a memory via the memory interface, and the third instruction includes a reference to the output data address register; and In response to the third instruction, a memory write operation is performed at the output data memory address via the memory interface.

11. The neural network processor of claim 10, wherein the output data memory address is a first output data memory address, the third instruction comprises a third sub-instruction, wherein the third sub-instruction indicates that the first output data memory address is to be incremented or decremented, and the controller is configured to: extracting the third sub-instruction from the third instruction; In response to the third sub-instruction, generating a second output data memory address by incrementing or decrementing the first output data memory address; and The second output data memory address is stored in the output data address register to overwrite the first output data memory address.

12. The neural network processor of claim 11, wherein the controller is configured to generate the second output data memory address based on a start output data memory address and an end output data memory address, wherein the start output data memory address and the end output data memory address define an output data memory region.

13. The neural network processor of claim 12, further comprising a configuration register configured to store an indication of an address management mode, the start output data memory address, and the end output data memory address, and wherein the controller is configured to indicate a cyclic address mode in response to the indication: If the second output data memory address is lower than the starting output data memory address: determining a first offset between the second output data memory address and the ending output data memory address; updating the second output data memory address based on the ending output data memory address and the first offset; and storing the updated second output data memory address in the output data address register to overwrite the first output data memory address; If the second output data memory address is higher than the end output data memory address: determining a second offset between the second output data memory address and the starting output data memory address; updating the second output data memory address based on the starting output data memory address and the second offset; and storing the updated second output data memory address in the output data address register to overwrite the first output data memory address; and If the second output data memory address is between the start output data memory address and the end output data memory address, the second output data memory address is stored in the output data address register to overwrite the first output data memory address.

14. The neural network processor of claim 11, wherein the controller is configured to generate the second output data memory address based on a starting input data memory address and an ending input data memory address, wherein the starting input data memory address and the ending input data memory address define an input data memory region.

15. The neural network processor of claim 14, wherein the controller is configured to: If the second output data memory address is lower than the starting input data memory address: determining a first offset between the second output data memory address and the ending input data memory address; updating the second output data memory address based on the ending input data memory address and the first offset; and storing the updated second output data memory address in the output data address register to overwrite the first output data memory address; If the second output data memory address is higher than the end input data memory address: determining a second offset between the second output data memory address and the starting input data memory address; updating the second output data memory address based on the starting input data memory address and the second offset; and storing the updated second output data memory address in the output data address register to overwrite the first output data memory address; and If the second output data memory address is between the starting input data memory address and the ending input data memory address, the second output data memory address is stored in the input data address register to overwrite the first output data memory address.

16. The neural network processor of claim 1 , further comprising a configuration register configured to store a loop count value, wherein the controller is configured to: receiving a third instruction from the instruction buffer, wherein the third instruction includes a loop start indicator and a reference to the configuration register; receiving a fourth instruction from the instruction buffer; executing the fourth instruction; receiving a fifth instruction from the instruction buffer, wherein the fifth instruction includes an end loop indicator; and In response to the end loop indicator, execution of the fourth instruction is repeated a number of times based on the loop count value.

17. The neural network processor of claim 1, wherein the first and second instructions are associated with a first program counter (PC) value and a second PC value, respectively, and the controller is configured to: receiving a third instruction from the instruction buffer, wherein the third instruction includes an end indicator; receiving a fourth instruction from the instruction buffer, wherein the fourth instruction includes a macroinstruction indicator, a macroinstruction PC value equal to the first PC value, and a repeat count value; and In response to the macroinstruction indicator and the end indicator, the first and second instructions are executed a number of times equal to the repeat count value.

18. The neural network processor of claim 1, wherein the controller is configured to: receiving a third instruction from the instruction buffer; extracting a first instruction type indicator from the third instruction; Responsive to the first instruction type indicator, extracting a first number of sub-instructions from the third instruction; receiving a fourth instruction from the instruction buffer; extracting a second instruction type indicator from the fourth instruction; and Responsive to the second instruction type indicator, a second number of sub-instructions are extracted from the fourth instruction, wherein the first and second numbers are different.

19. The neural network processor of claim 18, wherein the first instruction type and the second instruction type indicate different ones of: a first instruction type including one sub-instruction, a second instruction type including two sub-instructions, a third instruction type including three sub-instructions, and a fourth instruction type including five sub-instructions.

20. A method comprising: receiving a first instruction from an instruction buffer of a neural network processor; In response to the first instruction: fetching input data elements from a memory external to the neural network processor to input data registers of the neural network processor; and Retrieving weight elements from a weight buffer of the neural network processor to a weight register of the neural network processor; receiving a second instruction from the instruction buffer; and In response to the second instruction: Retrieving the input data element and the weight element from the input data register and the weight register to a computing engine of the neural network processor, respectively; performing a computation operation between the input data elements and the weight elements using the computation engine to generate an output data element; and The output data element is stored at an output data register of the neural network processor.

21. The method according to claim 20, further comprising: determining an input data memory address based on a circular addressing scheme; as well as The input data element is retrieved from the memory by performing a memory read operation using the input data memory address.

22. The method according to claim 20, further comprising: determining an output data memory address based on a circular addressing scheme; as well as The output data element is stored at the memory by performing a memory write operation using the output data memory address.

23. A neural network processor, comprising: Memory interface; Instruction buffer; Weight buffer; Input data register; Weight register; Output data register; Address register; an address generation engine configurable to set an input data address, an output data address, and a weight address in the address register; a data load / store engine configurable to fetch input data from the memory interface to the input data register based on the input data address, fetch output data from the output data register to the memory interface based on the output data address, and fetch weights from the weight buffer to the weight register based on the weight address; a computation engine configurable to perform computations based on the input data and the weights to generate the output data; as well as a controller configured to, in response to one or more instructions from the instruction buffer: fetching a first sub-instruction for the address generation engine to set one of the input data address or the output data address; extracting a second sub-instruction for the address generation engine to set the weight address; fetching a third sub-instruction for the calculation engine to perform the calculation; fetching a fourth sub-instruction for the data load / store engine to obtain the weight at the weight address of the weight buffer; fetching a fifth sub-instruction for the data load / store engine to obtain the input data at the input data address via the memory interface; and In parallel, the address generation engine is configured with the first and second sub-instructions, the calculation engine is configured with the third sub-instruction, and the data load / store engine is configured with the fourth and fifth instructions.

24. The neural network processor of claim 23, wherein the one or more instructions include a first instruction and a second instruction, wherein the first instruction has a different bit length than the second instruction.

Images