Vector operation acceleration with convolution computation unit

Abstract

The present application describes a hybrid hardware accelerator, system, and device for performing various computations in neural network applications using the same set of hardware resources. An exemplary accelerator may include a weight selector, an activation input interface, and a plurality of MAC circuits organized as a plurality of multiplier-accumulator (MAC) lanes. Each of the plurality of MAC lanes may be configured to receive a control signal indicating whether to perform a convolution or vector operation; to receive one or more weights according to the control signal; to receive one or more activations according to the control signal; and to generate output data based on one or more weights and one or more input activations according to the control signal and to supply the output data to an output buffer. Each of the plurality of MAC lanes includes a plurality of multiplier circuits and a plurality of adder-subtractor circuits.

Description

Technology Field

[0001] The present disclosure generally relates to hardware designs for improving neural network computing efficiency, and in particular to a hybrid convolution-vector operation processing entity (PE) cluster for processing neural network computations such as convolution and vector operations. Background Technology

[0002] Neural Network (NN) computations involve convolution computations and various vector operations, such as vector reduction operations (e.g., max / minimum / sum / average reduction or index reduction) or pooling operations (e.g., max pooling, average pooling). Existing hardware architectures rely on SIMD (Single Instruction Multiple Data) vector processors or dedicated hardware (e.g., Tensor Processing Units (TPUs) or other ASIC designs) to implement these vector operations, in addition to processing units to handle convolution computations. Vector processors can handle one vector at a time, but they are inefficient for processing multiple vectors in parallel. Furthermore, SIMD processors may require specialized hardware to perform operations across multiple elements of the same vector for pooling operations or preliminary operations (e.g., transposition) before performing reduction operations. Moreover, the requirement to install a standalone hardware vector processor to handle vector operations in NN computation may increase the cost and design complexity of the NN processing unit.

[0003] Various embodiments of this specification may include a hardware accelerator, a PE cluster, and a system for handling convolution calculations and vector operations using the same set of hardware.

[0004] In some embodiments, the technology described herein relates to a vector operation accelerator for neural network computation. The accelerator may include a plurality of weight selectors configured to acquire weights; a plurality of activation input interfaces configured to acquire activations; and a plurality of MAC circuits organized as a plurality of Multiplier-Accumulation (MAC) lanes. In some embodiments, each of the plurality of MAC lanes may be configured to receive a control signal indicating whether to perform a convolution or vector operation; to receive one or more weights from at least one of the plurality of weight selectors according to the control signal; to receive one or more activations from at least one of the plurality of activation input interfaces according to the control signal; and to generate output data based on one or more weights and one or more input activations according to the control signal and supply the output data to an output buffer, wherein: each of the plurality of MAC lanes includes a plurality of first circuits for performing a multiplication operation according to the control signal and a plurality of second circuits for performing an addition or subtraction operation.

[0005] In some embodiments, a plurality of second circuits within a MAC lane are organized as a tree, and a second circuit at the leaf level of the tree is configured to receive data from a plurality of first circuits.

[0006] In some embodiments, each of the plurality of second circuits is configured to receive a first input and a second input; to determine whether to perform addition or subtraction based on a control signal; to generate a sum or average of the first input and the second input in response to the control signal indicating the performance of addition; and to generate a minimum or maximum value between the first input and the second input in response to the control signal indicating the performance of subtraction.

[0007] In some embodiments, the first input and the second input each comprise a vector having the same number of dimensions, and to generate a minimum value between the first input and the second input, each of the plurality of second circuits is further configured to generate an output vector containing the minimum value of the vector in each corresponding dimension.

[0008] In some embodiments, the accelerator may further include a weight matrix generating circuit configured to generate weights for a vector reduction operation, wherein the vector reduction operation includes one or more of mean reduction, minimum reduction, maximum reduction, average reduction, addition reduction, or pooling.

[0009] In some embodiments, each of the plurality of weight selectors includes a multiplexer coupled to a weight matrix generation circuit and a weight cache.

[0010] In some embodiments, each of the plurality of weight selectors is configured to acquire a weight from a weight cache in response to a control signal indicating the performance of a convolution calculation; and to acquire a weight from a weight matrix generation circuit in response to a control signal indicating the performance of a vector calculation.

[0011] In some embodiments, the accelerator may further include an adder-subtractor circuit outside the tree corresponding to the MAC lane, wherein the adder-subtractor circuit is configured to receive data from a second circuit at the root level of the MAC lane and to write the data to an output buffer.

[0012] In some embodiments, the adder-subtractor circuit is further configured to write a first set of data received from a second circuit at the root level of a MAC lane to an output buffer during a first iteration of calculation; and during a second iteration of calculation: to receive a set of temporary data from a second circuit at the root level of a MAC lane, to retrieve a first set of data from an output buffer, to calculate a second set of data based on a set of temporary data, a first set of data, and a control signal indicating whether to perform a convolution calculation or a vector operation, and to write a second set of data to an output buffer.

[0013] In some embodiments, multiple MAC lanes are configured to each receive multiple weight vectors generated by a weight matrix generation circuit to perform multiple vector operations in parallel.

[0014] In some embodiments, a first subset of multiple MAC lanes is configured to receive weights from a weight cache, and a second subset of multiple MAC lanes is configured to receive weights generated by a weight matrix generation circuit, and the first subset of multiple MAC lanes is further configured to perform convolution calculations, and the second subset of multiple MAC lanes is further configured to perform vector operations, and the convolution calculations and vector operations are performed in parallel.

[0015] In some embodiments, the technology described herein relates to a hybrid convolution-vector operation processing system. The system may include a plurality of weight selectors configured to acquire weights; a plurality of activation input interfaces configured to acquire activations; and a plurality of MAC circuits organized as a plurality of multiplier-accumulator (MAC) lanes. Each of the plurality of MAC lanes is configured to receive a control signal indicating whether to perform a convolution or vector operation; to receive one or more weights from at least one of the plurality of weight selectors according to the control signal; to receive one or more activations from at least one of the plurality of activation input interfaces according to the control signal; and to generate output data based on one or more weights and one or more input activations according to the control signal and supply the output data to an output buffer, wherein: each of the plurality of MAC lanes includes a plurality of first circuits for performing a multiplication operation according to the control signal and a plurality of second circuits for performing an addition or subtraction operation.

[0016] The methods of operation and functions of related elements of the structure, as well as the combination of parts and the economics of manufacturing, as well as the systems, methods, and other features of the non-transient computer-readable media disclosed herein, will become more apparent with reference to the following detailed description with reference to the accompanying drawings and the appended claims—all of which form part of this specification—in which like reference numerals indicate corresponding parts in various drawings. However, it should be explicitly understood that the drawings are for illustrative and illustrative purposes only and are not intended to define the limitations of the invention. Brief explanation of the drawing

[0017] FIG. 1 illustrates an exemplary system diagram for processing neural network computations in a hybrid PE array according to various embodiments. FIG. 2a illustrates an exemplary architectural diagram of a PE array according to various embodiments. FIG. 2b illustrates an exemplary architecture diagram of a hybrid PE array for processing both convolution calculations and vector operations according to various embodiments. FIG. 3 illustrates an exemplary internal structure diagram of a Multiplier-Accumulator (MAC) lane of a hybrid PE array according to various embodiments. FIG. 4 illustrates exemplary logic circuit designs of accumulators and subtractors for each MAC lane according to various embodiments. FIG. 5a illustrates an exemplary use case for performing vector reduction using a hybrid PE array according to various embodiments. FIG. 5b illustrates another exemplary use case for performing multiple vector reductions in parallel using a hybrid PE array according to various embodiments. FIG. 5c illustrates another exemplary use case for performing pooling using a hybrid PE array according to various embodiments. FIG. 6 illustrates an exemplary system design of a hybrid PE array according to various embodiments. FIG. 7 illustrates an exemplary computer system in which any of the embodiments described in this specification may be implemented. Specific details for implementing the invention

[0018] The embodiments described herein provide a hardware device, system, or array of processing entities (PEs) with the ability to perform various neural network computations in parallel by reusing hardware resources. Here, neural network computations involve convolution computations and vector operations, such as vector reduction operations (e.g., maximum / minimum / sum / average reduction related to vector values, or maximum / minimum reduction related to vector indices) and pooling operations (e.g., maximum pooling, average pooling), which constitute almost all computations involved in neural network training and applications. In some embodiments, the hardware device described herein includes an array of PEs along with other auxiliary logic circuits and can handle different types of neural network computations by reusing the same hardware resources. For simplification, the hardware device may be referred to as a hybrid PE array in the following designs. In some embodiments, the auxiliary logic circuits may include a weight matrix generation circuit configured to generate functional weights (compared to weights in the neural network).

[0019] While existing NN hardware designs use SIMD vector processors or standalone / distinct hardware (e.g., TPU) to handle vector operations in addition to processors configured to perform convolution operations, the hybrid PE array described herein reuses the same hardware resources, such as multiplier-accumulator (MAC), to perform convolution operations and simultaneously provide scalable vector operation parallel processing capabilities. In particular, the MAC resource may be configured to perform either summation or comparison via a control signal, wherein the summation function may be triggered to implement convolution calculation, vector sum reduction (e.g., adding corresponding values ​​of multiple input vectors and generating an output vector having the sum), vector average reduction (e.g., adding corresponding values ​​of multiple input vectors and generating an output vector having the sum divided by the number of input vectors), or average pooling (e.g., calculating the average for each patch of a feature map), etc., and the comparison function may be triggered to implement vector maximum reduction, vector minimum reduction, or maximum pooling (e.g., finding the maximum value within each patch of a feature map), etc.

[0020] In the following description, specific and non-limiting embodiments of the present invention will be described with reference to the drawings. Specific features and aspects of any embodiment disclosed herein may be used together with and / or combined with specific features and aspects of any other embodiment disclosed herein. It should also be understood that such embodiments are examples and merely illustrative of a small number of embodiments within the scope of the present invention. Various modifications and alterations obvious to those skilled in the art to which the present invention pertains are deemed to be within the spirit, scope, and concept of the present invention as further defined in the appended claims.

[0021] FIG. 1 illustrates an exemplary system diagram for processing neural network computations in a hybrid PE array (160) according to various embodiments. The diagram in FIG. 1 illustrates a hardware architecture configured to perform general neural network computations, such as convolution computations and vector operations, using the same hardware resources (e.g., the hybrid PE array (160)). The embodiments described in this disclosure may be implemented as part of the neural network computation of FIG. 1 or in other suitable environments.

[0022] Conventional neural networks, such as convolutional neural networks (CNNs), may involve various computations such as convolution and vector operations. For example, a convolution layer within a neural network (e.g., CNN) may perform convolution based on one or more input feature maps (IFMs) (including activations) obtained from an input source (e.g., an input image) or a previous layer (e.g., a tensor output from a previous layer) and one or more weight tensors corresponding to a given layer (e.g., from a weight source (130) such as a weight cache or a weight generator). The weight tensors may be used to convolve through the IFM to extract various features (e.g., convolution computations). The convolution process may be executed in parallel on a hybrid PE array (160). Each PE may refer to a processor having processing power and storage capacity (e.g., a buffer or cache). In some embodiments, each PE may include one or more logic gates or circuits configured as multipliers and accumulators (MACs). The PEs of the hybrid PE array (160) may be interconnected using wiring and may be arranged in a plurality of lanes referred to as PE lanes or MAC lanes. The hybrid PE array (160) may be manufactured as a neural network accelerator or a data processing system.

[0023] As another example, the pooling layer of a CNN may provide an approach to downsampling the IFM by summarizing the presence of features in patches of the feature map. Common pooling methods, such as average pooling and maximum pooling, may summarize the average presence of features and the most activated presence of features, respectively. These pooling operations may involve vector operations that differ from convolution operations. In particular, convolution operations involve convolving windows through the IFM and performing multiplication and accumulation to extract features, whereas vector operations may involve comparison or subtraction. In neural networks, other types of vector operations, such as vector reduction operations, are also common. The vector to be processed may be obtained from a vector memory (a type of input source (150)), which may be the same source from which activations are obtained for convolution. In some embodiments, the MAC of the hybrid PE array (160) may be configured to switch between "convolution mode" and "vector mode" to perform convolution calculations and vector operations to provide services to the convolution layer and pooling layer, respectively.

[0024] The configuration of the hybrid PE array (160) may be based on control signals or instructions issued by the instruction decoder (110). The instruction decoder (110) may decode instructions from a higher-level processor, such as a CPU or GPU. Based on the decoded instructions, the instruction decoder (110) may transmit corresponding control signals to different components. For example, the instruction decoder (110) may transmit an input activation / vector load control signal to the input source (150) indicating whether to fetch an activation for the hybrid PE array (160) or to fetch a vector. The instruction decoder (110) may also transmit a weight load control signal to the weight source (130) indicating whether weights should be fetched from the weight cache or obtained from the weight matrix generator (more details in FIG. 2b). Different weights play an important role in neural network computation. For example, weights fetched from the weight cache may refer to weights from filters corresponding to convolution layers configured to extract features from the IFM. Weights generated from the weight matrix generator may be designed to perform a specific type of vector operation. That is, depending on whether the current operation is a convolution operation or a vector operation, the source of the weights may differ. As another example, if the weights have a specific pattern, the instruction decoder (110) may instruct the weight matrix generator to generate weights so that the weights are generated internally rather than being fetched from the weight cache. This may minimize external memory access and thus improve the overall performance of the neural network computation.

[0025] The instruction decoder (110) may also transmit a compute control signal to the hybrid PE array (160) to perform a desired operation based on inputs from the input source (150) and weights from the weight source (130). The control signal may configure the MAC of the hybrid PE array (160) to perform summation (e.g., in the case of convolution) or subtraction (e.g., in the case of comparison in vector operations). After the hybrid PE array (160) has completed the calculation according to the control signal, the output data may be supplied to an output buffer (170) for temporary activation or for storing a vector. In some embodiments, the instruction decoder (110) may also control the output buffer to supply data from a previous iteration to the hybrid PE array (160) to accompany the current iteration.

[0026] FIG. 2a illustrates an exemplary architectural diagram of a PE array according to various embodiments. In the PE array of FIG. 2a, the arrangement of PEs is for illustrative purposes only and may be implemented in different ways depending on the use case.

[0027] As illustrated in the left portion of FIG. 2a, the PE array (200) may contain a matrix of PEs. Each PE may contain multiple multipliers (MUL gates). The multipliers within each PE may operate in parallel, and the PEs within the PE array (200) may operate in parallel. For ease of reference, the following description denotes the number of columns (220) of PEs in the PE array (200) as X, the number of rows (210) of PEs in the PE array (200) as Y2, and the number of multipliers within each PE as Y1. Each row (210) of PEs may be referred to as a PE cluster, and each PE cluster may be coupled to Y1 adder trees (230) to aggregate partial sums generated by the multipliers within the PE cluster. That is, the first multiplier of each PE in the PE cluster is coupled to the first adder tree (230) for aggregation, the second multiplier of each PE in the PE cluster is coupled to the second adder tree (230) for aggregation, and so on. The aggregation results from the adder tree (230) across all PE clusters (a total of Y1 × Y2 adder trees) may be supplied to the adder (250) for aggregation. The adder (250) may refer to a digital circuit that performs number addition, which is part of a Network-on-Chip (NoC) subsystem.

[0028] FIG. 2b illustrates an exemplary architectural diagram of a hybrid PE array for processing both convolutional computation and vector operations according to various embodiments. The exemplary diagram includes a plurality of hardware components interacting with one another, such as a weight matrix generator (270), a plurality of weight selectors (272) configured to receive weights from either the weight matrix generator (270) or a weight cache (not shown in FIG. 2b), a plurality of MAC (multiplier-accumulator circuit) lanes (273) each corresponding to a plurality of weight selectors (272), and a plurality of MACs (274) for receiving data from each of the plurality of MAC lanes (273) and outputting the data to an output buffer (not shown in FIG. 2b). These components are for illustrative purposes only, and depending on the embodiment, the architecture may include more, fewer, or alternative components. For example, a plurality of MACs (274) may be implemented as the last MAC in a plurality of MAC lanes, but have a different configuration from the other MACs in the MAC lanes. The weight selector (272) may also be implemented using a multiplexer coupled with the weight matrix generator (270) and the weight cache.

[0029] All components illustrated in FIG. 2b are controlled directly or indirectly by a control signal from an instruction decoder (e.g., the instruction decoder (110) of FIG. 1). The control signal may include weight load control and compute control for configuring a hybrid PE array to perform either or both of convolution calculations or vector operations using a plurality of MAC lanes (273). For example, if the upper application layer instructs to perform convolution calculations, the corresponding load control may be transmitted to a plurality of weight selectors (272) to notify them to select weights from a weight cache. Weights from the weight cache may originate from filters of the convolution layer of the neural network and may be used to convolve through an IFM to extract features. When the upper application layer instructs to perform a vector operation such as vector reduction (e.g., vector averaging, which merges multiple vectors into one with an average value), the weight load control may be transmitted to the weight matrix generator (270) to generate weights, and the weight selector (272) may be configured to select the generated weights and block weights from the weight cache. These generated weights may be functionally different from the weights for convolution calculation. For example, the generated weights may be used to calculate the average value of the input vectors (e.g., in the case of X vectors, each vector may be assigned 1 / X of the weight). In some embodiments, when the associated weights are highly organized (according to a specific pattern), the weight matrix generator (270) may also be triggered for convolution calculation. The weight matrix generator (270) may be instructed to generate these weights according to a specific pattern for the MAC lane.In this way, the MAC lane may prevent access to the weight cache for weights, thereby preventing memory access latency. Then, the weight selector (272) may forward the received weights to the MAC lane for the corresponding calculation.

[0030] Similarly, multiple MAC lanes (273) may be controlled by compute control signals. Compute control signals may instruct the MAC lanes to receive input activation for convolution calculation or to receive vectors for other vector operations. In some embodiments, in parallel, some of the MAC lanes (also referred to as the first subset of MAC lanes) may be configured for convolution calculation, and other MAC lanes (also referred to as the second subset of MAC lanes) may be configured for vector operations. This means that the hybrid PE array may perform both convolution calculation and vector operations simultaneously (e.g., during the same iteration). Compute control signals may further configure the MACs of each lane according to the specific workload to be performed. For example, in the case of convolution calculation on the MAC lanes, as part of the convolution calculation, the multiplier of the MAC lane may perform multiplication and the accumulator of the MAC lane may perform summation. In the case of vector reduction, such as vector maximum reduction or maximum pooling in a neural network pooling layer, the multiplier of the MAC lane may perform multiplication (with weights generated by the weight matrix generator), and the accumulator of the MAC lane may be configured to perform subtraction to realize the functionality of comparison. Comparison may help determine the maximum value in each dimension of the input vector (in the case of vector maximum reduction) or the maximum value within each patch of the feature map (in the case of maximum pooling). In some embodiments, the accumulator of the MAC lane is designed to be configurable to perform either summation or subtraction. A more detailed circuit design of the hybrid accumulator may be seen in FIG. 4.

[0031] In some embodiments, the last layer MAC (274) may act as a bridge between the MAC lane and the output buffer. For example, the MAC (274) may receive data (e.g., partial sum, temporary activation, temporary vector output) from the MAC lane and store the data in the corresponding output buffer. In some cases, after the MAC (274) stores the data in the output buffer in the first iteration, they may also read data from the output buffer as part of the calculation for the next iteration, along with new data received from the MAC lane (273).

[0032] FIG. 3 illustrates an exemplary internal structure diagram of a MAC (multiplier-accumulator) lane (300) of a hybrid PE array according to various embodiments. The diagram in FIG. 3 is for illustrative purposes only and may include more, fewer, or alternative components depending on the embodiment. As described above, each MAC lane (300) may include a plurality of multipliers (310) and accumulators, which may be implemented as digital gates or circuits (e.g., arithmetic logic units (ALUs)). The accumulator is designed to perform subtraction as well as summation according to specific compute instructions from the upper layer. In this way, the same piece of hardware (MAC lane (300)) may be reused for different types of computation without additional specialized processors (e.g., vector processors, TPUs). In this application, the accumulator may also be referred to as an adder-subtractor circuit (320) because of their hybrid functionality: sum (summing) and comparison (subtraction).

[0033] In some embodiments, the adder-subtractor circuit (320) of the MAC lane (300) may be organized as an adder tree (330). The leaf-level adder-subtractor circuit of the adder tree (330) may be configured to receive data from a plurality of multipliers (310). For example, each multiplier (310) may perform multiplication (for either convolution or vector operation) based on input vectors / activations and weights. The results of multiplication from two or more multipliers (310) may be supplied to the leaf-level adder-subtractor circuit of the adder tree (330) for summation or subtraction (comparison) according to a control signal. For example, each adder-subtractor circuit (320) may be configured to receive a first input and a second input; to determine whether to perform addition or subtraction based on a control signal; It may be configured to generate the sum or average of the first input and the second input in response to a control signal indicating the performance of addition; and to generate a minimum or maximum value between the first input and the second input in response to a control signal indicating the performance of subtraction. If the control signal indicates to perform vector minimum reduction, the first input and the second input may each contain a vector having the same number of dimensions, and the adder-subtractor circuit may generate an output vector containing the minimum value of the vector in each corresponding dimension.

[0034] In some embodiments, the adder tree (330) may include a plurality of adder-subtractor circuits (320) at the leaf level, one adder-subtractor circuit (340) at the root level, and one or more intermediate levels. The number of adder-subtractor circuits is reduced by half from one level to the next. In some embodiments, the root adder-subtractor circuit (340) may obtain the calculation result of the adder tree (330) (e.g., sum of values ​​or indices, minimum or maximum value) and transmit the result to an adder-subtractor circuit (350) outside the adder tree (330) corresponding to a MAC lane. The external adder-subtractor circuit (350) may be similar to the adder-subtractor circuit within the adder tree (330), except that the external adder-subtractor circuit (350) may write to and read from the output buffer. In some embodiments, the external adder-subtractor circuit (350) and the root adder-subtractor circuit (340) may be the same circuit as part of the adder tree (330).

[0035] In some embodiments, during the first iteration of the calculation, the external adder-subtractor circuit (350) may be configured to write a first set of data received from the root adder-subtractor circuit to an output buffer. During the second iteration of the calculation, the external adder-subtractor circuit (350) may be configured to receive a set of temporary data again from the root adder-subtractor circuit (350), retrieve a first set of data from the output buffer, calculate a second set of data based on the set of temporary data, the first set of data, and a control signal indicating whether to perform a convolution calculation or a vector operation, and write a second set of data to an output buffer.

[0036] FIG. 4 illustrates an exemplary logic circuit design (400) of an accumulator and a subtractor for each MAC lane according to various embodiments. The design (400) describes one method of implementing a hybrid circuit that may be configured to perform either addition or subtraction. Depending on the embodiment, the hybrid circuit may be implemented using other digital components.

[0037] As illustrated, the exemplary circuit includes a plurality of multiplexers (410 and 440) for selecting a signal from a plurality of inputs based on a control signal. The control signal may include a first signal (420) indicating whether addition or subtraction is being performed, and a second signal (430) indicating whether to obtain a minimum value or a maximum value from the input values ​​(if subtraction is being performed). These signals may control the selection logic of the multiplexers (410 and 440) to select an appropriate input.

[0038] FIG. 5a illustrates an exemplary use case for performing vector reduction using a hybrid PE array according to various embodiments. Although the hybrid PE array is used for convolutional computation in the convolution layer of a neural network, it may also be used for other types of computation, such as vector reduction, which requires different logic processing. The illustrated use case of FIG. 5a involves vector reduction across the channel dimension of an input tensor (520) to generate a single output vector (530). The single output vector (530) may include rows of vectors denoted as V0(:0) to V0(:, z-1)), where z is the channel dimension index. A weight matrix (510) used for vector reduction may be generated by a weight generation circuit. As illustrated, the weight matrix (510) may include a first row of 1s and all other elements as 0s, while the input tensor (520) may include multiple vectors. Note that even though the operator in FIG. 5a is a multiplication operator, it represents a vector operator rather than a matrix multiplication. Vector operators may be defined as part of compute control signals, which may include vector sum reduction, vector minimum or reduction, etc. For simplification, vector reduction operators may be denoted as the "reduce0" function. Through the "reduce0" function, each column of a vector within the input tensor (520) may be used as a parameter of the "reduce0" function. For example, V0(:0), which is the first vector of a single output vector, may be expressed as reduce0(A(0,0), A(1,0), ..., A(y-1,0)), where y refers to one of the row or column dimensions of the input tensor (520), and "reduce0" may be a sum, minimum, maximum, average, etc. This example illustrates a simple application of a hybrid PE array for channel dimension vector reduction.

[0039] FIG. 5b illustrates another exemplary use case for performing multiple vector reductions in parallel using a hybrid PE array according to various embodiments. The example in FIG. 5b illustrates that the hybrid PE array is highly flexible due to the rich dynamics of the weight matrix (540). Depending on requirements from a higher-level application, the weight matrix generation circuit may generate multiple weight rows in the weight matrix (540) to implement parallel vector reduction operations. Two weight rows may contain different weights to achieve two different vector reductions. For example, row 0 in the weight matrix (540) containing all 1s may correspond to a vector sum reduction, which may generate V0(:0)=sum(A(0,0), A(1,0), ...A(y-1,0)) using an input tensor (550), where y refers to one of the row or column dimensions of the input tensor (520); Row 1 containing all 1 / y may correspond to vector mean reduction, which may be used to generate V1(:0)=sum(A(0,0), A(1,0), …A(y-1,0)) / y. The output vectors (V0(:0) and V1(:0)) may be stored in the output vector tensor (560).

[0040] In some embodiments, the weight matrix (540) may include a first row of weights generated by a weight matrix generation circuit for vector operations, and a second row of weights fetched from a weight cache for convolution calculations. In this way, a single weight matrix (540) may be used to trigger both vector operations and convolution calculations in parallel. In particular, the "multiplication" operator of FIG. 5b may include an array of operators including a vector reduction operator corresponding to the first row and a multiplication operator corresponding to the second row (in the case of convolution). This example illustrates that a hybrid PE array may be configured to perform multiple identical vector reductions, multiple different vector reductions, or mixed vector reductions and convolutions during a single computation cycle.

[0041] FIG. 5c illustrates another exemplary use case for performing pooling using a hybrid PE array according to various embodiments. The use case of FIG. 5c involves a pooling operation. Pooling operations for downsampling a feature map by summarizing the presence of features in a patch of the feature map (e.g., a patch having a size of 3*3) are common in neural networks. The pooling operation may be implemented as a vector operation when the patch is represented as a vector. To do this, the hybrid PE array may first convert the patch into a vector and organize the vector as an input tensor (580) according to a corresponding control signal. Generally, the pooling process involves convolving the patch (e.g., a 3*3 patch) through an activation tensor, but the convolving step may be smaller than the size of the patch. Thus, the vector of the input tensor (580) may have overlapping elements. The weight matrix (570) may include weights generated from a weight generation circuit, wherein some rows of weights may be configured to perform one type of pooling (e.g., sum pooling) and other rows of weights may be configured to perform another type of pooling (e.g., max pooling). Through the weight matrix (570) and the input tensor (580), multiple pooling calculations may be executed using the same PE array during the same calculation cycle. The output vector may be stored in the output vector tensor (590).

[0042] FIG. 6 illustrates an exemplary system design of a hybrid PE array according to various embodiments. The hybrid PE array may be implemented as a hardware accelerator (600). The components of the accelerator (600) in FIG. 6 are for illustrative purposes only. Depending on the embodiment, the accelerator (600) may include more, fewer, or alternative components. In some embodiments, the accelerator (600) may include some or all of the components of FIG. 1, FIG. 2a, and FIG. 2b, such as MAC lanes. Each MAC lane within the accelerator (600) may include a plurality of multiplier and adder trees as shown in FIG. 3. The adder tree may include a plurality of multi-function adder-subtractors as shown in FIG. 4.

[0043] From a functional perspective, in some embodiments, the accelerator (600) may include a weight selection circuit (610), an activation selection circuit (620), a plurality of MAC lanes (630), and a weight matrix generation circuit (640). In some embodiments, the weight selection circuit (610) may be implemented as a multiplexer and may be coupled with a weight matrix generation circuit (640) and a weight cache, denoted as two weight sources. The weight selection circuit (610) may be instructed to obtain weights from these two weight sources according to a control signal. For different types of calculations, weights may be obtained from different sources. The activation selection circuit (620) may be configured to obtain an activation or a vector according to a different control signal, depending on the target calculation (e.g., convolution, pooling, vector operation).

[0044] In some embodiments, each MAC lane (630) may be configured to receive a control signal indicating whether to perform a convolution or vector operation; to receive one or more weights from at least one of a plurality of weight selectors according to the control signal; to receive one or more activations from at least one of a plurality of activation input interfaces according to the control signal; and to generate output data based on one or more weights and one or more input activations according to the control signal and supply the output data to an output buffer, wherein: each of the plurality of MAC lanes includes a plurality of first circuits for performing a multiplication operation according to the control signal and a plurality of second circuits for performing an addition or subtraction operation.

[0045] In some embodiments, the weight matrix generating circuit (640) may be configured to generate weights for a vector reduction operation, wherein the vector reduction operation includes one or more of mean reduction, minimum reduction, maximum reduction, average reduction, addition reduction, or pooling.

[0046] FIG. 7 illustrates an exemplary computing device in which any of the embodiments described herein may be implemented. The computing device may be used to implement one or more components of the method and system illustrated in FIGS. 1 through 6. The computing device (700) may include a bus (702) for transmitting information or other communication mechanisms and one or more hardware processors (704) coupled to the bus (702) for processing information. The hardware processor(s) (704) may be, for example, one or more general-purpose microprocessors.

[0047] The computing device (700) may also include main memory (707), such as random access memory (RAM), cache, and / or other dynamic storage devices, which are coupled to a bus (702) to store instructions and information to be executed by the processor(s) (704). The main memory (707) may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor(s) (704). Such instructions may make the computing device (700) into a special-purpose machine that is customized to perform the actions specified in the instructions when stored in a storage medium accessible to the processor(s) (704). The main memory (707) may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical or magnetic disks. Volatile media may include dynamic memory. Common forms of media may include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media having a pattern of holes, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chips or cartridges, or networked versions thereof.

[0048] The computing device (700) may implement the technology described herein by using logic embedded in customized hardware that, in combination with the computing device, may make the computing device (700) a special purpose machine or may program said computing device to become a special purpose machine, one or more ASICs or FPGAs, firmware and / or program logic. According to one embodiment, the technology of this specification is performed by the computing device (700) in response to the processor(s) (704) executing one or more sequences of one or more instructions contained in the main memory (707). Such instructions may be read into the main memory (707) from another storage medium, such as a storage device (708). The execution of the sequence of instructions contained in the main memory (707) may cause the processor(s) (704) to perform the process steps described herein. For example, the process / method disclosed herein may be implemented by computer program instructions stored in the main memory (707). When these instructions are executed by the processor(s) (704), they may perform steps as described above and illustrated in the corresponding drawings. In an alternative embodiment, a circuit embedded in the hardware may be used instead of or in combination with software instructions.

[0049] The computing device (700) also includes a communication interface (710) coupled to a bus (702). The communication interface (710) may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, the communication interface (710) may be a local area network (LAN) card for providing a data communication connection to a compatible LAN (or a WAN component for communicating with a WAN). A wireless link may also be implemented.

[0050] The performance of a given operation may reside within a single machine, or it may be distributed among processors deployed across multiple machines. In some exemplary embodiments, the processor or processor implementation engine may be located in a single geographic location (e.g., a home environment, an office environment, or within a server farm). In other exemplary embodiments, the processor or processor implementation engine may be distributed across multiple geographic locations.

[0051] Each of the processes, methods, and algorithms described in the aforementioned sections may be embodied by a code module executed by one or more computer systems or computer processors comprising computer hardware, and may be fully or partially automated by them. The processes and algorithms may also be partially or wholly implemented in circuits specific to the application.

[0052] Where the functions disclosed herein are implemented in the form of software function units and sold or used as independent products, they may be stored on a processor-executable, non-volatile, computer-readable storage medium. A specific technical solution or an aspect contributing to the present technology disclosed herein (in whole or in part) may be embodied in the form of a software product. A software product may be stored on a storage medium and includes a number of instructions for a computing device (which may be a personal computer, a server, a network device, and others) to execute all or some steps of the method of the embodiments of the present application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, other media operable to store program code, or any combination thereof.

[0053] A specific embodiment further provides a system comprising a non-transient computer-readable storage medium that stores instructions executable by a processor to enable the processor and the system to perform an operation corresponding to a step of any method of the embodiment disclosed above. A specific embodiment further provides a non-transient computer-readable storage medium configured to have instructions executable by one or more processors to enable one or more processors to perform an operation corresponding to a step of any method of the embodiment disclosed above.

[0054] The embodiments disclosed herein may be implemented through a cloud platform, server, or group of servers (hereinafter collectively referred to as the "Service System") that interacts with a client. The client may be a terminal device or a client registered by a user on a platform, wherein the terminal device may be a mobile terminal, a personal computer (PC), and any device on which a platform application program may be installed.

[0055] The various features and processes described above may be used independently of each other or may be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, in some embodiments, certain method or process blocks may be omitted. The methods and processes described herein are also not limited to any specific sequence, and the blocks or states associated therewith may be performed in other appropriate sequences. For example, the described blocks or states may be performed in an order different from that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Exemplary blocks or states may be performed serially, in parallel, or in any other manner. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added to, removed from, or rearranged in comparison to the disclosed exemplary embodiments.

[0056] Various operations of the exemplary methods described herein may be performed, at least in part, by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the non-transient computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly program the computer to perform the function, but may create a predictive model that learns from training samples to perform the function.

[0057] Various operations of the exemplary methods described herein may be performed, at least in part, by one or more processors that are temporarily or permanently configured (e.g., by software) to perform the relevant operations. Whether temporarily or permanently configured, such processors may comprise a processor implementation engine that operates to perform one or more operations or functions described herein.

[0058] Likewise, the method described herein may be implemented at least partially by a processor, and specific processors or processors are examples of hardware. For example, at least part of the operation of the method may be performed by one or more processors or processor implementation engines. Additionally, one or more processors may operate in a "cloud computing" environment or as "software as a service (SaaS)" to support the performance of the relevant operation. For example, at least part of the operation may be performed by a group of computers (as an example of a machine including a processor), and these operations are accessible via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., Application Program Interfaces (APIs)).

[0059] The performance of a given operation may reside within a single machine, or it may be distributed among processors deployed across multiple machines. In some exemplary embodiments, the processor or processor implementation engine may be located in a single geographic location (e.g., a home environment, an office environment, or within a server farm). In other exemplary embodiments, the processor or processor implementation engine may be distributed across multiple geographic locations.

[0060] Throughout this specification, multiple instances may implement a component, operation, or structure described as a single instance. Although individual operations of one or more methods are illustrated and described as distinct operations, nothing is required that one or more of the individual operations be performed simultaneously or that the operations be performed in the order in which they are illustrated. Structures and functionalities presented as distinct components in exemplary configurations may be implemented as a combined structure or component. Likewise, structures and functionalities presented as a single component may be implemented as distinct components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter of this specification.

[0061] As used herein, "or" is inclusive and non-exclusive unless otherwise explicitly indicated or otherwise indicated by the context. Accordingly, in this specification, "A, B or C" means "A, B, A and B, A and C, B and C, or A, B and C" unless otherwise explicitly indicated or otherwise indicated by the context. Additionally, "and" means both joint and several unless otherwise explicitly indicated or otherwise indicated by the context. Accordingly, in this specification, "A and B" means "A and B jointly or individually" unless otherwise explicitly indicated or otherwise indicated by the context. Furthermore, regarding the resources, operations, or structures described herein, multiple instances may be provided as a single instance. Additionally, the boundaries between various resources, operations, engines, and data repositories are somewhat arbitrary, and specific operations are exemplified in the context of specific exemplary configurations. Other assignments of functionality are conceived and may fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as separate resources in exemplary configurations may be implemented as combined structures or resources. Likewise, structures and functions presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of embodiments of this disclosure as indicated by the appended claims. Accordingly, this specification and drawings should be conceived in an illustrative sense rather than a restrictive one.

[0062] The terms "include" or "comprise" are used to indicate the presence of a previously declared feature, but they do not exclude the addition of other features. Among other things, conditional language such as "can," "could," "might," or "may" is generally intended to convey that a certain embodiment includes a certain feature, element, and / or step, while another embodiment does not, unless otherwise specifically stated or otherwise understood within the context in which it is used. Accordingly, such conditional language is generally not intended to imply that one or more embodiments necessarily include logic for determining whether the feature, element, and / or step is required in any particular embodiment or whether such feature, element, and / or step is to be performed or included in any particular embodiment, regardless of the presence or absence of user input or prompts.

[0063] Although an overview of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of the embodiments of the present disclosure. Such embodiments of the subject matter may be referred to herein by the term “invention,” individually or collectively, merely for convenience and without the intention of voluntarily limiting the scope of the present application to any single disclosure or concept where more than one embodiment is disclosed.

[0064] The embodiments exemplified herein are described in sufficient detail to enable a person skilled in the art to practice the disclosed teachings. Other embodiments may be used or derived therefrom so that structural and logical substitutions and modifications may be made without departing from the scope of the disclosure. Accordingly, specific details for practicing the foregoing invention should not be considered in a restrictive sense, and the scope of various embodiments is defined only by the appended claims, together with the full scope of equivalents to which such claims are granted.

Claims

Claim 1 As a neural network accelerator, an instruction decoder configured to decode a neural network computation instruction from a processor into a weight load control signal, an activation load control signal, and a compute control signal; a plurality of weight selectors configured to acquire a weight according to the weight load control signal, wherein the weight load control signal indicates whether to acquire the weight from a weight cache or from a weight generator; a plurality of activation select circuits configured to acquire an activation or a vector from memory according to the activation load control signal, wherein the activation load control signal indicates whether to acquire the activation or the vector; and a plurality of circuit lanes, wherein each circuit lane of the plurality of circuit lanes is configured to receive the weight acquired by the plurality of weight selectors and the activation or the vector acquired by the plurality of activation select circuits; and to determine whether to perform a convolution operation or a vector operation according to the compute control signal. A neural network accelerator configured to generate output data by performing the convolution operation or the vector operation based on the weight and the activation or the vector, and the instruction decoder further configured to instruct the plurality of weight selectors to obtain the weight from the weight generator instead of obtaining the weight from the weight cache in response to the weight having a pattern in order to reduce memory access. Claim 2 A neural network accelerator according to claim 1, wherein the weight cache includes weights of filters corresponding to convolution layers within a neural network for convolution operations, and the weight generator generates weights for performing vector operations. Claim 3 A neural network accelerator according to claim 1, wherein the weight load control signal indicates whether the current operation is a convolution operation or a vector operation, and in response that the current operation is the convolution operation, the plurality of weight selectors obtain the weight from the weight cache, and in response that the current operation is the vector operation, the plurality of weight selectors obtain the weight from the weight generator. Claim 4 A neural network accelerator according to claim 1, wherein the compute control signal instructs the plurality of circuit lanes to perform summation for convolution operations or subtraction for vector operations. Claim 5 A neural network accelerator according to claim 1, wherein, according to the compute control signal, a subset of the plurality of circuit lanes is configured to perform convolution operations and, at the same time, the remaining circuit lanes are configured to perform vector operations. Claim 6 A neural network accelerator according to claim 1, wherein each lane of the circuit is organized as a tree, the circuit at the leaf level of the tree is configured to perform multiplication based on one or more of the weights and one or more of the activation or vectors, and the circuit at the non-leaf level of the tree is configured according to the compute control signal indicating whether to perform addition or subtraction based on data received from the circuit at the previous level of the tree. Claim 7 A neural network accelerator according to claim 1, wherein the vector operation comprises one or more of mean reduction, minimum reduction, maximum reduction, average reduction, addition reduction, or pooling. Claim 8 A neural network accelerator according to claim 1, wherein each of the plurality of weight selectors comprises a multiplexer coupled to the weight generator and the weight cache. Claim 9 A hybrid convolution-vector operation processing system comprises: an instruction decoder configured to decode a neural network computation instruction from a processor into a weight load control signal and a compute control signal; a plurality of weight selectors configured to acquire a weight according to the weight load control signal, wherein the weight load control signal indicates whether to acquire the weight from a weight cache or from a weight generator; a plurality of activation selectors configured to acquire an activation or a vector from memory; and a plurality of circuit lanes, wherein each circuit lane of the plurality of circuit lanes receives the weight acquired by the plurality of weight selectors and the activation or the vector acquired by the plurality of activation selectors; and determines whether to perform a convolution operation or a vector operation according to the compute control signal. A hybrid convolution-vector operation processing system configured to generate output data by performing the convolution operation or the vector operation based on the weight and the activation or the vector, wherein the instruction decoder is further configured to instruct the plurality of weight selectors to obtain the weight from the weight generator instead of obtaining the weight from the weight cache in order to reduce memory access in response to the weight having a pattern. Claim 10 A hybrid convolution-vector operation processing system according to claim 9, wherein the weight cache includes weights of filters corresponding to convolution layers within a neural network for convolution operations, and the weight generator generates weights for performing vector operations. Claim 11 A hybrid convolution-vector operation processing system according to claim 9, wherein the weight load control signal indicates whether the current operation is a convolution operation or a vector operation, and in response that the current operation is the convolution operation, the plurality of weight selectors obtain the weight from the weight cache, and in response that the current operation is the vector operation, the plurality of weight selectors obtain the weight from the weight generator. Claim 12 A hybrid convolution-vector operation processing system according to claim 9, wherein the compute control signal instructs the plurality of circuit lanes to perform summation for convolution operations or subtraction for vector operations. Claim 13 A hybrid convolution-vector operation processing system according to claim 9, wherein, in accordance with the compute control signal, a subset of the plurality of circuit lanes is configured to perform a convolution operation and, at the same time, the remaining circuit lanes are configured to perform a vector operation. Claim 14 A hybrid convolution-vector operation processing system according to claim 9, wherein each lane of the circuit is organized as a tree, the circuit at the leaf level of the tree is configured to perform multiplication based on one or more of the weights and one or more of the activation or vectors, and the circuit at the non-leaf level of the tree is configured according to the compute control signal indicating whether to perform addition or subtraction based on data received from the circuit at the previous level of the tree. Claim 15 A hybrid convolution-vector operation processing system according to claim 9, wherein the vector operation comprises one or more of mean reduction, minimum reduction, maximum reduction, average reduction, addition reduction, or pooling. Claim 16 A hybrid convolution-vector operation processing system according to claim 9, wherein each of the plurality of weight selectors comprises a multiplexer coupled to the weight generator and the weight cache. Claim 17 A computer implementation method comprising: receiving a neural network computation instruction from a processor; decoding the neural network computation instruction into a weight load control signal, an activation load control signal, and a compute control signal by an instruction decoder; transmitting the weight load control signal to a plurality of weight selectors to obtain a weight, wherein the weight load control signal indicates whether the plurality of weight selectors will obtain the weight from a weight cache or from a weight generator; transmitting the activation load control signal to a plurality of activation selector circuits to obtain an activation or a vector from memory, wherein the activation load control signal indicates whether the activation or the vector will be obtained. A computer implementation method comprising the step of transmitting the compute control signal to a plurality of circuit lanes, wherein the plurality of circuit lanes are configured to receive, according to the compute control signal, the weight obtained by the plurality of weight selectors and the activation or the vector obtained by the plurality of activation selectors; to determine whether to perform a convolution operation or a vector operation according to the compute control signal; and to perform the convolution operation or the vector operation based on the weight and the activation or the vector to generate output data, and the instruction decoder is further configured to instruct the plurality of weight selectors to obtain the weight from the weight generator instead of obtaining the weight from the weight cache in response to the weight having a pattern, in order to reduce memory access. Claim 18 A computer implementation method according to claim 17, wherein the weight load control signal indicates whether the current operation is a convolution operation or a vector operation, and in response that the current operation is the convolution operation, obtaining the weight includes obtaining the weight from the weight cache, and in response that the current operation is the vector operation, obtaining the weight includes obtaining the weight from the weight generator. Claim 19 A dual-operation neural network accelerator comprising a plurality of MAC circuits organized into a plurality of Multiplier-Accumulation (MAC) lanes, wherein each of the plurality of MAC lanes comprises a plurality of first circuits for performing multiplication operations; A plurality of second circuits organized as a tree including a root level and leaf levels, wherein the root level includes one second circuit and the leaf level includes a plurality of second circuits, wherein the plurality of second circuits in the leaf level are configured to receive data from the plurality of first circuits and to perform addition or subtraction operations on the received data and to supply the result of the addition or subtraction operation toward the root level, wherein the second circuit in the root level is configured to generate an output based on the result of the addition or subtraction operation and to transmit the output to an adder-subtractor circuit outside the tree of the plurality of second circuits, wherein the adder-subtractor circuit is configured to read previous data from an output buffer, to calculate a new output based on the previous data and the output received from the second circuit in the root level, and to write the new output to the output buffer, wherein each of the plurality of second circuits is a combination of an accumulator and a subtractor and to perform both addition and subtraction using the same circuit. A dual operation neural network accelerator configured such that each of the plurality of MAC lanes is configured to (1) fetch weights from a weight cache for convolution operations and (2) obtain weights generated by a weight matrix generation circuit for vector operations, and the weight matrix generation circuit is configured to internally generate a row of weights for one of the plurality of MAC lanes. Claim 20 A dual operation neural network accelerator according to claim 19, wherein each of the plurality of second circuits comprises an adder-subtractor logic gate configurable to perform the addition or subtraction operation. Claim 21 A dual operation neural network accelerator according to claim 20, wherein the plurality of second circuits are configured to receive a first control signal and a second control signal, the first control signal configures the adder-subtractor logic gate to perform addition or subtraction, and the second control signal configures the adder-subtractor logic gate to post-process the result of the addition or subtraction. Claim 22 In claim 19, the second circuit at the root level is additionally configured to read a previous output generated in a previous iteration from the buffer of the dual operation neural network accelerator; and to generate the output based on the previous output and the result of the addition or subtraction operation. Claim 23 A dual operation neural network accelerator according to claim 19, further comprising: a plurality of weight selectors configured to acquire weights; and a plurality of activation input interfaces configured to acquire activations, wherein each of the plurality of MAC lanes is configured to receive a control signal indicating whether to perform the convolution operation or the vector operation; to receive one or more weights from at least one of the plurality of weight selectors according to the control signal; to receive one or more activations from at least one of the plurality of activation input interfaces according to the control signal; and to input the one or more weights and the one or more activations to the MAC lanes of the plurality of MAC lanes to generate output data according to the control signal. Claim 24 In paragraph 23, the vector operation comprises one or more of mean reduction, minimum reduction, maximum reduction, average reduction, addition reduction, or pooling, a dual-operation neural network accelerator. Claim 25 In paragraph 24, a dual operation neural network accelerator wherein each of the plurality of weight selectors comprises a multiplexer coupled to the weight matrix generation circuit and the weight cache. Claim 26 A dual operation neural network accelerator according to claim 25, wherein each of the plurality of weight selectors is configured to obtain one weight from the weight cache in response to the control signal indicating the performance of the convolution operation; and to obtain one weight from the weight matrix generation circuit in response to the control signal indicating the performance of the vector calculation. Claim 27 A dual-computation neural network accelerator according to claim 25, wherein a first subset of the plurality of MAC lanes is configured to receive weights from the weight cache, and a second subset of the plurality of MAC lanes is configured to receive weights generated by the weight matrix generation circuit, and further wherein the first subset of the plurality of MAC lanes is additionally configured to perform convolution calculations, and the second subset of the plurality of MAC lanes is additionally configured to perform vector operations, and wherein the convolution calculations and the vector operations are performed in parallel. Claim 28 A dual operation neural network accelerator according to claim 19, wherein each of the plurality of second circuits comprises a plurality of multiplexers for selecting a signal from a plurality of inputs based on a control signal, the control signal comprising a first control signal indicating whether addition is being performed or subtraction is being performed, and a second control signal indicating whether to obtain a minimum value or a maximum value from an input value. Claim 29 As a Multiplier-Accumulation (MAC) lane within a neural network accelerator, a plurality of first circuits for performing multiplication operations; A plurality of second circuits organized as a tree including a root level and leaf levels, wherein the root level includes one second circuit and the leaf level includes a plurality of second circuits, wherein the plurality of second circuits in the leaf level are configured to receive data from the plurality of first circuits and to perform addition or subtraction operations on the received data and to supply the result of the addition or subtraction operation toward the root level, wherein the second circuit in the root level is configured to generate an output based on the result of the addition or subtraction operation and to transmit the output to an adder-subtractor circuit outside the tree of the plurality of second circuits, wherein the adder-subtractor circuit is configured to read previous data from an output buffer, to calculate a new output based on the previous data and the output received from the second circuit in the root level, and to write the new output to the output buffer, wherein each of the plurality of second circuits is a combination of an accumulator and a subtractor and to perform both addition and subtraction using the same circuit. A MAC lane configured to (1) fetch weights from a weight cache for convolution operations and (2) obtain weights generated by a weight matrix generation circuit for vector operations, wherein the weight matrix generation circuit is configured to internally generate a row of weights for the MAC lane. Claim 30 In claim 29, a MAC lane wherein each of the plurality of second circuits comprises an adder-subtractor logic gate configurable to perform the addition or subtraction operation. Claim 31 A MAC lane according to claim 30, wherein the plurality of second circuits are configured to receive a first control signal and a second control signal, the first control signal configures the adder-subtractor logic gate to perform addition or subtraction, and the second control signal configures the adder-subtractor logic gate to post-process the result of the addition or subtraction. Claim 32 In claim 29, the second circuit at the root level is additionally configured to read a previous output generated in a previous iteration from the buffer of the neural network accelerator; and to generate the output based on the previous output and the result of the addition or subtraction operation, MAC lane. Claim 33 In claim 29, the MAC lane is coupled to a plurality of weight selectors configured to acquire weights, and a plurality of activation input interfaces configured to acquire activations. Claim 34 A MAC lane according to claim 33, wherein the plurality of first circuits are configured to receive a control signal indicating whether to perform a convolution operation or a vector operation; to receive one or more weights from at least one of the plurality of weight selectors according to the control signal; and to receive one or more activations from at least one of the plurality of activation input interfaces according to the control signal. Claim 35 A computer implementation method comprising: receiving a control signal indicating whether to perform a convolution operation or a vector operation; receiving one or more weights from at least one of a plurality of weight selectors according to the control signal; and receiving one or more activations from at least one of a plurality of activation input interfaces according to the control signal.The method comprises the step of inputting one or more weights and one or more activations to a Multiplier-Accumulation (MAC) lane to generate output data according to the control signal, wherein the MAC lane comprises a plurality of first circuits for performing multiplication operations and a plurality of second circuits organized as a tree including a root level and leaf levels, wherein the root level comprises one second circuit and the leaf level comprises a plurality of second circuits, wherein the plurality of second circuits in the leaf level are configured to receive data from the plurality of first circuits and to perform addition or subtraction operations on the received data and to supply the result of the addition or subtraction operation toward the root level, wherein the second circuit in the root level is configured to generate the output data based on the result of the addition or subtraction operation and to transmit the output to an adder-subtractor circuit outside the tree of the plurality of second circuits, and wherein the adder-subtractor circuit reads previous data from an output buffer, the previous data and the root A computer implementation method configured to calculate a new output based on the output received from the second circuit at the level and to write the new output to the output buffer, wherein each of the plurality of second circuits is a combination of an accumulator and a subtractor and is configured to perform both addition and subtraction using the same circuit, and the MAC lane is configured to (1) fetch weights from a weight cache for convolution operations and (2) obtain weights generated by a weight matrix generation circuit for vector operations, and the weight matrix generation circuit is configured to internally generate a row of weights for the MAC lane. Claim 36 A computer-implemented method according to claim 35, wherein each of the plurality of second circuits comprises an adder-subtractor logic gate configurable to perform the addition or subtraction operation. Claim 37 A computer implementation method according to claim 36, wherein the plurality of second circuits are configured to receive a first control signal and a second control signal, the first control signal configures the adder-subtractor logic gate to perform addition or subtraction, and the second control signal configures the adder-subtractor logic gate to post-process the result of the addition or subtraction.

Images