Neural network accelerator, hybrid convolution-vector operation processing system and computer implementation method

Abstract

Neural network accelerators, hybrid convolution-vector operation processing systems, and computer-implemented methods are described. The neural network accelerator includes an instruction decoder configured to decode a neural network computing instruction from a processor into a weight load control signal, an activation load control signal, and a computing control signal; a plurality of weight selectors configured to obtain a weight according to a weight load control signal indicating whether the weight is obtained from the weight cache or the weight generator; a plurality of activation input interfaces configured to obtain an activation or a vector from the memory in accordance with an activation load control signal indicating whether to obtain the activation or the vector; and a plurality of circuit channels. The instruction decoder is further configured to, in response to the weights having a mode, instruct the plurality of weight selectors to obtain weights from the weight generator, rather than from the weight cache, to reduce memory access.

Description

[0001] This application is a divisional application of the application filed on September 5, 2023, with application number 202380066171.1 and invention title "Acceleration of Vector Operations with Convolution Computation Unit". Technical Field

[0002] This invention generally relates to a hardware design for improving the computational efficiency of neural networks, and in particular to a hybrid convolution-vector operation PE (processing entity) cluster for handling neural network computations such as convolution and vector operations. Background Technology

[0003] Neural network (NN) computation involves convolution computations and various vector operations, such as vector reduction operations (e.g., reducing to maximum / minimum / sum / average or reducing to index) or pooling operations (e.g., max pooling, average pooling). In addition to processing units for handling convolution computations, existing hardware architectures rely on SIMD (Single Instruction Multiple Data) vector processors or dedicated hardware (e.g., Tensor Processing Units (TPUs) or other ASIC designs) to implement these vector operations. Vector processors can process one vector at a time, but parallel processing of multiple vectors is inefficient. Furthermore, SIMD processors may require specialized hardware to perform operations across multiple elements of the same vector before performing reduction operations, such as pooling or preliminary operations (e.g., transpose). Moreover, the requirement to install separate hardware vector processors to handle vector operations in neural network computations can increase the cost and design complexity of neural network processing units. Summary of the Invention

[0004] Various embodiments of this specification may include hardware accelerators, PE clusters, and systems for processing convolutional computations and vector operations using the same set of hardware.

[0005] In some aspects, the technology described herein relates to a neural network accelerator, comprising: an instruction decoder configured to decode neural network computation instructions from a processor into a weight load control signal, an activation load control signal, and a computation control signal; a plurality of weight selectors configured to obtain weights according to the weight load control signal, wherein the weight load control signal indicates whether the weights are obtained from a weight cache or a weight generator; a plurality of activation input interfaces configured to obtain activations or vectors from memory according to the activation load control signal, wherein the activation load control signal indicates whether to obtain the activation or the vector; and a plurality of circuit channels. Each circuit channel is configured to: receive the weights obtained by the plurality of weight selectors and the activations or vectors obtained by the plurality of activation selectors; determine whether to perform a convolution operation or a vector operation according to the computation control signal; and perform the convolution operation or vector operation based on the weights and the activations or vectors to generate output data. Furthermore, the instruction decoder is further configured to, in response to the weights having a pattern, instruct the plurality of weight selectors to obtain the weights from the weight generator instead of from the weight cache, thereby reducing memory access.

[0006] In some aspects, the technology described herein relates to a hybrid convolution-vector operation processing system, comprising: an instruction decoder configured to decode neural network computation instructions from a processor into weight load control signals and computation control signals; a plurality of weight selectors configured to obtain weights based on the weight load control signals, wherein the weight load control signals indicate whether weights are obtained from a weight cache or a weight generator; a plurality of activation selectors configured to obtain activations or vectors from memory; and a plurality of circuit channels. Each of the circuit channels is configured to: receive the weights obtained by the plurality of weight selectors and the activations or vectors obtained by the plurality of activation selectors; determine whether to perform a convolution operation or a vector operation based on the computation control signals; and perform the convolution operation or vector operation based on the weights and the activations or vectors to generate output data. Furthermore, the instruction decoder is further configured to, in response to the weights having a pattern, instruct the plurality of weight selectors to obtain the weights from the weight generator instead of from the weight cache, thereby reducing memory access.

[0007] In some aspects, the technology described herein relates to a computer-implemented method, comprising: receiving neural network computation instructions from a processor; decoding the neural network computation instructions into a weight load control signal, an activation load control signal, and a computation control signal via an instruction decoder; sending the weight load control signal to a plurality of weight selectors to obtain weights, wherein the weight load control signal indicates whether the plurality of weight selectors obtain the weights from a weight cache or from a weight generator; sending the activation load control signal to a plurality of activation selectors to obtain an activation or a vector from memory, wherein the activation load control signal indicates whether to obtain the activation or the vector; and sending the computation control signal to a plurality of circuit channels. The plurality of circuit channels are configured according to the computation control signal to: receive the weights obtained by the plurality of weight selectors and the activation or the vector obtained by the plurality of activation selectors; determine whether to perform a convolution operation or a vector operation according to the computation control signal; and perform the convolution operation or the vector operation based on the weights and the activation or the vector to generate output data. Furthermore, the instruction decoder is further configured to, in response to the weight having a pattern, instruct the plurality of weight selectors to obtain the weight from the weight generator instead of from the weight cache, in order to reduce memory access.

[0008] In some aspects, the techniques described herein relate to vector operation accelerators for neural network computation. The accelerator may include multiple weight selectors configured to acquire weights; multiple activation input interfaces configured to acquire activations; and multiple multiplier-accumulator (MAC) circuitry organized into multiple MAC channels. In some embodiments, each of the multiple MAC channels may be configured to: receive a control signal indicating whether to perform a convolution operation or a vector operation; receive one or more weights from at least one of the multiple weight selectors according to the control signal; receive one or more activations from at least one of the multiple activation input interfaces according to the control signal; and generate output data based on one or more weights and one or more input activations according to the control signal, and feed the output data into an output buffer, wherein each of the multiple MAC channels includes multiple first circuitry for performing multiplication operations and multiple second circuitry for performing addition or subtraction operations according to the control signal.

[0009] In some respects, multiple second circuits within a MAC channel are organized as a tree, and the second circuits at the leaf level of the tree are configured to receive data from multiple first circuits.

[0010] In some respects, each of the plurality of second circuits is configured to: receive a first input and a second input; determine whether to perform addition or subtraction based on a control signal; generate a sum or average of the first input and the second input in response to a control signal indicating that addition should be performed; and generate a minimum or maximum value between the first input and the second input in response to a control signal indicating that subtraction should be performed.

[0011] In some respects, each of the first and second inputs comprises a vector having the same number of dimensions, and in order to generate the minimum value between the first and second inputs, each of the plurality of second circuits is further configured to generate an output vector containing the minimum value of the vector in each corresponding dimension.

[0012] In some aspects, the accelerator may also include: a weight matrix generation circuit configured to generate weights for vector reduction operations, wherein the vector reduction operations include one or more of reduction mean, reduction minimum, reduction maximum, reduction average, reduction summation, or pooling.

[0013] In some respects, each of the multiple weight selectors includes a multiplexer coupled to the weight matrix generation circuit and the weight cache.

[0014] In some respects, each of the multiple weight selectors is configured to: obtain weights from a weight cache in response to a control signal instructing the execution of a convolution computation; and obtain weights from a weight matrix generation circuit in response to a control signal instructing the execution of a vector computation.

[0015] In some aspects, the accelerator may also include an adder-subtractor circuit outside the tree corresponding to the MAC channel, wherein the adder-subtractor circuit is configured to receive data from the second circuit at the root level of the MAC channel and write the data to the output buffer.

[0016] In some respects, the adder-subtractor circuit is also configured to: during a first computation iteration, write a first set of data received from a second circuit at the root level of the MAC channel into an output buffer; and during a second computation iteration: receive a temporary data set from the second circuit at the root level of the MAC channel, retrieve the first set of data from the output buffer, compute a second set of data based on the temporary data set, the first set of data, and a control signal indicating whether to perform a convolution computation or a vector operation, and write the second set of data into the output buffer.

[0017] In some respects, multiple MAC channels are configured to receive multiple weight vectors generated by a weight matrix generation circuit, so as to perform multiple vector operations in parallel.

[0018] In some aspects, a first subset of multiple MAC channels is configured to receive weights from a weight buffer, a second subset of multiple MAC channels is configured to receive weights generated by a weight matrix generation circuit, and the first subset of multiple MAC channels is also configured to perform convolution computations, the second subset of multiple MAC channels is also configured to perform vector operations, and the convolution computations and vector operations are performed in parallel.

[0019] In some aspects, the technology described herein relates to a hybrid convolution-vector operation processing system. This system may include: multiple weight selectors configured to acquire weights; multiple activation input interfaces configured to acquire activations; and multiple multiplier-accumulator (MAC) circuitry organized into multiple MAC channels. Each of the multiple MAC channels is configured to: receive a control signal indicating whether to perform a convolution operation or a vector operation; receive one or more weights from at least one of the multiple weight selectors according to the control signal; receive one or more activations from at least one of the multiple activation input interfaces according to the control signal; and generate output data based on one or more weights and one or more input activations according to the control signal, and feed the output data into an output buffer, wherein each of the multiple MAC channels includes multiple first circuitry for performing multiplication operations and multiple second circuitry for performing addition or subtraction operations according to the control signal.

[0020] These and other features of the systems, methods, and non-transitory computer-readable media disclosed herein, as well as the methods of operation and function of the related elements of the structure, the combination of components, and the economics of manufacture, will become more apparent upon consideration of the following description and appended claims with reference to the accompanying drawings, all of which form part of this specification, wherein like reference numerals denote corresponding parts in the various drawings. However, it should be clearly understood that the drawings are for illustration and description only and are not intended to be a definition of limitation of the invention. Attached Figure Description

[0021] Figure 1 An exemplary system diagram for processing neural network computations in a hybrid PE array is shown according to various embodiments.

[0022] Figure 2A An exemplary architecture diagram of a PE array according to various embodiments is shown.

[0023] Figure 2B An exemplary architecture diagram of a hybrid PE array for processing convolution computations and vector operations is shown according to various embodiments.

[0024] Figure 3 An exemplary internal structure diagram of a MAC (multiplier-accumulator) channel in a hybrid PE array according to various embodiments is shown.

[0025] Figure 4 Exemplary logic circuit designs for accumulators and subtractors in each MAC channel according to various embodiments are shown.

[0026] Figure 5A Exemplary use cases of performing vector reduction using a hybrid PE array according to various embodiments are shown.

[0027] Figure 5B Another exemplary use case is shown, which uses a hybrid PE array to perform multiple vector reductions in parallel according to various embodiments.

[0028] Figure 5C Another exemplary use case for performing pooling using a hybrid PE array is shown, according to various embodiments.

[0029] Figure 6 Exemplary system designs for hybrid PE arrays according to various embodiments are shown.

[0030] Figure 7 An example computer system in which any of the embodiments described herein can be implemented is shown. Detailed Implementation

[0031] The embodiments described herein provide hardware devices, systems, and PE (processing entity) arrays capable of performing various neural network computations in parallel by reusing hardware resources. Here, neural network computations involve convolution computations and vector operations such as vector reduction operations (e.g., reducing values ​​in a vector to maximum / minimum / sum / average, or reducing values ​​in a vector to maximum / minimum) and pooling operations (e.g., max pooling, average pooling), which constitute virtually all computations involved in neural network training and applications. In some embodiments, the hardware device described herein includes a PE array and other auxiliary logic circuitry, and is capable of handling different types of neural network computations by reusing the same hardware resources. For simplicity, the hardware device may be referred to as a hybrid PE array in the following design. In some embodiments, the auxiliary logic circuitry may include a weight matrix generation circuitry configured to generate function weights (compared to weights in the neural network).

[0032] While existing neural network (NN) hardware designs utilize SIMD vector processors or standalone / separate hardware (e.g., TPUs) to handle vector operations, in addition to configuring processors for performing convolution computations, the hybrid PE array described in this paper reuses the same hardware resources, such as multiplier-accumulators (MACs), for performing convolution operations, while providing scalable parallel processing capabilities for vector operations. Specifically, the MAC resources can be configured via control signals to perform summation or comparisons, where summation functions can be triggered to perform convolution computations, vector summation reduction (e.g., summing corresponding values ​​of multiple input vectors and generating an output vector from the sum), vector mean reduction (e.g., summing corresponding values ​​of multiple input vectors and then dividing the sum by the number of input vectors to generate an output vector), or average pooling (e.g., calculating the average of each patch of a feature map), and comparison functions can be triggered to perform vector maximum reduction, vector minimum reduction, or max pooling (e.g., finding the maximum value within each patch of a feature map), etc.

[0033] In the following description, specific, non-limiting embodiments of the invention will be described with reference to the accompanying drawings. Specific features and aspects of any of the embodiments disclosed herein may be used and / or combined with specific features and aspects of other embodiments disclosed herein. It should also be understood that these embodiments are merely examples and illustrate only a few embodiments within the scope of the invention. Various changes and modifications that will be apparent to those skilled in the art are considered to be within the spirit, scope, and contemplation of the invention as further defined in the appended claims.

[0034] Figure 1 An exemplary system diagram for processing neural network computations in a hybrid PE array 160 is shown according to various embodiments. Figure 1 The diagram illustrates a hardware architecture that can be configured to perform common neural network computations, such as convolution and vector operations, using the same hardware resources (e.g., a hybrid PE array 160). The embodiments described in this disclosure can be implemented as follows: Figure 1 Part of the neural network computation or other suitable environment.

[0035] Typical neural networks, such as convolutional neural networks (CNNs), may involve various computations, such as convolution operations and vector operations. For example, convolutional layers within a neural network (e.g., a CNN) can typically be convolved based on one or more input feature maps (IFMs) (including activations) obtained from an input source (e.g., such as an input image) or a previous layer (e.g., tensor outputs from a previous layer) and one or more weight tensors corresponding to a given layer (e.g., from a weight source 130, such as a weight cache or weight generator). The weight tensors can be used to convolve the IFMs to extract various features (e.g., convolution computations). The convolution process can be performed in parallel within a hybrid PE array 160. Each PE can refer to a processor with processing power and storage capacity (e.g., a buffer or cache). In some embodiments, each PE may include one or more logic gates or circuits configured as multipliers and accumulators (MACs). The PEs in the hybrid PE array 160 can be interconnected by wires and can be arranged into multiple channels called PE channels or MAC channels. The hybrid PE array 160 can be fabricated as a neural network accelerator or a data processing system.

[0036] As another example, pooling layers in CNNs can provide a method for downsampling the IFM by summarizing the presence of features in patches of feature maps. Common pooling methods, such as average pooling and max pooling, can summarize the average presence of features and the most active presence of features, respectively. These pooling operations may involve vector operations different from convolution computations. Specifically, convolution computations involve convolving a window through the IFM and performing multiplication and accumulation to extract features, while vector operations may involve comparisons or subtractions. Other types of vector operations are also common in neural networks, such as vector reduction operations. The vectors to be processed can be obtained from a vector memory (an input source 150), which can be the same source from which convolution activations are obtained. In some embodiments, the MAC in the hybrid PE array 160 can be configured to switch between a "convolution mode" and a "vector mode" to perform convolution computations and vector operations, serving convolutional layers and pooling layers, respectively.

[0037] The configuration of the hybrid PE array 160 can be based on control signals or instructions issued by the instruction decoder 110. The instruction decoder 110 can decode instructions from upper-level processors such as CPUs or GPUs. The instruction decoder 110 can send corresponding control signals to different components based on the decoded instructions. For example, the instruction decoder 110 can send an input activation / vector load control signal to the input source 150, indicating whether to acquire activation or vector for the hybrid PE array 160. The instruction decoder 110 can send a weight load control signal to the weight source 130, indicating whether weights should be obtained from the weight cache or from the weight matrix generator. Figure 2B(More details in [link]). Different weights play a crucial role in neural network computation. For example, weights retrieved from the weight cache might refer to weights from filters corresponding to convolutional layers, configured to extract features from the IFM. Weights generated from the weight matrix generator can be designed to perform specific types of vector operations. That is, the source of the weights may differ depending on whether the current operation is a convolution or a vector operation. As another example, if the weights have a certain pattern, the instruction decoder 110 can instruct the weight matrix generator to generate weights internally instead of retrieving them from the weight cache. This minimizes external memory access, thereby improving the overall performance of neural network computation.

[0038] The instruction decoder 110 can also send computation control signals to the hybrid PE array 160 to perform desired operations based on inputs from input source 150 and weights from weight source 130. The control signals can configure the MACs in the hybrid PE array 160 to perform summation (e.g., for convolution) or subtraction (e.g., for comparison in vector operations). After the hybrid PE array 160 completes computation according to the control signals, the output data can be fed into the output buffer 170 to store temporary activations or vectors. In some embodiments, the instruction decoder 110 can also control the output buffer to feed data from previous iterations back to the hybrid PE array 160 for participation in the current iteration.

[0039] Figure 2A An exemplary architecture diagram of a PE array according to various embodiments is shown. Figure 2A The arrangement of PEs in the PE array is for illustrative purposes and can be implemented in other ways depending on the use case.

[0040] like Figure 2AAs shown on the left, the PE array 200 may include a PE matrix. Each PE may include multiple multipliers (MUL gates). Multipliers within each PE can operate in parallel, and the PEs within the PE array 200 can operate in parallel. For ease of reference, the following description denotes the number of columns 220 of PEs in the PE array 200 as X, the number of rows 210 of PEs in the PE array 100 as Y2, and the number of multipliers within each PE as Y1. Each row 210 of a PE may be called a PE cluster, and each PE cluster may be coupled to Y1 adder trees 230 for aggregating the partial sums generated by the multipliers within the PE cluster. That is, the first multiplier in each PE within a PE cluster is coupled to the first adder tree 230 for aggregation, the second multiplier in each PE in a PE cluster is coupled to the second adder tree 230 for aggregation, and so on. The aggregation result of the adder trees 230 of all PE clusters (a total of Y1xY2 adder trees) can be fed into adder 250 for aggregation. Adder 250 may refer to digital circuitry that performs digital addition, which is part of a network-on-chip (NoC) subsystem.

[0041] Figure 2B An exemplary architecture diagram of a hybrid PE array for processing convolution computations and vector operations is shown according to various embodiments. This example diagram includes multiple hardware components that interact with each other, such as a weight matrix generator 270, or a weight cache configured to... Figure 2B (Not shown) Multiple weight selectors 272 for receiving weights, multiple MAC (multiplier-accumulator) channels 273 corresponding to the multiple weight selectors 272 respectively, and a buffer for receiving data from the multiple MAC channels 273 and outputting the data to the output buffer. Figure 2B Multiple MACs 274 (not shown). These components are for illustrative purposes only, and the architecture may include more, fewer, or alternative components depending on the implementation. For example, multiple MACs 274 may be implemented as the last MAC in multiple MAC channels, but with a different configuration than the other MACs in the MAC channels. The weight selector 272 can be implemented using a multiplexer coupled to the weight matrix generator 270 and the weight cache.

[0042] Figure 2B All components shown are transmitted from the instruction decoder (e.g. Figure 1The control signals of the instruction decoder 110 in the network are directly or indirectly controlled. These control signals may include weight load control and computation control, used to configure the hybrid PE array to perform convolution computations or vector operations, or both, using multiple MAC channels 273. For example, if the upper application layer instructs the execution of a convolution computation, the corresponding load control can be sent to multiple weight selectors 272, instructing them to select weights from the weight cache. The weights from the weight cache may come from filters in the convolutional layers of the neural network and can be used for convolution via IFM to extract features. If the upper application layer instructs the execution of a vector operation, such as vector reduction (e.g., vector averaging, which combines multiple vectors into a single vector with a mean), the weight load control can be sent to the weight matrix generator 270 to generate weights, and the weight selectors 272 can be configured to select the generated weights and block weights from the weight cache. These generated weights may functionally differ from the weights used in the convolution computation. For example, the generated weights may be used to calculate the average of the input vectors (e.g., for X vectors, each vector may be assigned a weight of 1 / X). In some embodiments, if the weights involved are highly organized (following a specific pattern), the weight matrix generator 270 can also be triggered to perform convolution calculations. The weight matrix generator 270 can be instructed to generate these weights according to a specific pattern of the MAC channel. This allows the MAC channel to avoid accessing the weight cache to retrieve weights, thereby avoiding memory access latency. The weight selector 272 can then forward the received weights to the MAC channel for corresponding calculations.

[0043] Similarly, multiple MAC channels 273 can be controlled by computation control signals. These signals can instruct the MAC channels to receive input activations for convolution computations or vectors for other vector operations. In some embodiments, some MAC channels (also referred to as a first subset of MAC channels) can be configured in parallel for convolution computations, while other MAC channels (also referred to as a second subset of MAC channels) can be configured for vector operations. This means that the hybrid PE array can perform convolution computations and vector operations simultaneously (e.g., during the same iteration). The computation control signals can also configure the MACs in each channel according to the specific workload to be performed. For example, for convolution computations on MAC channels, multipliers in the MAC channels can perform multiplication operations as part of the convolution computation, and accumulators within the MAC channels can perform summation operations. For vector reduction, such as maximum vector reduction or max pooling in a neural network pooling layer, multipliers in the MAC channels can perform multiplication (using weights generated by a weight matrix generator), and accumulators in the MAC channels can be configured to perform subtraction to implement comparison functionality. Comparisons may help determine the maximum value in each dimension of the input vector (for vector maximum reduction) or the maximum value within each patch of the feature map (for max pooling). In some embodiments, the accumulator in the MAC channel is designed to be configurable to perform summation or subtraction. More detailed circuit design for hybrid accumulators can be found in... Figure 4 Found it.

[0044] In some embodiments, the final MAC layer 274 can act as a bridge between the MAC channel and the output buffer. For example, MAC 274 can receive data from the MAC channel (e.g., partial summation, temporary activation, temporary vector output) and store the data in the corresponding output buffer. In some cases, after MAC 274 stores data in the output buffer in the first iteration, it can also read data from the output buffer, along with new data received from MAC channel 273, as part of the computation for the next iteration. Figure 3 An exemplary internal structure diagram of a MAC (multiplier-accumulator) channel 300 in a hybrid PE array according to various embodiments is shown. Figure 3The diagrams are for illustrative purposes only, and may include more, fewer, or alternative components depending on the implementation. As described above, each MAC channel 300 may include multiple multipliers 310 and accumulators, which can be implemented as digital gates or circuits (e.g., arithmetic logic units (ALUs)). Depending on the specific computation instructions from the upper layer, the accumulators can perform not only summation but also subtraction. In this way, the same piece of hardware (MAC channel 300) can be reused for different types of computations without the need for additional special processors (e.g., vector processors, TPUs). In this application, the accumulators may also be referred to as adder-subtractor circuits 320 because they have mixed functions: summation (addition) and comparison (subtraction). In some embodiments, the adder-subtractor circuit 320 in the MAC channel 300 can be organized as an adder tree 330. The leaf-level adder-subtractor circuits of the adder tree 330 can be configured to receive data from multiple multipliers 310. For example, each multiplier 310 can perform multiplication (for convolution or vector operations) based on an input vector / activation and weights. The multiplication results from two or more multipliers 310 can be fed into the leaf-level adder-subtractor circuits of the adder tree 330 to perform summation or subtraction (comparison) according to a control signal. For example, each adder-subtractor circuit 320 can be configured to receive a first input and a second input; determine whether to perform addition or subtraction based on a control signal; generate a sum or average of the first and second inputs in response to a control signal instructing the performance of addition; and generate a minimum or maximum value between the first and second inputs in response to a control signal instructing the performance of subtraction. If the control signal instructs the execution of vector minimum reduction, then the first input and the second input can each include vectors with the same number of dimensions, and the adder-subtractor circuit can generate an output vector that includes the minimum value of each corresponding dimension of the vector.

[0045] In some embodiments, the adder tree 330 may include multiple adder-subtractor circuits 320 at the leaf level, one adder-subtractor circuit 340 at the root level, and one or more intermediate levels. The number of adder-subtractor circuits is halved from one level to the next. In some embodiments, the root-level adder-subtractor circuit 340 may obtain the computation result of the adder tree 330 (e.g., the sum, minimum, or maximum value of a value or index) and send the result to an adder-subtractor circuit 350 outside the adder tree 330 corresponding to the MAC channel. The external adder-subtractor circuit 350 may resemble the adder-subtractor circuits within the adder tree 330, except that it can write to and read from the output buffer. In some embodiments, the external adder-subtractor circuit 350 and the root-level adder-subtractor circuit 340 may be the same circuit and are part of the adder tree 330.

[0046] In some embodiments, during a first computation iteration, the external adder-subtractor circuit 350 may be configured to write a first set of data received from the root-level adder-subtractor circuit into an output buffer. During a second computation iteration, the external adder-subtractor circuit 350 may be configured to again receive a temporary data set from the root-level adder-subtractor circuit 35, retrieve the first set of data from the output buffer, compute a second set of data based on the temporary data set, the first set of data, and a control signal indicating whether to perform a convolution computation or a vector operation, and write the second set of data into the output buffer.

[0047] Figure 4 Exemplary logic circuit designs 400 for accumulators and subtractors in each MAC channel according to various embodiments are shown. Design 400 illustrates one way of implementing a hybrid circuit that can be configured to perform addition or subtraction. Depending on the implementation, the hybrid circuit can be implemented using other digital components.

[0048] As shown in the figure, the example circuit includes multiple multiplexers 410 and 440 for selecting signals from multiple inputs based on control signals. The control signals may include a first signal 420 indicating whether addition or subtraction is being performed, and a second signal 430 indicating (if subtraction is being performed) whether the minimum or maximum value is being obtained from the input values. These signals control the selection logic of multiplexers 410 and 440 to select the appropriate input.

[0049] Figure 5A Exemplary use cases of performing vector reduction using a hybrid PE array are illustrated according to various embodiments. While hybrid PE arrays are used for convolution computations in convolutional layers of neural networks, they can also be used for other types of computations, such as vector reduction that requires different logical processing. Figure 5A The use case illustrated involves vector reduction along the channel dimension of the input tensor 520 that generates a single output vector 530. The single output vector 530 can include rows of vectors, denoted as V0(:0) to V0(:,z-1), where z is the channel dimension index. The weight matrix 510 used for vector reduction can be generated by a weight generation circuit. As shown, the weight matrix 510 can include a first row of 1s and all other elements of 0, while the input tensor 520 comprises multiple vectors. Note that even though... Figure 5AThe operator in this context is the multiplication operator, which also represents vector operators rather than matrix multiplication. Vector operators can be defined as part of the computation of control signals, and can include vector summation, vector minimization, or reduction, etc. For simplicity, the vector reduction operator can be represented as the "reduce to 0" function. Using the "reduce to 0" function, each column vector in the input tensor 520 can be used as a parameter of the "reduce to 0" function. For example, the first vector V0(:0) of a single output vector can be represented as reduce to 0(A(0,0), A(1,0), ..., A(y-1,0)), where y represents one of the row or column dimensions of the input tensor 520, and "reduce to 0" can be a sum, minimum, maximum, mean, etc. This example illustrates a simple application of hybrid PE arrays in channel-dimensional vector reduction.

[0050] Figure 5B Another exemplary use case is shown, which uses a hybrid PE array to perform multiple vector reductions in parallel according to various embodiments. Figure 5B The example demonstrates the high flexibility of the hybrid PE array due to the rich dynamics in weight matrix 540. Depending on the requirements of the upper-layer application, the weight matrix generation circuit can generate multiple weight rows in weight matrix 540 to achieve parallel vector reduction operations. Two weight rows can include different weights to achieve two different vector reductions. For example, row 0 in weight matrix 540, which includes all 1s, can correspond to vector summation reduction, which can be used to generate V0(:0) = sum(A(0,0), A(1,0), ..., A(y-1,0)) with input tensor 550, where y refers to one of the row or column dimensions of input tensor 520; and row 1, which includes all 1 / y, can correspond to vector mean reduction, which can be used to generate V1(:0) = sum(A(0,0), A(1,0), ..., A(y-1,0)) / y. Output vectors V0(:0) and V1(:0) can be stored in output vector tensor 560.

[0051] In some embodiments, the weight matrix 540 may include a first row of weights generated by a weight matrix generation circuit for vector operations, and a second row of weights retrieved from a weight cache for convolution calculations. Thus, a single weight matrix 540 can be used to trigger vector operations and convolution calculations in parallel. Specifically, Figure 5B The "multiplication" operator in the array can include an array of operators, including the vector reduction operator corresponding to the first row and the multiplication operator (for convolution) corresponding to the second row. This example shows that a hybrid PE array can be configured to perform multiple identical vector reductions, multiple different vector reductions, or a mixture of vector reductions and convolutions within a single computation cycle.

[0052] Figure 5CAnother exemplary use case for performing pooling using a hybrid PE array is shown, according to various embodiments. Figure 5C The use cases in this example involve pooling operations. Pooling operations are common in neural networks and are used to downsample feature maps by summarizing the presence of features in patches of feature maps (e.g., 3x3 patches). When patches are represented as vectors, pooling operations can be implemented as vector operations. For this purpose, the hybrid PE array can first convert the patches into vectors and then organize the vectors into an input tensor 580 according to the corresponding control signals. Typically, the pooling process involves convolving the patches (e.g., 3x3 patches) with activation tensors, but the convolution step may be smaller than the size of the patches. Therefore, the vectors in the input tensor 580 can have overlapping elements. The weight matrix 570 can include weights generated from the weight generation circuit, some of which can be configured to perform one type of pooling (e.g., summation pooling), while other weight rows can be configured to perform another type of pooling (e.g., max pooling). Using the weight matrix 570 and the input tensor 580, multiple pooling computations can be performed using the same PE array within the same computation cycle. The output vector can be stored in the output vector tensor 590.

[0053] Figure 6 An exemplary system design of a hybrid PE array according to various embodiments is shown. The hybrid PE array can be implemented as a hardware accelerator 600. Figure 6 The components of accelerator 600 shown are for illustrative purposes only. Depending on the implementation, accelerator 600 may include more, fewer, or alternative components. In some embodiments, accelerator 600 may include... Figure 1 , Figure 2A and Figure 2B All or some components, such as MAC channels. Each MAC channel in accelerator 600 may include a tree of multiple multipliers and adders, such as... Figure 3 As shown. Figure 4 As shown, an adder tree can include multiple multi-functional adders-subtractors.

[0054] From a functional perspective, in some embodiments, accelerator 600 may include a weight selection circuit 610, an activation selection circuit 620, multiple MAC channels 630, and a weight matrix generation circuit 640. In some embodiments, the weight selection circuit 610 may be implemented as a multiplexer and coupled to the weight matrix generation circuit 640 and a weight buffer, the weight buffer representing two weight sources. The weight selection circuit 610 may be instructed to obtain weights from these two weight sources according to a control signal. For different types of computations, weights may be obtained from different sources. The activation selection circuit 620 may be configured to obtain activations or vectors based on a target computation (e.g., convolution, pooling, vector operations) according to another control signal.

[0055] In some embodiments, each MAC channel 630 may be configured to receive a control signal indicating whether to perform a convolution operation or a vector operation; receive one or more weights from at least one of a plurality of weight selectors according to the control signal; receive one or more activations from at least one of a plurality of activation input interfaces according to the control signal; and generate output data based on one or more weights and one or more input activations according to the control signal, and feed the output data into an output buffer, wherein: each of the plurality of MAC channels includes a plurality of first circuits for performing multiplication operations according to the control signal and a plurality of second circuits for performing addition or subtraction operations.

[0056] In some embodiments, the weight matrix generation circuit 640 can be configured to generate weights for vector reduction operations, wherein the vector reduction operations include one or more of reduction mean, reduction minimum, reduction maximum, reduction average, reduction summation, or pooling.

[0057] Figure 7 An example computing device is shown in which any of the embodiments described herein can be implemented. The computing device can be used to implement... Figures 1-6 The system and method shown may include one or more components. The computing device 700 may include a bus 702 or other communication mechanism for transmitting information, and one or more hardware processors 704 coupled to the bus 702 for processing information. The hardware processor 704 may be, for example, one or more general-purpose microprocessors.

[0058] The computing device 700 may also include main memory 707, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 702, for storing information and instructions to be executed by processor 704. Main memory 707 may also be used to store temporary variables or other intermediate information during instruction execution by processor 704. When stored in a storage medium accessible to processor 704, such instructions can present the computing device 700 as a special-purpose machine customized to perform the operations specified in the instructions. Main memory 707 may contain non-volatile and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common forms of media may include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROMs and EPROMs, FLASH-EPROMs, NVRAM, any other memory chips or cassette tapes, or network versions thereof.

[0059] The computing device 700 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic. This logic, combined with the computing device, can make the computing device 700 a dedicated machine or programmable thereto. According to one embodiment, the techniques herein are executed by the computing device 700 in response to processor 704 executing one or more sequences of one or more instructions contained in main memory 707. Such instructions may be read into main memory 707 from another storage medium, such as storage device 708. Execution of the sequence of instructions contained in main memory 707 can cause processor 704 to perform the processing steps described herein. For example, the processes / methods disclosed herein can be implemented by computer program instructions stored in main memory 707. When these instructions are executed by processor 704, they can perform the steps as shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0060] The computing device 700 also includes a communication interface 710 coupled to the bus 702. The communication interface 710 can provide bidirectional data communication coupling with one or more network links connected to one or more networks. As another example, the communication interface 710 can be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible device (or a WAN component communicating with a WAN). A wireless link can also be implemented.

[0061] The execution of certain operations may be distributed across processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0062] Each process, method, and algorithm described in the preceding chapters can be embodied in a code module executed by one or more computer systems or computer processors that include computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, partially or entirely, in dedicated circuitry.

[0063] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. The software product can be stored in a storage medium including multiple instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, disk, optical disk, another medium operable for storing program code, or any combination thereof.

[0064] Specific embodiments also provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to the steps in any of the methods of the above embodiments. Specific embodiments also provide a non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause one or more processors to perform operations corresponding to the steps in any of the methods of the above embodiments.

[0065] The embodiments disclosed herein can be implemented through a cloud platform, server, or group of servers (collectively referred to as the "service system") that interacts with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0066] The various features and processes described above can be used independently of each other or in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this invention. Furthermore, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are not limited to any particular sequence, and associated blocks or states can be executed in other suitable sequences. For example, described blocks or states may be executed in a different order than that specified in the disclosure, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently from those described herein. For example, elements may be added, removed, or rearranged compared to the disclosed example embodiments.

[0067] Various operations of the exemplary methods described herein can be performed at least in part by an algorithm. This algorithm may include program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly program the computer to perform the function, but may learn from training samples to build a predictive model that performs the function.

[0068] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute the engine of a processor implementation for performing one or more of the operations or functions described herein.

[0069] Similarly, the methods described herein can be implemented at least in part by a processor, with one or more specific processors being examples of hardware. For example, at least some operations of a method can be executed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also support the execution of the associated operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some operations can be executed by a set of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0070] The execution of certain operations may be distributed across processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0071] In this specification, multiple instances can implement components, operations, or structures described as a single instance. While individual operations of one or more methods are shown and described as separate operations, one or more separate operations may be performed concurrently, and the order in which they are performed is not required. Structures and functionalities presented as separate components in the example configuration can be implemented as composite structures or components. Similarly, structures and functionalities presented as single components can be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of this document's subject matter.

[0072] The word “or” as used herein is inclusive, not exclusive, unless otherwise expressly stated or the context otherwise indicates. Therefore, unless otherwise expressly stated or the context otherwise indicates, “A, B, or C” herein means “A, B, A and B, A and B, A and C, B and C, or A, B, and C.” Furthermore, unless otherwise expressly stated or the context otherwise indicates, “and” is both common and separate. Therefore, unless otherwise expressly stated or the context otherwise indicates, “A and B” herein means “A and B, commonly or separately.” Furthermore, multiple instances of the resources, operations, or structures described herein may be provided as a single instance. Moreover, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are described in the context of a particular illustrative configuration. Other functional assignments are contemplated, and they may fall within the scope of various embodiments of the invention. Generally, structures and functions presented as separate resources in the example configuration can be implemented as composite structures or resources. Similarly, structures and functions presented as single resources can be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of embodiments of the invention as represented by the appended claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0073] The terms “comprising” or “including” are used to indicate the presence of a subsequently declared feature, but do not preclude the addition of other features. Unless otherwise specifically stated or understood in the context of use, conditional language such as “can,” “should,” “may,” or “may” is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that features, elements, and / or steps are necessary in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether such features, elements, and / or steps are included or whether they should be performed in any particular embodiment, regardless of user input or prompting.

[0074] Although an overview of the subject matter has been described with reference to specific example embodiments, various modifications and changes can be made to these embodiments without departing from the broader scope of embodiments of the invention. For convenience only, the term "invention" may be used herein to refer to these embodiments of the subject matter, individually or collectively, and if multiple embodiments are indeed disclosed, it is not intended to voluntarily limit the scope of this application to any single disclosure or concept.

[0075] The embodiments illustrated herein have been described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, allowing for structural and logical substitutions and changes without departing from the scope of the invention. Therefore, the detailed description should not be construed as limiting, and the scope of the various embodiments is defined only by the appended claims and the full scope of their equivalents.

Claims

1. A neural network accelerator, comprising: The instruction decoder is configured to decode neural network computation instructions from the processor into weight load control signals, activation load control signals, and computation control signals; Multiple weight selectors are configured to obtain weights according to the weight load control signal, wherein the weight load control signal indicates whether the weights are obtained from a weight cache or from a weight generator; Multiple activation input interfaces are configured to obtain an activation or a vector from a memory based on the activation load control signal, wherein the activation load control signal indicates whether to obtain the activation or the vector; as well as Multiple circuit channels, each of which is configured as follows: Receive the weights obtained by the plurality of weight selectors and the activations or vectors obtained by the plurality of activation selectors. The calculation control signal determines whether to perform a convolution operation or a vector operation, and The convolution or vector operation is performed based on the weights and the activation or vector to generate output data; and The instruction decoder is further configured to, in response to the weight having a pattern, instruct the plurality of weight selectors to obtain the weight from the weight generator instead of from the weight cache, in order to reduce memory access.

2. The neural network accelerator as described in claim 1, wherein, The weight cache includes the weights of the filters corresponding to the convolutional layers in the neural network used for convolution operations, and the weight generator generates weights to perform vector operations.

3. The neural network accelerator as described in claim 1, wherein, The weighted load control signal indicates whether the current operation is a convolution operation or a vector operation, and In response to the current operation being a convolution operation, the plurality of weight selectors obtain the weights from the weight cache; as well as In response to the current operation being a vector operation, the plurality of weight selectors obtain the weights from the weight generator.

4. The neural network accelerator as described in claim 1, wherein: The calculation control signal instructs the plurality of circuit channels to perform convolution operations (summation) or vector operations (subtraction).

5. The neural network accelerator as claimed in claim 1, wherein: According to the computation control signal, a subset of the plurality of circuit channels is configured to perform convolution operations, and at the same time, the remaining circuit channels are configured to perform vector operations.

6. The neural network accelerator as claimed in claim 1, wherein, Each of the aforementioned circuit channels is organized as a tree. The circuitry at the leaf level of the tree is configured to perform multiplication based on one or more of the weights and one or more of the activations or the vectors, and The circuitry at the non-leaf level of the tree is configured to perform addition or subtraction based on data received from the circuitry of the previous level of the tree, as instructed by the computation control signal.

7. The neural network accelerator of claim 1, wherein, The vector operations include one or more of the following: reduced mean, reduced minimum, reduced maximum, reduced average, reduced summation, or pooling.

8. The neural network accelerator of claim 1, wherein, Each of the weight selectors includes a multiplexer coupled to the weight generator and the weight cache.

9. A hybrid convolution-vector operation processing system, comprising: The instruction decoder is configured to decode neural network computation instructions from the processor into weight load control signals and computation control signals; Multiple weight selectors are configured to obtain weights according to the weight load control signal, wherein the weight load control signal indicates whether the weights are obtained from a weight cache or from a weight generator; Multiple activation selectors are configured to obtain activations or vectors from memory; as well as Multiple circuit channels, each of which is configured as follows: Receive the weights obtained by the plurality of weight selectors and the activations or vectors obtained by the plurality of activation selectors. The calculation control signal determines whether to perform a convolution operation or a vector operation, and The convolution or vector operation is performed based on the weights and the activation or the vector to generate output data; and The instruction decoder is further configured to, in response to the weight having a pattern, instruct the plurality of weight selectors to obtain the weight from the weight generator instead of from the weight cache, in order to reduce memory access.

10. The hybrid convolution-vector operation processing system as described in claim 9, wherein, The weight cache includes the weights of the filters corresponding to the convolutional layers in the neural network used for convolution operations, and the weight generator generates weights to perform vector operations.

11. The hybrid convolution-vector operation processing system as described in claim 9, wherein, The weighted load control signal indicates whether the current operation is a convolution operation or a vector operation, and In response to the current operation being a convolution operation, the plurality of weight selectors obtain the weights from the weight cache; as well as In response to the current operation being a vector operation, the plurality of weight selectors obtain the weights from the weight generator.

12. The hybrid convolution-vector operation processing system as described in claim 9, wherein, The calculation control signal instructs the plurality of circuit channels to perform convolution operations (summation) or vector operations (subtraction).

13. The hybrid convolution-vector operation processing system as described in claim 9, wherein, According to the computation control signal, a subset of the plurality of circuit channels is configured to perform convolution operations, and at the same time, the remaining circuit channels are configured to perform vector operations.

14. The hybrid convolution-vector operation processing system as described in claim 9, wherein, Each of the aforementioned circuit channels is organized as a tree. The circuitry at the leaf level of the tree is configured to perform multiplication based on one or more of the weights and one or more of the activations or the vectors, and The circuitry at the non-leaf level of the tree is configured to perform addition or subtraction based on data received from the circuitry of the previous level of the tree, as instructed by the computation control signal.

15. The hybrid convolution-vector operation processing system as described in claim 9, wherein, The vector operations include one or more of the following: reduced mean, reduced minimum, reduced maximum, reduced average, reduced summation, or pooling.

16. The hybrid convolution-vector operation processing system as described in claim 9, wherein, Each of the weight selectors includes a multiplexer coupled to the weight generator and the weight cache.

17. A computer-implemented method, comprising: Receive neural network computation instructions from the processor; The neural network calculation instructions are decoded into weight load control signals, activation load control signals, and calculation control signals using an instruction decoder. The weight load control signal is sent to a plurality of weight selectors to obtain weights, wherein the weight load control signal indicates to the plurality of weight selectors whether to obtain the weights from a weight cache or from a weight generator; The activation load control signal is sent to a plurality of activation selectors to obtain an activation or a vector from memory, wherein the activation load control signal indicates whether to obtain the activation or the vector; The calculation control signal is sent to multiple circuit channels, wherein the multiple circuit channels are configured according to the calculation control signal as follows: Receive the weights obtained by the plurality of weight selectors and the activations or vectors obtained by the plurality of activation selectors. The calculation control signal determines whether to perform a convolution operation or a vector operation, and The convolution or vector operation is performed based on the weights and the activation or the vector to generate output data; and The instruction decoder is further configured to, in response to the weight having a pattern, instruct the plurality of weight selectors to obtain the weight from the weight generator instead of from the weight cache, in order to reduce memory access.

18. The computer-implemented method of claim 17, wherein, The weighted load control signal indicates whether the current operation is a convolution operation or a vector operation, and In response to the current operation being a convolution operation, obtaining the weights includes obtaining the weights from the weight cache; as well as In response to the current operation being a vector operation, obtaining the weights includes obtaining the weights from the weight generator.

Images