Parallel depth processing architecture for neural networks

Abstract

Methods and apparatus for performing machine learning tasks, in particular neural network processing architectures and circuits for improving performance through depth-parallel processing. One exemplary neural network processing circuit generally includes multiple groups of processing element (PE) circuits, each group of PE circuits including multiple PE circuits configured to process inputs in parallel at multiple depths.

Description

[Technical Field]

[0001] Cross-referencing of related technologies This application claims priority to U.S. Patent Application No. 17 / 465,550, filed 2 September 2021, which is incorporated herein by reference in its entirety for all applicable purposes.

[0002] The aspects of this disclosure relate to machine learning, and more particularly to circuits, neural network processing architectures, and techniques for unit-depth parallel processing for performing machine learning tasks. [Background technology]

[0003] Machine learning is generally the process of generating a trained model (e.g., an artificial neural network, tree, or other structure) that represents a generalized fit to a set of training data. By applying the trained model to new data, inferences are generated, and these inferences can be used to gain insights into the new data. In some cases, applying a model to new data is described as "performing inferences" on that new data.

[0004] The rapid increase in the use of machine learning (or artificial intelligence) to enable a variety of machine learning (or artificial intelligence) tasks has led to a growing demand for more efficient processing of machine learning model data. In some cases, dedicated hardware such as machine learning accelerators can be used to enhance the capabilities of processing systems for processing machine learning model data. However, such hardware requires space and power, and it is not always available on processing devices. For example, "edge processing" devices such as mobile devices, always-on devices, and Internet of Things (IoT) devices typically have to balance processing power with power and packaging constraints. Furthermore, accelerators may move data over a common data bus, which can lead to significant power consumption and introduce latency for other processes sharing the data bus. Therefore, other forms of processing systems for processing machine learning model data are being explored. [Overview of the project]

[0005] The systems, methods, and devices of this disclosure each have several embodiments, and not just one of these embodiments represents their desirable attributes. Several features, without limiting the scope of this disclosure, as expressed in the following claims, are briefly discussed below. After reviewing this discussion, and especially after reading the section titled "Modes for Carrying Out the Invention," you will understand how the features of this disclosure provide the advantages described herein.

[0006] Certain aspects of this disclosure relate to processing circuits, which generally comprise a plurality of groups of processing element (PE) circuits. Each group of PE circuits comprises a plurality of PE circuits configured to process inputs in parallel at a plurality of depths. Each PE circuit generally comprises one or more multiplier circuits and a local accumulator having inputs coupled to the outputs of the one or more multiplier circuits. Each multiplier circuit may be configured to calculate partial products, and the local accumulator may be configured to produce a sum from the partial products calculated by each of the one or more multiplier circuits.

[0007] Certain aspects of this disclosure relate to a method for neural network processing, which generally includes receiving an input for processing. For each of several segments of the received input, intermediate outputs for each of several depths in the neural network are generated substantially in parallel, based on the weights in the neural network associated with each depth. Each intermediate output for each depth is accumulated to a final output. At least the final output is output to a memory bus.

[0008] Other embodiments provide a processing system configured to perform the methods described above and the methods described herein; a non-temporary computer-readable medium containing instructions, when executed by one or more processors of the processing system, that cause the processing system to perform the methods described above and the methods described herein; a computer program product, embodied on a computer-readable storage medium, containing code for performing the methods described above and the methods described herein; and a processing system comprising means for performing the methods described above and the methods described herein.

[0009] To achieve the above and related objectives, one or more embodiments include features that are fully described below and, in particular, pointed out in the claims. The following description and accompanying drawings detail specific exemplary features of one or more embodiments. However, these features represent only a small number of the various methods that can employ the principles of the various embodiments. [Brief explanation of the drawing]

[0010] A more detailed explanation of the features of this disclosure, as described above, can be obtained by referring to the embodiments partially shown in the accompanying drawings. However, it should be noted that the accompanying drawings only illustrate certain typical embodiments of this disclosure and should not be considered to limit the scope of this disclosure, as other similarly effective embodiments may be recognized in this description.

[0011] [Figure 1A] This figure shows various types of neural network embodiments that may be implemented according to the embodiments of this disclosure. [Figure 1B] This figure shows various types of neural network embodiments that may be implemented according to the embodiments of this disclosure. [Figure 1C] This figure shows various types of neural network embodiments that may be implemented according to the embodiments of this disclosure. [Figure 1D] This figure shows various types of neural network embodiments that may be implemented according to the embodiments of this disclosure. [Figure 2] This figure shows one embodiment of a conventional convolution operation that can be implemented according to various aspects of this disclosure. [Figure 3] This is a block diagram of an exemplary neural network processing architecture having tightly coupled memory (TCM) and processing elements (PE) in which a particular aspect of the present disclosure may be implemented. [Figure 4] This diagram illustrates the use of processing elements in a neural network processing architecture where inputs of different depths are processed sequentially. [Figure 5] FIG. is a diagram showing an exemplary neural network processing architecture in which a plurality of processing element groups are configured to process a plurality of depths of a plurality of inputs in parallel according to a particular aspect of the present disclosure. [Figure 6] FIG. is an exemplary data flow timing diagram for processing different depths of a plurality of inputs in parallel according to a particular aspect of the present disclosure. [Figure 7] FIG. is a block diagram of an exemplary circuit related to parallel processing of input depth units according to a particular aspect of the present disclosure. [Figure 8] FIG. shows the utilization of processing elements in a neural network processing architecture in which different depths of inputs are processed in parallel. [Figure 9] FIG. is a diagram showing exemplary operations that can be performed by a processing system to process a plurality of depths of an input in parallel according to a particular aspect of the present disclosure. [Figure 10] FIG. is a block diagram showing an exemplary electronic device having a neural network processing circuit in which a plurality of depths of an input are processed in parallel according to a particular aspect of the present disclosure.

[0012] For ease of understanding, the same reference numbers are used to designate the same elements common to the drawings, where possible. It is contemplated that the elements and features of one aspect can be beneficially incorporated into other aspects without further elaboration of the elements and features of one aspect. DETAILED DESCRIPTION OF THE INVENTION

[0013] Aspects of the present disclosure provide an apparatus, method, processing system, and computer-readable medium for processing a plurality of depths of an input in parallel.

[0014] Brief Introduction to Neural Networks, Deep Neural Networks, and Deep Learning A neural network is organized into layers of interconnected nodes. Generally, a node (or neuron) is where computations are performed. For example, a node can combine input data with a set of weights (or coefficients) that amplify or attenuate that input data. Thus, the amplification or attenuation of an input signal can be seen as assigning relative importance to various inputs related to the task the network is trying to learn. Generally, the product of the input and the weights is summed (or accumulated), and this sum is then passed through the node's activation function to determine whether and to what extent the signal should proceed further through the network.

[0015] In most basic implementations, a neural network may have an input layer, a hidden layer, and an output layer. "Deep" neural networks generally have two or more hidden layers.

[0016] Deep learning is a method for training deep neural networks. Generally, deep learning maps inputs to a network to outputs from the network, and therefore, it is sometimes called a "universal approximator" because it can learn to approximate an unknown function f(x)=y between any input x and any output y. In other words, deep learning finds the correct f to transform x into y.

[0017] More specifically, deep learning trains each layer of a node based on a distinct set of features that are outputs from the preceding layer. Therefore, as each layer of a deep neural network progresses, the features become more complex. This is why deep learning is powerful: it learns to represent the input at a successively higher level of abstraction in each layer, thereby constructing a useful feature representation of the input data, and thus progressively extracting higher-level features from the input data to perform complex tasks such as object recognition.

[0018] For example, when presented with visual data, the first layer of a deep neural network can learn to recognize relatively simple features in the input data, such as edges. In another embodiment, when presented with auditory data, the first layer of a deep neural network can learn to recognize spectral power at specific frequencies in the input data. Then, a second layer of the deep neural network can learn, based on the output of the first layer, to recognize combinations of features, such as simple shapes in the visual data or combinations of sounds in the auditory data. Subsequently, higher layers can learn to recognize complex shapes in the visual data or words in the auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases. Therefore, deep learning architectures can work particularly well when applied to problems with natural hierarchical structures.

[0019] Layered coupling in neural networks Neural networks, such as deep neural networks (DNNs), may be designed using various coupling patterns between layers.

[0020] Figure 1A shows an embodiment of a fully connected neural network 102. In the fully connected neural network 102, each node in the first layer transmits its output to all nodes in the second layer, so that each node in the second layer receives input from all nodes in the first layer.

[0021] Figure 1B shows an embodiment of a locally connected neural network 104. In the locally connected neural network 104, nodes in the first layer may be connected to a limited number of nodes in the second layer. More generally, the locally connected layers of the locally connected neural network 104 may be configured such that each node in the layer has the same or similar connectivity pattern but with different connectivity strengths (or weights) (for example, values ​​associated with local regions 110, 112, 114, and 116 of the nodes in the first layer). Since the nodes in the upper layers within a given region may receive inputs that have been tuned through training to the characteristics of a limited portion of all inputs to the network, the locally connected connectivity patterns may result in spatially distinct receptive fields in the upper layers.

[0022] One type of locally connected neural network is a convolutional neural network (CNN). Figure 1C shows an example of a convolutional neural network 106. The convolutional neural network 106 may be configured such that the connection strength associated with the input for each node in the second layer (for example, for a local region 108 superimposed on another local region of a node in the first layer) is shared. Convolutional neural networks are particularly well-suited for problems where the spatial location of the input is meaningful.

[0023] One type of convolutional neural network is the deep convolutional network (DCN). A deep convolutional network is a network of multiple convolutional layers, which may be further constructed using, for example, pooling layers and normalization layers.

[0024] Figure 1D shows an embodiment of the DCN100 designed to recognize visual features in an image 126 generated by an image capture device 130. For example, if the image capture device 130 is a camera mounted inside or on a vehicle (or otherwise moving with the vehicle), the DCN100 may be trained using various supervised learning techniques to identify traffic signs and even numbers on traffic signs. Similarly, the DCN100 may also be trained for other tasks, such as identifying lane markings or traffic signals. These are just some exemplary tasks, and many other tasks are possible.

[0025] In the embodiment shown in Figure 1D, DCN100 includes a feature extraction section and a classification section. Upon receiving image 126, the convolutional layer 132 applies a convolutional kernel (e.g., as shown and described in Figure 2) to image 126 to generate a first set of feature maps 118 (or intermediate activations). Generally, a “kernel” or “filter” includes a multidimensional array of weights designed to highlight various aspects of the input data channel. In various embodiments, “kernel” and “filter” may be used interchangeably to refer to a set of weights applied in a convolutional neural network.

[0026] Next, the first set of feature maps 118 can be subsampled by a pooling layer (e.g., a max pooling layer, not shown) to generate a second set of feature maps 120. The pooling layer can reduce the size of the first set of feature maps 118 while retaining much of the information to improve the model's performance. For example, the second set of feature maps 120 can be downsampled by the pooling layer from a 28x28 matrix to a 14x14 matrix.

[0027] This process can be repeated through many layers. In other words, the second set of feature maps 120 can be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0028] In the embodiment shown in Figure 1D, a second set 120 of feature maps is provided to a fully connected layer 124, which then generates an output feature vector 128. Each feature in the output feature vector 128 may contain a number corresponding to a possible feature of image 126, such as "label", "60", and "100". In some cases, a softmax function (not shown) can be used to convert the numbers in this output feature vector 128 into probabilities. In such cases, the output 122 of DCN 100 is the probability that image 126 contains one or more features.

[0029] The softmax function (not shown) can convert the individual elements of the output feature vector 128 into probabilities such that the output 122 of DCN100 represents one or more probabilities that image 126 contains one or more features, such as a marker with the number "60" above it, as in image 126. Therefore, in this embodiment, the probabilities in output 122 for "marker" and "60" should be higher than the probabilities of other elements of output 122, such as "30", "40", "50", "70", "80", "90", and "100".

[0030] Before training DCN100, the output 122 generated by DCN100 may be inaccurate. Therefore, the error between output 122 and a priori known target output can be calculated. For example, in this case, the target output is the instruction that image 126 contains the label and the digit "60". Then, using this known target output, the weights of DCN100 can be adjusted through training so that subsequent outputs 122 of DCN100 achieve that target output (with high probability).

[0031] To adjust the weights of DCN100, a learning algorithm can compute a gradient vector for those weights. This gradient vector may indicate the amount by which the error will increase or decrease if the weights are adjusted in a particular manner. The weights can then be adjusted to reduce the error. This method of adjusting the weights is sometimes called "backpropagation" because this adjustment process involves a "backward path" through the layers of DCN100.

[0032] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate for the entire system stabilizes or until the error rate reaches a target level.

[0033] After training, DCN100 can be presented with a new image, and DCN100 can generate inferences such as the classification or probability of various features present in the new image.

[0034] Convolutional techniques related to convolutional neural networks Convolution is generally used to extract useful features from an input dataset. For example, in a convolutional neural network as described above, convolution allows for the extraction of various features using kernels and / or filters whose weights are automatically learned during training. Inference is then performed by combining the extracted features.

[0035] An activation function can be applied before and / or after each layer of a convolutional neural network. An activation function is generally a mathematical function that determines the output of a node in a neural network. Therefore, the activation function determines whether a node should pass information based on whether the node's input is related to the model's prediction. In one embodiment, if y = conv(x) (i.e., y is a convolution of x), both x and y can generally be considered "activators". However, with respect to a particular convolution operation, x may also be referred to as a "pre-activation" or "input activation" because x exists before that particular convolution, and y may be referred to as an output activation or feature map.

[0036] Figure 2 illustrates a three-dimensional convolution in which the input 210 is organized in three-dimensional space and processed using multiple convolution kernels 220 to generate the output 230. The input 210 can be a three-dimensional input such as a three-dimensional image having spatial data in each of the three dimensions (e.g., X (horizontal) dimension, Y (vertical) dimension, and Z (depth) dimension), or a sequence of two-dimensional images over time (having X (horizontal) and Y (vertical) dimensions) (e.g., video) where time can correspond to the Z (depth) dimension. In this particular embodiment, a 12-pixel × 12-pixel image having some number n depth cycles can be processed through corresponding convolution kernels 220 having dimensions of 3-pixel × 3-pixel × n depth cycles. The output 230 may have the same size along the X and Y dimensions and a depth dimension corresponding to the number of convolution kernels 220 used to process the input 210.

[0037] Output retention techniques, in which the output of a neural network is stored and maintained in memory associated with a processing element, can enable the rapid generation of output 230. Generally, using output retention techniques involves placing depth cycles (corresponding to different kernels from multiple convolutional kernels 220) within the inner loop of a multiloop logic structure, while the outer loop of the multiloop logic structure is used for iteration across the X and Y dimensions. However, output retention techniques can allow the reuse of accumulator circuits across depth cycles with respect to a given input in the X and Y dimensions. However, each depth cycle generally involves reloading the weights associated with the depth cycle into the processing element used to process the depth cycle. Furthermore, when a kernel is updated, the inputs in the X and Y dimensions may be reloaded and processed within one or more processing elements. Some processing techniques can support weight retention processing of inputs. In such cases, the inputs in the X and Y dimensions may be iterated over within the inner loop of the multiloop structure.

[0038] Exemplary neural network processing architecture and dataflow Figure 3 is a block diagram of an exemplary neural network processing architecture 300, showing an exemplary data flow sequence in which a particular aspect of the present disclosure may be implemented. The neural network processing architecture 300 may include a plurality of processing elements (PEs) 302 for performing data computations (e.g., multiply-accumulate (MAC) operations) and other operations. The PEs 302 can be implemented using any of a variety of suitable circuits, such as a neural processing unit (NPU), a neural signal processor (NSP), or a compute-in-memory (CIM) circuit. The processing architecture 300 may also include a global memory 304 (labeled “global buffer”), a tightly coupled memory (TCM) 306, an activation buffer 308, a weight buffer 310, a digital post-processing module 312, a memory bus 314, a register bus 316, and a PE bus 318. As used herein, TCM generally refers to memory accessed by a dedicated connection from a processor such as a PE 302. The activation buffer 308 and the weight buffer 310 may each be configured as a set of register trays. The data stored in the weight buffer 310 may be broadcast to parallel inputs (for example, inputs processed in parallel by PE 302). The memory bus 314 can connect the global memory 304 to the TCM 306. The register bus 316 can connect the TCM 306 to the activation buffer 308, the weight buffer 310, and the digital post-processing module 312. The PE bus 318 can connect the PE 302 to the activation buffer 308, the weight buffer 310, and the digital post-processing module 312. In this way, the PE 302 can access various memory resources to read data from the activation buffer 308 and / or the weight buffer 310, and output the results of processing the data read from the activation buffer 308 and / or the weight buffer 310 to the digital post-processing module 312.

[0039] Example parallel processing of input depth Parallel processing generally allows multiple parts or segments of an input to be processed substantially simultaneously by a processing unit such as a neural signal processor (NSP) or neural processing unit (NPU). Generally, by leveraging the independence of various inputs, parallel processing can accelerate the processing of a set of inputs and complete the execution of operations on the set of inputs using fewer computational resources (e.g., time) and / or using computational resources more efficiently (e.g., in proportion to the total computational resources available in the computing system) than would be used in sequential processing of the set of inputs. Typically, an NSP or NPU supports input parallel processing on various parts of an input (e.g., defined chunks of a two-dimensional input, such as blocks of pixels in an image), and these parts of the input may be designed to be processed sequentially at different depths (e.g., different channels or times) (e.g., using different kernel (depth) cycles, as discussed above with respect to Figure 2). That is, for input sub-indexes {1,2,3,...,n} and depth m, these NSPs or NPUs may be designed to process values ​​at input sub-indexes {1,2,3,...,n} at depth 1, then at depth 2, and so on, up to depth m. Since these NSPs or NPUs may be designed to support depth-sequential processing, their capabilities may not be fully utilized. Furthermore, increasing the depth of the neural network may exacerbate the effects of underutilized NSP or NPU processing capabilities, and therefore, as the depth of the neural network increases, it negatively impacts the performance metrics of these NSPs or NPUs. For example, assuming some maximum number of operations per second that an NSP or NPU can support when all processing elements (PEs) within the NSP or NPU are used, the number of operations per second that an NSP or NPU can support when using depth-sequential processing may be significantly lower.

[0040] To enable improved performance, processing elements within an NSP or NPU may be designed to support increased activation depth. However, increasing the activation depth supported by processing elements may reduce mapping flexibility, decrease clock frequency, and / or reduce the number of inputs that can be processed in parallel using the same hardware resources.

[0041] Figure 4 shows one embodiment 400 of the use of processing elements in a neural network processing architecture in which inputs are processed sequentially at different depths within the neural network. In embodiment 400, the architecture can support input parallelism and filter channel parallelism so that multiple segments of the input (e.g., in the X and Y dimensions as shown in Figure 2) can be processed in parallel using multiple filters in parallel, but it cannot support depth parallelism where the same segment of the input can be processed at different depths within the neural network (e.g., using different kernels). As shown, the architecture of embodiment 400 may have 32 computing units (labeled "CU-1" to "CU-32"), each of which may contain 64 processing elements (PEs) that can be used to process parts of the input provided to the architecture in parallel. Each PE may be configured to process 4 activations and 32 filter channels. In this embodiment, a layer of the ResNet50 network may have an input tensor of size 7×7×5¹² and several corresponding kernel tensors of size 3×3×5¹². For any 7×7 input channel, 25 operations can be performed to convolve 25 distinct but potentially overlapping receptive fields with a 3×3 kernel using a stride distance of 1. In this embodiment, a 5×5 feature map can be generated as the output of this layer. Input segments 1-25 can be processed through corresponding processing elements in the architecture's computing units 1-16, as shown in the figure. However, since the architecture cannot support depth-based parallelism, computing resources that could be used to process the input at a different depth may instead be left unused.

[0042] Therefore, in Example 400, a significant amount of computing resources may not be used, and the performance may not be scalable. For example, the resource utilization can be calculated as the product of the number of computing units on which the problem is being executed, the number of inputs, the number of activations, and the number of filter channels. In this example, as discussed, the problem can be defined according to the parameters depth = 512, input X = 7, input Y = 7, filter X = 3, filter Y = 3, stride X = 1, and stride Y = 1. As shown, there are 4 activations and 32 filter channels for each PE, and 25 input segments are mapped across 16 computing units out of 32 computing units where 512 filter channels are mapped such that each computing unit has 32 filter channels, and 25 input segments are mapped across 25 out of 128 activation inputs. Since the architecture used in Example 400 does not support parallel processing of depth units, the remaining 16 out of 32 computing units are not further utilized. The number of depth cycles executed to complete this problem is filter X ,

[0043] , * , * , * , * , * , * , * , * ، * , , * filter Y * depth / activation = 3 * 3 * can be calculated as 512 / 4 = 1152 depth cycles, and the utilization efficiency compared to the maximum theoretical usage is (16<​​​​​​​​​​​​​​Aspects of this disclosure provide techniques for performing depth-parallel processing of the input portion in a neural network processing architecture. By enabling both input parallel processing and depth parallel processing, aspects of this disclosure can enable increased utilization of computing resources (e.g., processing elements) in such architectures. Therefore, the techniques described herein can improve the inference performance of neural network processing architectures because the increased utilization of available computing resources in the architecture can consequently result in an increase in the number of operations that the architecture can perform compared to architectures designed to support depth-sequential processing.

[0044] Furthermore, aspects of this disclosure provide a scalable processing architecture that enables various types of parallel processing for any given workload. The processing architectures described herein may allow options for processing the workload using parallel inputs, parallel kernels, or parallel depths. Moreover, because parts of the input can be processed in parallel at different depths, aspects of this disclosure can reduce the number of processing cycles required to process multiple depth cycles in a neural network.

[0045] Figure 5 shows an embodiment of a neural network processing architecture 500 according to a particular aspect of the present disclosure, in which multiple processing element groups are configured to process multiple parts of an input in parallel at multiple depths. Generally, these parts of the input may be defined by a tiling pattern used to divide the input. A tiling pattern with a small number of tiles can generally divide the input into larger subparts, while a tiling pattern with a large number of tiles can generally divide the input into smaller subparts. The choice of tiling pattern used to divide the input into various parts for processing can represent a trade-off between the performance and accuracy of the inference produced by the neural network. Using a larger number of tiles may allow the data to be processed at a finer granularity level in the neural network, but at a higher computational cost, while using a smaller number of tiles (and, accordingly, larger subparts of the input) may allow the data to be processed at a lower computational cost in the neural network, but at a higher degree of generality.

[0046] As shown in the figure, the neural network processing architecture 500 may include multiple processing element groups 510 configured to process multiple inputs (or portions of inputs) in parallel at multiple depths. In this exemplary architecture, two processing element groups 510A and 510B are shown, but the reader should understand that there may be three or more processing element groups. Each processing element group 510 may include multiple processing elements 512, the output of each processing element may be output to an associated tap register 514. In this embodiment, processing element group 510A includes four processing elements 512A to 512D, and processing element group 510B includes four processing elements 512E to 512H, but the reader should understand that each processing element group may include more or fewer than four processing elements. As shown in the figure, each processing element 512 includes a multiply-accumulate (MAC) circuit 516 and a local accumulator 518 having an input coupled to the output of the MAC circuit. The output of the local accumulator 518 in the processing element 512 may be coupled to the input of the corresponding tap register 514. In some embodiments, the MAC circuit 516 may include multiple multiplier circuits configured to generate partial products based on the multiplication of input values ​​and weight values, and the local accumulator 518 may be implemented as an adder circuit configured to combine the partial products generated by the individual multipliers in the MAC circuit 516 into a local sum.

[0047] In general, tap registers 514 within a processing element group 510 may be coupled such that the output of one tap register 514 acts as an input to another tap register 514. The output of the final tap register (e.g., tap register 514D or 514H) within the processing element group 510 may be coupled to the input of a global accumulator circuit 520 (also referred to as the "final accumulator" and labeled "FINAL-ACC"). For each depth cycle processed in parallel, a given tap register 514 may be configured to shift data provided as input from a preceding tap register 514 simultaneously with the multiply-accumulate operation performed by the processing element 512 corresponding to the given tap register 514. For example, tap register 514B may be configured to shift data provided as input from tap register 514A simultaneously with the operation performed by processing element 512B.

[0048] The output of the global accumulator circuit 520 may be coupled to a bus 530 that connects the processing element group 510 to the digital post-processing logic 540. Generally, the value of the global accumulator circuit 520 can be used by the digital post-processing logic 540 to further process the input data from which the value of the global accumulator circuit 520 was generated (e.g., biasing, batch normalization (BN), linear / nonlinear thresholding, quantization, etc.). In some embodiments, the output of the tap register 514 may also be coupled to the bus 530. In such cases, the bus 530 can be an addressable bus that allows the digital post-processing logic 540 to selectively retrieve data placed on the bus 530 by the global accumulator circuit 520 (and the tap register 514).

[0049] FIG. 6 shows an exemplary data flow timing diagram 600 for parallel processing of different depths of multiple inputs according to certain aspects of the present disclosure. In this example, the inputs can be defined as part of a larger input data set, such as data values in a set of coordinates in a three-dimensional space having an X-axis, a Y-axis, and a Z-axis, where the depth dimension corresponds to the Z-axis in the three-dimensional space. As shown in the timing diagram 600, the processing architecture 500 supports parallel processing of multiple inputs over multiple depth cycles in the Z-axis. Generally, each depth cycle can be associated with a set of weights used to calculate the results in that depth cycle as defined in the model.

[0050] As shown, portions of the inputs can be processed simultaneously over a number E of depth cycles. For example, over E depth cycles, portions of the inputs located at the same locations in the first and second dimensions and different locations in the third dimension can be processed simultaneously. Values stored in the tap registers with respect to the inputs can be accumulated (e.g., through a serial shift) onto a global accumulator as these inputs are processed. After a given depth cycle is executed, the value stored in the global accumulator can be output to digital post-processing logic for further use. Based on the number E of depth cycles over which a portion of the inputs is provided and processed in parallel, in order to minimize the adverse effect on the number of operations supported by the neural network processing architecture over a period of time, the number T of taps (e.g., 8 taps as shown in FIG. 6) can be determined (e.g., calculated or selected). For example, when E = total depth cycles D / number of taps T, if E > T + 2, there may be little or no adverse effect on the processing capacity of the neural network processing architecture (e.g., there are few or no unused processing elements). If E < T + 2, dummy cycles can be added to minimize the adverse effect on the number of operations supported by the neural network processing architecture over a period of time.

[0051] Figure 7 is a block diagram of an exemplary circuit relating to depth-based parallel processing of an input portion according to a particular aspect of the present disclosure. The exemplary circuit of Figure 7 includes a more detailed version of the processing element 512 of Figure 5. As shown in Figure 7, a processing element group 510 of a neural network processing circuit configured to process multiple depths of an input portion simultaneously may include multiple processing elements 512 and a global accumulator circuit 520. This neural network processing circuit may include multiple processing element groups 510, and each processing element group 510 within the neural network processing circuit may be configured to process different portions of the input at multiple depths.

[0052] Each processing element 512, as shown in the figure, generally includes a multiply-accumulate (MAC) circuit 516 (e.g., a MAC adder tree) and a local accumulator 518. Generally, the MAC circuit 516 can generate the result of a mathematical operation on a given input and output the result of the mathematical operation to the local accumulator 518. The output of the local accumulator 518 may be provided as input to the selection circuit 712 along with the value stored in the corresponding tap register 514. The selection circuit 712 can be, for example, a multiplexer circuit (e.g., a 2:1 multiplexer where one of two values ​​input to the multiplexer is output based on the value of a selector control signal), a three-state buffer, multiple switches, or other circuitry that allows selection so that either the output value 716 from the local accumulator 518 or the output value 718 from the tap register 514 is output to the input of the tap register 514. Generally, when the local accumulator 518 finishes calculating the final cumulative result at the end of E depth cycles (for example, after E depth cycles have been processed), the selector control signal can control the selector circuit 712 to select the output of the local accumulator 518 as the input to the tap register 514. That is, the selector control signal can be high when the number of depth cycles reaches E. Otherwise, the selector circuit 712 can select the current value of the tap register 514 as the input to the tap register 514 to save the data currently stored in the tap register.

[0053] In some processing elements 512, a second selection circuit 714 can be used to influence the value stored in the tap register 514. The second selection circuit 714 can be used, for example, in processing elements that follow the first processing element used for the input in the first depth cycle. That is, with respect to the input in the first depth cycle processed by the first processing element 512A, subsequent processing elements 512B and 512C (and other processing elements that follow processing element 512A) can use their respective second selection circuits 714 to influence the value stored in their respective tap registers 514 for each processing element. Similar to the selection circuit 712, the second selection circuit 714 can be implemented by, for example, a multiplexer circuit (e.g., a 2:1 multiplexer where one of two values ​​input to the multiplexer is output based on the value of the selector input), a three-state buffer, multiple switches, or other circuits that allow selection of one of multiple values. The value stored in the tap register 514 can be controlled using control signals, such as a signal for shifting out the value of the tap register 514 over a number of depth cycles equal to the number of taps T. The value selected by the second selection circuit 714 can be the current value of the tap register 514, or the value of the tap register 514 in a preceding processing element (for example, in processing element 512B, as shown in the figure, the output of the tap register 514A of processing element 512A). Therefore, with respect to processing elements 512B and 512C, the tap register 514B (and the tap register 514C) may have a first input coupled to the output of the selection circuit 712B (and the selection circuit 712C) and a second input coupled to the output of the second selection circuit 714B (and the selection circuit 714C).

[0054] The global accumulator circuit 520 includes a selector circuit 722 and a global accumulator 724. The selector circuit 722 can take as input the current value in the global accumulator 724 and the output of a tap register 514 associated with the last processing element used to process the input at a defined number of depths (for example, tap register 514C associated with processing element 512C, as shown in the figure). The value of the global accumulator 724 can be set using a selector input to the selector circuit 722 so that the value stored in the global accumulator 724 is accumulated over several depth cycles. Generally, after E cycles have been processed (for example, at depth cycle E+1), the value stored in the global accumulator 724 can be output to the bus 530, and the global accumulator circuit 520 can be reset.

[0055] Bus 530 is generally coupled to the outputs of one or more tap registers 514 associated with one or more processing elements 512 and the outputs of a global accumulator 724 in an accumulator circuit 520. Generally, bus 530 can mediate data requests from the digital post-processing logic 540 and data dispatch to the digital post-processing logic 540. For example, bus 530 can select which outputs to make available to the digital post-processing logic 540 based on a tiling defined with respect to the neural network, which is performed on the neural network processing circuit. Therefore, the values ​​made available to the digital post-processing logic 540 through bus 530 may include one or more values ​​output by processing elements 512, outputs output by multiple processing elements 512, and / or values ​​stored in the global accumulator 724 and output to bus 530.

[0056] Figure 8 shows one embodiment of the use of processing elements in a neural network processing architecture where parts of the input are processed in parallel at different depths. As shown, multiple computing units can be aggregated to support processing parts of the input at multiple depths. In this embodiment, two computing units can be aggregated to form a virtual computing unit supporting 128 inputs with respect to a computing unit supporting 64 input parts. Each processing element can process the input using four activations and 32 filter channels. As shown, 25 parts of the input (e.g., 25 different segments of an image) may be processed in parallel at four different depths (e.g., based on a ResNet50 layer having a 7x7x512 input tensor and several corresponding kernel tensors of size 3x3x512, convolving 25 different fields). Because the input parts can be processed in parallel at different depths of the neural network, a significantly smaller number of processing element circuits may remain unused compared to the number of unused processing element circuits shown in Figure 4. In this embodiment, the problem processed using the neural network processing architecture described herein, which can process inputs at different depths in parallel, can be defined according to the parameters discussed above with respect to Figure 4 (e.g., depth = 512, input X = 7, input Y = 7, filter X = 3, filter Y = 3, stride X = 1, and stride Y = 1). The number of clock cycles in which the problem is processed is determined by filter X * Filter Y * Depth / Activation = 3 * 3 * 512 / 4 / 5 = 231 cycles, or it can be defined as approximately 20% of the number of cycles used in the depth sequential processing discussed above. The utilization efficiency compared to the maximum theoretical usage is (32 * 50 * 4 * 32) / (32 * 64 * 4 *32) can be set to 78%. Furthermore, the number of operations per second can be significantly increased compared to the number of operations per second that can be performed in depth sequential processing of the input through a neural processing architecture operating at the same clock rate. Therefore, as shown in the figure, by processing the input using depth parallel processing, less computing resources can be wasted, and the overall performance of the neural processing architecture can be improved compared to the performance of a neural processing architecture that processes the input using depth sequential processing.

[0057] Exemplary behavior Figure 9 is a flowchart illustrating an exemplary operation 900 relating to neural network processing according to a particular aspect of this disclosure. Operation 900 can be performed by a neural network processing circuit, such as the neural network processing architecture 500 shown in Figure 5 or Figure 7.

[0058] Operation 900 can be started in block 910 by receiving an input for processing. The input can be a multidimensional array or multidimensional tensor of data that can be segmented for parallel processing in the neural network.

[0059] In some embodiments, the input may include data corresponding to a three-dimensional space. The first dimension in the three-dimensional space may correspond to the horizontal dimension. The second dimension in the three-dimensional space may correspond to the vertical dimension. The third dimension in the three-dimensional space may correspond to the depth dimension. For example, the three-dimensional space can be a Euclidean space where the data is represented in terms of height, width, and depth dimensions, such as data from a three-dimensional image of an object. In another embodiment, the three-dimensional space may include spatial data on a two-dimensional plane, and the third dimension corresponds to the time channel in the input. This three-dimensional data may include, for example, video data, audio data, or other information where time is one of the dimensions.

[0060] In some embodiments, the input may consist of multiple segments. For example, these segments may be sized so that the same portion of the input can be processed in parallel or substantially in parallel at different depths of the neural network. Each segment of the multiple segments may be the same pixel block (e.g., with respect to horizontal and vertical coordinates) in different images, corresponding, for example, different depths in a 3D image or different timestamps in time data (e.g., different frames in a video). In some embodiments, the multiple segments may be based on tiling, which is used to define how the neural network should process the input. Each segment of the multiple segments may represent a sub-part or tile of the input, and its size may be defined by the number of tiles into which the input is divided. For example, if the number of tiles is i, an image input having X pixels on the horizontal axis and Y pixels on the vertical axis would be (X * It can be divided into tiles with a size of Y) / i pixels.

[0061] In block 920, with respect to a segment of the input, intermediate outputs are generated substantially in parallel at each of the multiple depths within the neural network. The intermediate outputs may be generated based on weights in the neural network associated with filters or kernels used to process a portion of the input at a given depth within the neural network.

[0062] In some embodiments, generating an intermediate output may include generating an intermediate output through a multiply-accumulate (MAC) circuit. The value generated by the MAC circuit may be stored in a tap register. In some embodiments, generating an intermediate output may further include shifting the value in the tap register between each of a plurality of processing cycles.

[0063] In block 930, with respect to a given segment of the input, each intermediate output is accumulated to the final output. In some embodiments, accumulating intermediate outputs to the final output may involve accumulating shifted values ​​from tap registers, each associated with a different value, over multiple processing cycles. As discussed, the number of processing cycles may correspond to the number of depths being processed in parallel within the neural processing unit.

[0064] In block 940, at least the final output is output to the memory bus. In some embodiments, the final output may be output to the memory bus based on a signal indicating that processing is complete with respect to a threshold number of depth cycles. By outputting the final output to the memory bus based on this signal, incomplete data can be prevented from being placed on the bus.

[0065] As discussed, the memory bus can be a selective bus that enables selective dispatch or availability of data to digital post-processing blocks for further use. In some embodiments, each intermediate output may also be output to the memory bus. The memory bus can selectively make values ​​available to digital post-processing blocks, for example, based on information about how the inputs are tiled for processing in a neural network.

[0066] Exemplary devices using depth parallelism Figure 10 shows an exemplary electronic device 1000. The electronic device 1000 may be configured to perform the methods described herein, including the operation 900 described with respect to Figure 9.

[0067] The electronic device 1000 includes a central processing unit (CPU) 1002, which in some embodiments may be a multi-core CPU. Instructions executed in the CPU 1002 can be loaded, for example, from program memory associated with the CPU 1002, or from memory 1024.

[0068] The electronic device 1000 also includes additional processing blocks adapted to specific functions, such as a graphics processing unit (GPU) 1004, a digital signal processor (DSP) 1006, a neural network circuit 1007 having a set of PEs 1009 for performing depth parallel processing of inputs to the neural network circuit, a multimedia processing block 1010, and a wireless connectivity processing block 1012. In one implementation, the neural network circuit 1007 is implemented in one or more of the CPU 1002, GPU 1004, and / or DSP 1006.

[0069] In some embodiments, the wireless connection processing block 1012 may include components relating to, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and / or wireless data transmission standards. The wireless connection processing block 1012 is further connected to one or more antennas 1014 to facilitate wireless communication.

[0070] The electronic device 1000 may also include a navigation processor 1020 which may include one or more sensor processors 1016 associated with any type of sensor, one or more image signal processors (ISPs) 1018 associated with any type of image sensor, and / or satellite-based positioning system components (e.g., Global Positioning System (GPS) or Global Navigation Satellite System (GLONASS)), and inertial positioning system components.

[0071] The electronic device 1000 may also include one or more input and / or output devices 1022, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone. In some embodiments, one or more of the processors of the electronic device 1000 may be based on an advanced RISC machine (ARM) instruction set, where RISC stands for "reduced instruction set computing."

[0072] The electronic device 1000 also includes a memory 1024, which represents one or more static and / or dynamic memories, such as dynamic random access memory (DRAM), flash-based static memory, etc. In this embodiment, the memory 1024 includes a computer executable component that can be run by one or more of the aforementioned processors of the electronic device 1000, including a neural network circuit 1007. The illustrated components, and other components not illustrated, may be configured to perform various aspects of the methods described herein.

[0073] In some embodiments, such as when the electronic device 1000 is a server device, various components, such as one or more of the multimedia processing block 1010, wireless connection processing block 1012, antenna 1014, sensor processor 1016, ISP 1018, or navigation processor 1020, can be omitted from the embodiment shown in Figure 10.

[0074] Exemplary Qualities In addition to the various embodiments described above, certain combinations of these embodiments are within the scope of this disclosure, some of which are detailed in the following clauses.

[0075] Clause 1: A processing circuit comprising a plurality of groups of processing element (PE) circuits, each group of PE circuits being configured to process inputs in parallel at a plurality of depths, each PE circuit comprising a plurality of PE circuits, each comprising one or more multipliers configured to calculate a partial product, and a local accumulator having an input coupled to the output of the one or more multipliers, the local accumulator being configured to produce a sum from the partial products produced by each of the one or more multipliers.

[0076] Clause 2: The processing circuit of Clause 1, further comprising a register having an input coupled to the output of a local accumulator, each PE circuit.

[0077] Clause 3: The processing circuit of Clause 2, further comprising multiple global accumulators, wherein the output of a register in each PE circuit is coupled to the input of another register in another PE circuit within the group of PE circuits, or to the input of one of the global accumulators.

[0078] Clause 4: The processing circuit of Clause 3, further comprising a bus, wherein a separate output of a register within each PE circuit is coupled to the bus.

[0079] Clause 5: The processing circuit of Clause 4, wherein each output of the global accumulator is further coupled to a bus.

[0080] Clause 6: Any processing circuit from Clauses 2 to 5, wherein each PE circuit further includes a first selection circuit having a first input coupled to the output of a local accumulator and an output coupled to the input of a register.

[0081] Clause 7: The processing circuit of Clause 6, wherein the first selection circuit in each PE circuit has a second input coupled to the output of a register.

[0082] Clause 8: The processing circuit of Clause 7, wherein the first selection circuit includes a 2:1 multiplexer, a 3-state buffer, or multiple switches.

[0083] Clause 9: Any one of the processing circuits from Clauses 6 to 8, further comprising a second selection circuit having an output coupled to the input of a register, a first input coupled to the output of a register, and a second input coupled to the output of another register in another PE circuit within the group of PE circuits.

[0084] Clause 10: The processing circuit of Clause 9, wherein the second selection circuit includes a 2:1 multiplexer, a 3-state buffer, or multiple switches.

[0085] Clause 11: A processing circuit from any of Clauses 1 to 10, further comprising a plurality of global accumulators, each global accumulator having an input coupled to one output of a group of PE circuits.

[0086] Clause 12: Any processing circuit from Clauses 1 to 11, wherein multiple PE circuits are further configured to process multiple depths of multiple inputs in parallel.

[0087] Clause 13: A method for neural network processing, comprising: receiving an input for processing; generating, substantially in parallel, intermediate outputs for each of a plurality of depths in the neural network with respect to each segment of a plurality of segments of the received input, based on the weights in the neural network associated with each depth in the neural network; accumulating the intermediate outputs for each respective depth into a final output; and outputting the final output to a memory bus.

[0088] Clause 14: The method of Clause 13, wherein the input includes data from a three-dimensional space, where the first dimension in the three-dimensional space corresponds to the horizontal dimension, the second dimension in the three-dimensional space corresponds to the vertical dimension, and the third dimension in the three-dimensional space corresponds to the depth dimension.

[0089] Clause 15: The method of Clause 14, wherein data from a three-dimensional space includes video data, and the depth dimension corresponds to the time channel in the video data.

[0090] Clause 16: Any method of Clauses 13-15, wherein generating intermediate outputs for each depth in a neural network includes generating intermediate outputs through a sum-of-accumulate (MAC) circuit and storing the values ​​of the MAC circuit in a tap register.

[0091] Clause 17: The method of Clause 16, further comprising shifting values ​​in a tap register between each of a plurality of processing cycles to generate intermediate outputs for each depth in the neural network.

[0092] Clause 18: Any method of Clauses 13-17, wherein the accumulation of intermediate outputs for each depth to the final output includes accumulating shifted values ​​associated with each of each depth over multiple processing cycles, which are stored in one or more tap registers.

[0093] Clause 19: Outputting the final output to the memory bus is any method of Clauses 13-18, including outputting each intermediate output to the memory bus.

[0094] Clause 20: Outputting the final output to the memory bus is based on a signal indicating that processing is complete with respect to a threshold number of depth cycles, using one of the methods in Clauses 13-19.

[0095] Clause 21: A device comprising a memory bus, a digital post-processing block, and a processor configured to perform any of the operations of Clauses 13 to 20.

[0096] Clause 22: A device comprising means for performing any of the operations of Clauses 13 to 20.

[0097] Additional considerations The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The embodiments discussed herein do not limit the scope, applicability, or embodiments described herein. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may also be applied to other embodiments. For example, changes may be made to the function and configuration of the elements discussed without departing from the scope of the disclosure. Various embodiments may omit, substitute, or add various procedures or components as needed. For example, the methods described may be performed in a different order than described, and various steps may be added, omitted, or combined. Also, features described in some embodiments may be combined in some other embodiments. For example, an apparatus may be implemented or a method may be practiced using any number of embodiments described herein. Furthermore, the scope of the disclosure is intended to encompass apparatus or methods that are practiced using other structures, functions, or structures and functions in addition to, or other than, the various embodiments of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein can be embodied by one or more elements of the claims.

[0098] As used herein, the term “exemplary” means “serving as an example, case, or illustration.” No embodiment described herein as “exemplary” should necessarily be construed as being preferable or advantageous to any other embodiment.

[0099] As used herein, the phrase “at least one of” a list of items refers to any combination of those items, including a single element. For example, “at least one of a, b, or c” is intended to include a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0100] As used herein, the term “determining” encompasses a wide range of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0101] The methods disclosed herein include one or more steps or actions for achieving the method. The steps and / or actions of those methods can be interchanged with one another without departing from the claims. In other words, unless a particular order of steps or actions is specified, the order and / or use of any particular steps and / or actions can be modified without departing from the claims. Furthermore, various operations of the methods described above may be performed by any preferred means capable of performing the corresponding function. These means may include, but are not limited to, various hardware components and / or software components, including circuits, application-specific integrated circuits (ASICs), or processors, and / or various hardware modules and / or software modules. Generally, where operations are shown in the figures, those operations may have corresponding equivalent means-plus-function components, similarly numbered. Exemplary means-plus-function components may include, among other things, means for receiving, means for generating, means for accumulating, and means for outputting. Means for receiving may include, in particular, input / output blocks such as the input / output block 1012 shown in Figure 10, and wireless communication blocks (e.g., transceivers) such as the wireless communication block 1012 shown in Figure 10. Means for generating may include one or more processors or processing circuits such as the neural network circuit 1007 and / or processing element 1009 shown in Figure 10. Means for accumulating may include one or more processors or processing circuits such as the neural network circuit 1007 and / or processing element 1009 shown in Figure 10. Means for outputting may include, in particular, input / output blocks such as the input / output block 1012 shown in Figure 10, and wireless communication blocks (e.g., transceivers) such as the wireless communication block 1012 shown in Figure 10.

[0102] The following claims are not intended to be limited to the embodiments shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, a singular reference to an element is intended to mean "one or more" rather than "only one" unless otherwise explicitly stated. Unless otherwise explicitly stated, the term "several" refers to one or more. No element of a claim should be construed under Section 112(f) of the U.S. Patent Act unless it is explicitly enumerated using the phrase "means for..." or, in the case of a method claim, enumerated using the phrase "steps for...". All structural and functional equivalents of elements of various embodiments described throughout this disclosure, which are known to or will become known to those skilled in the art, are expressly incorporated by reference herein and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, whether or not such disclosure is expressly enumerated in the claims.

Claims

1. A processing circuit comprising multiple groups of processing element (PE) circuits, Each group of PE circuits includes multiple PE circuits configured to process inputs in parallel at multiple depths, and a global accumulator circuit. Each PE circuit, One or more multiplier circuits, each configured to calculate a partial product, A local accumulator having an input coupled to the output of one or more multiplier circuits, configured to generate a sum from the partial products calculated by each of the one or more multiplier circuits, A tap register coupled to the output of the local accumulator and one or more tap registers of other PE circuits in the group of PE circuits, wherein the tap registers in each group of PE circuits are sequentially connected, including a tap register The output of the last tap register in the sequentially connected tap registers of each group of the PE circuit is coupled to the input of the global accumulator circuit of that group of the PE circuit. Each tap register is configured to simultaneously shift the output received from the coupled local accumulator and any output received from the coupled tap register via the sequentially connected tap registers. A processing circuit in which the global accumulator circuit accumulates values ​​received from the sequentially connected tap registers and outputs the accumulated value when a given depth cycle is executed.

2. The processing circuit according to claim 1, further comprising a bus, wherein another output of the tap register in each PE circuit is coupled to the bus.

3. The processing circuit according to claim 2, wherein each output of the global accumulator circuit is further coupled to the bus.

4. The processing circuit according to claim 1, wherein the plurality of PE circuits are further configured to process a plurality of inputs in parallel at a plurality of depths.

5. A method for a processing circuit comprising a plurality of groups of processing element (PE) circuits, wherein each group of PE circuits comprises a plurality of PE circuits configured to process inputs in parallel at a plurality of depths, and a global accumulator circuit, the method is A step of calculating a partial product using one or more multiplication circuits of the PE circuits among the plurality of PE circuits, A step of generating a sum from the partial products calculated by each of the one or more multiplication circuits, In a tap register, the steps include receiving the generated sum and the output from the preceding tap register among the multiple sequentially connected tap registers of the PE circuit, The step includes simultaneously shifting the received generated sum and the received output from the tap register to the subsequent tap register in the plurality of sequentially connected tap registers of the group of PE circuits, The above method is in a global accumulator circuit that is coupled to the last tap register among the plurality of sequentially connected tap registers, The steps include: accumulating the values ​​received from the sequentially connected tap registers; The steps include: outputting the accumulated value when a given depth cycle is executed; A way to further enhance it.

Images