Parallel depth-wise processing architectures for neural networks

Abstract

Methods and apparatuses for performing machine learning tasks, and in particular neural network processing architectures and circuits for improving performance via deep parallelism. An example neural network processing circuit typically includes a complex array of processing element (PE) circuits, wherein each set of PE circuits includes a plurality of PE circuits configured to process the input in parallel at a plurality of depths.

Description

[Technical Field]

[0001] This application claims priority to U.S. Application No. 17 / 465,550, filed on September 2, 2021, which is assigned to the assignee of this application and is incorporated herein by reference in its entirety.

[0002] The various forms of this disclosure relate to machine learning, and more particularly to circuits, neural network processing architectures, and techniques for depth-wise parallel processing to perform machine learning tasks. [Previous Technology]

[0003] Machine learning is generally the process of generating trained models (e.g., artificial neural networks, trees, or other structures) that represent a generalized fit to a set of training data. Applying the trained model to new data generates inference, which can be used to understand the new data. In some cases, applying a model to new data is described as "inference" about the new data.

[0004] With the surge in use of machine learning for various machine learning (or artificial intelligence) tasks, the expectation for more efficient processing of machine learning model data is growing. In some cases, dedicated hardware (such as machine learning accelerators) can be used to enhance the processing system's ability to process machine learning model data. However, such hardware requires space and power, which are not always available on processing devices. For example, "edge processing" devices (such as mobile devices), always-on devices, Internet of Things (IoT) devices, etc., often have to balance processing power with power and packaging limitations. Furthermore, accelerators may move data between common data buses, which can result in significant power consumption and introduce latency to other processes sharing the data bus. Therefore, other forms of processing systems are being considered for processing machine learning model data. [Summary of the Invention]

[0005] The systems, methods, and apparatuses disclosed herein each have several variations, with no single variation responsible for its desired properties. Without limiting the scope of this disclosure as expressed by the appended claims, some features will now be briefly discussed. After considering this discussion, and especially after reading the section entitled "Implementation," one will understand how the features of this disclosure provide the advantages described herein.

[0006] Certain aspects of this disclosure relate to processing circuitry. Processing circuitry typically includes multiple arrays of processing element (PE) circuits. Each array of PE circuits includes a plurality of PE circuits configured to process inputs in parallel at a plurality of depths. Each PE circuit typically includes one or more multiplication circuits and a local accumulator, the local accumulator having inputs coupled to the outputs of one or more multiplication circuits. Each multiplication circuit can be configured to compute partial products, and the local accumulator can be configured to generate a sum based on the partial products computed by each of the one or more multiplication circuits.

[0007] Certain aspects of this disclosure relate to methods for neural network processing. The method typically includes receiving input for processing. For each of a plurality of segments of the received input, intermediate outputs for a corresponding depth in a plurality of depths in the neural network are generated in parallel, substantially based on weights associated with that corresponding depth in the neural network. The intermediate outputs for each corresponding depth are accumulated into a final output. At least this final output is output to a memory bus.

[0008] Other configurations include a processing system configured to perform the above-described methods and the methods described herein; a non-transitory computer-readable medium containing instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the above-described methods and the methods described herein; a computer program product embodied on a computer-readable storage medium, including code for performing the above-described methods and the methods further described herein; and a processing system including components for performing the above-described methods and the methods further described herein.

[0009] To achieve the foregoing and related objectives, the one or more embodiments include features fully described below and specifically pointed out in the claims. The following description and drawings illustrate certain illustrative features of the one or more embodiments in detail. However, these features only indicate some of the various ways in which the principles of the various embodiments can be employed.

Implementation Method

[0022] Various forms of this disclosure provide apparatus, methods, processing systems, and computer-readable media for processing inputs of multiple depths in parallel. An introduction to neural networks, deep neural networks, and deep learning is also provided.

[0023] Neural networks are organized into layers of interconnected nodes. Typically, nodes (or neurons) are where computation occurs. For example, a node can combine input data with a set of weights (or coefficients) that amplify or attenuate the input data. Thus, the amplification or attenuation of the input signal can be viewed as an assignment of the relative salience of each input to the task the network is attempting to learn. Typically, the input weight products are summed (or accumulated), and then the sum is passed via the node's activation function to determine whether and to what extent the signal should proceed further in the network.

[0024] In the most basic implementation, a neural network can have an input layer, a hidden layer, and an output layer. "Deep" neural networks typically have more than one hidden layer.

[0025] Deep learning is a method for training deep neural networks. Typically, deep learning maps inputs to the network to outputs from the network, and is thus sometimes called a "universal approximator" because deep learning can learn to approximate an unknown function f(x) = y between any input x and any output y. In other words, deep learning finds the correct f to transform x into y.

[0026] More specifically, deep learning trains each layer of nodes based on different sets of features, which are the outputs of the previous layer. Thus, with each successive layer of the deep neural network, the features become increasingly complex. Therefore, deep learning is powerful because it can progressively extract higher-level features from the input data and perform complex tasks (such as object recognition) by learning to represent the input with progressively higher-level abstractions in each layer, thereby building useful feature representations of the input data.

[0027] For example, if presented as visual data, the first layer of a deep neural network can learn to identify relatively simple features in the input data, such as edges. In another instance, if presented as auditory data, the first layer of a deep neural network can learn to identify spectral power in the input data at a specific frequency. Subsequently, the second layer of the deep neural network can learn to identify combinations of features, such as simple shapes in visual data or sound combinations in auditory data, based on the output of the first layer. Higher layers can then learn to identify complex shapes in visual data or words in auditory data. Higher layers can also learn to identify common visual objects or spoken phrases. Thus, deep learning architectures can perform particularly well when applied to problems with a natural hierarchical structure. Layer connections in a neural network

[0028] Neural networks, such as deep neural networks (DNNs), can be designed with various connection patterns between layers.

[0029] Figure 1A shows an example of a fully connected neural network 102. In the fully connected neural network 102, each node in the first layer transmits its output to each node in the second layer, such that each node in the second layer receives input from each node in the first layer.

[0030] Figure 1B illustrates an example of a locally connected neural network 104. In the locally connected neural network 104, nodes in the first layer can be connected to a finite number of nodes in the second layer. More typically, the locally connected layers in the locally connected neural network 104 can be configured such that each node in the layer will have the same or similar connection pattern, but with connection strengths (or weights) that can have different values ​​(e.g., values ​​associated with local regions 110, 112, 114, and 116 of the first-layer nodes). The locally connected connection pattern can produce spatially different receptive fields in higher layers because higher-layer nodes in a given region can receive input via attributes tuned to a restricted portion of the total input to the network.

[0031] One type of locally connected neural network is the spiral neural network (CNN). Figure 1C shows an example of a spiral neural network 106. The spiral neural network 106 can be configured such that the connection strength associated with the input for each node in the second layer is shared (e.g., for a local region 108 that overlaps with another local region of a node in the first layer). Spiral neural networks are well-suited for problems in which the spatial location of the input is meaningful.

[0032] One type of spiral neural network is the deep spiral network (DCN). A deep spiral network is a network with multiple spiral layers, which can be further configured using, for example, pooling and normalization layers.

[0033] Figure 1D illustrates an example of a DCN 100 designed to recognize visual features in an image 126 generated by an image capturing device 130. For example, if the image capturing device 130 is a camera mounted inside or on a vehicle (or otherwise moving with the vehicle), the DCN 100 can be trained using various supervised learning techniques to recognize traffic signs and even numbers on traffic signs. The DCN 100 can also be trained for other tasks, such as recognizing lane markings or traffic lights. These tasks are just some examples, and many other tasks are possible.

[0034] In the example of Figure 1D, DCN 100 includes a feature extraction part and a classification part. After receiving image 126, swirl layer 132 applies a swirl kernel (e.g., as depicted and described in Figure 2) to image 126 to produce a first feature map set 118 (or intermediate start). Typically, a "kernel" or "filter" contains a multidimensional array of weights designed to emphasize different states of the input data channels. In various instances, "kernel" and "filter" can be used interchangeably to represent the set of weights applied in a swirl neural network.

[0035] Subsequently, the first feature map set 118 can be subsampled by a pooling layer (e.g., a max pooling layer, not shown) to generate a second feature map set 120. The pooling layer can reduce the size of the first feature map set 118 while retaining most of the information to improve model performance. For example, the second feature map set 120 can be downsampled from a 28x28 matrix to a 14x14 matrix by a pooling layer.

[0036] This process can be repeated in many layers. In other words, the second feature map set 120 can be further spun through one or more subsequent spun layers (not shown) to produce one or more subsequent feature map sets (not shown).

[0037] In the example of Figure 1D, the second feature map set 120 is provided to the fully connected layer 124, which then produces an output feature vector 128. Each feature of the output feature vector 128 may include a number corresponding to a possible feature of the image 126, such as "sign", "60", and "100". In some cases, a softmax function (not shown) can convert the numbers in the output feature vector 128 into probabilities. In this case, the output 122 of DCN 100 is the probability that the image 126 includes one or more features.

[0038] The softmax function (not shown) can convert the independent elements of the output feature vector 128 into probabilities so that the output 122 of DCN 100 is one or more probabilities that the image 126 includes one or more features (such as a sign with the number "60" above it, as in image 126). Thus, in this example, the probabilities for "sign" and "60" in the output 122 should be higher than the probabilities of other elements in the output 122 (such as "30", "40", "50", "70", "80", "90", and "100").

[0039] Before training DCN 100, the output 122 generated by DCN 100 may be incorrect. Therefore, the error between output 122 and the pre-known target output can be calculated. For example, here, the target output is an indication of image 126 including a "sign" and the number "60". Using the known target output, the weights of DCN 100 can then be adjusted by training so that subsequent outputs 122 of DCN 100 achieve the target output (with high probability).

[0040] To adjust the weights of DCN 100, the learning algorithm can compute gradient vectors for the weights. The gradient vectors indicate the amount by which the error will increase or decrease when the weights are adjusted in a particular way. Subsequently, the weights can be adjusted to reduce the error. This method of adjusting weights can be called "backpropagation" because the adjustment process involves "back-passing" through the layers in DCN 100.

[0041] In practice, the error gradient of the weights can be calculated on a small number of instances so that the calculated gradient approximates the true error gradient. This approximation method can be called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing, or until the error rate reaches the target level.

[0042] After training, DCN 100 can be presented with new images, and DCN 100 can generate inferences, such as classifications or probabilities of various features in the new images. Rotary technology for rotary neural networks.

[0043] Coil is often used to extract useful features from an input dataset. For example, in a coil neural network such as those described above, coil can extract different features using kernels and / or filters, where the weights of the kernels and / or filters are automatically learned during training. The extracted features are then combined for inference.

[0044] An activation function can be applied before and / or after each layer of a spiral neural network. The activation function is typically a mathematical function used to determine the output of a node in the neural network. Thus, the activation function determines whether a node should pass information based on whether the node's input is relevant to the model's prediction. In one instance, where y = conv(x) (i.e., y is a spiral of x), x and y are generally considered as "activation." However, for a specific spiral operation, x can also be called "pre-activation" or "input activation" because x exists before the specific spiral, and y can be called output activation or feature map.

[0045] Figure 2 depicts a three-dimensional spiral, where input 210 is organized into a three-dimensional space and processed using a plurality of spiral kernels 220 to produce output 230. Input 210 can be a three-dimensional input, such as a three-dimensional image with spatial data in each of three dimensions (e.g., dimension X (horizontal), dimension Y (vertical), and dimension Z (depth)), or a sequence of two-dimensional images (with dimensions X (horizontal) and Y (vertical)) that vary over time (e.g., video), where time may correspond to dimension Z (depth). In this particular instance, a 12-particle × 12-particle image with a certain number of n depth cycles can be processed via a corresponding spiral kernel 220 with a dimension of 3-particle × 3-particle × n depth cycles. Output 230 can have the same size along dimensions X and Y as well as the depth dimension, the depth dimension corresponding to the number of spiral kernels 220 used to process input 210.

[0046] Output fixation techniques (whereby the output of the neural network is stored in memory associated with the processing elements) can allow for the rapid generation of output 230. Typically, using output fixation involves: deep cycles (corresponding to different cores from a plurality of spin cores 220) residing in the inner loop of a multi-loop logic structure, where the outer loop of the multi-loop logic structure is used for iteration in dimensions X and Y. However, output fixation techniques can allow accumulator circuitry to be reused in deep cycles for a given input in dimensions X and Y. However, each deep cycle typically requires reloading the weights associated with that deep cycle into the processing element used to process the deep cycle. Furthermore, when the core is updated, the inputs in dimensions X and Y can be reloaded into one or more processing elements and processed. Some processing techniques can support weight-fixed processing of inputs. In this case, the inputs in dimensions X and Y can be iterated in the inner loop of the multi-loop structure. Example neural network processing architecture and data flow.

[0047] Figure 3 is a block diagram of an example neural network processing architecture 300, illustrating an example data flow sequence in which certain forms of the present disclosure can be implemented. The neural network processing architecture 300 may include a plurality of processing elements (PEs) 302 for performing data computations (e.g., multiplication and accumulation (MAC) operations) and other operations. PEs 302 may be implemented using any of a variety of suitable circuitry such as a neural processing unit (NPU), a neural signal processor (NSP), or a computation in memory (CIM) circuit. The processing architecture 300 may also include a global memory 304 (labeled "global buffer"), a tightly coupled memory (TCM) 306, a startup buffer 308, a weight buffer 310, a digital post-processing module 312, a memory bus 314, a register bus 316, and a PE bus 318. As used herein, TCM generally refers to memory accessed from a processor such as PEs 302 via a dedicated connection. The start buffer 308 and weight buffer 310 can each be configured as multiple register trays. Data stored in the weight buffer 310 can be broadcast to parallel inputs (e.g., inputs processed in parallel by PE 302). Memory bus 314 can couple global memory 304 to TCM 306. Register bus 316 can couple TCM 306 to start buffer 308, weight buffer 310, and digital post-processing module 312. PE bus 318 can couple PE 302 to start buffer 308, weight buffer 310, and digital post-processing module 312. In this way, PE 302 can access various memory resources to read data from start buffer 308 and / or weight buffer 310, and output the results of processing the data read from start buffer 308 and / or weight buffer 310 to digital post-processing module 312. Example of parallel processing at the depth of input.

[0048] Parallel processing typically allows processing units (such as neural signal processors (NSPs) or neural processing units (NPUs)) to process multiple portions or segments of an input substantially simultaneously. Typically, parallel processing can leverage the independence of individual inputs to accelerate the processing of the input set and complete the execution of operations on the input set using fewer computational resources (e.g., time) and / or more efficient use of computational resources (e.g., a proportion of all available computational resources in the computing system) compared to the computational resources used when processing the input set in sequence. Typically, an NSP or NPU can be designed to support input parallelism for different portions of the input (e.g., defined blocks of a two-dimensional input, such as primitive blocks in an image) and to process these portions of the input sequentially at different depths (e.g., different channels or times) using different kernel (depth) cycles, as discussed above with respect to Figure 2. That is, for an input partial index {1, 2, 3, ..., n} and a depth m, such an NSP or NPU can be designed to process the value at input partial index {1, 2, 3, ..., n} at depth 1, then process the value at input partial index {1, 2, 3, ..., n} at depth 2, and so on, up to depth m. Because such NSPs and NPUs can be designed to support depth-sequential processing, their capabilities may not be fully utilized. Furthermore, increasing the depth of the neural network may exacerbate the impact of underutilized NSP or NPU processing capabilities, thus negatively impacting the performance metrics of such NSPs or NPUs as the depth of the neural network increases. For example, given a maximum number of operations per second that an NSP or NSU can support when using all processing elements (PEs) in the NSP or NPU, the number of operations per second that an NSP or NPU can support when using depth-sequential processing may be significantly lower.

[0049] To allow for improved performance, the processing elements in the NSP or NPU can be designed to support increased startup depth. However, increasing the startup depth supported by the processing elements may reduce mapping flexibility, reduce clock frequency, and / or reduce the number of inputs that can be processed in parallel using the same hardware resources.

[0050] Figure 4 illustrates an example 400 of processing elements utilized in a neural network processing architecture, wherein the input is processed cascaded at different depths in the neural network. In example 400, the architecture may support input parallelism and filter channel parallelism so that multiple segments of the input (e.g., in dimensions X and Y as shown in Figure 2) can be processed in parallel using multiple parallel filters, but may not support depth parallelism, where the same segments of the input can be processed at different depths in the neural network (e.g., using different kernels). As shown, the architecture in example 400 may have 32 computational units (labeled "CU-1" to "CU-32"), and each of these computational units may include 64 processing elements (PEs), which can be used to process portions of the input provided to the architecture in parallel. Each PE may be configured to process 4 startups and 32 filter channels. In this example, a layer of the ResNet50 network may have an input tensor of size 7x7x512 and a number of corresponding kernel tensors of size 3x3x512. For any 7x7 input channel, 25 operations can be performed to oscillate 25 distinct but potentially overlapping receptive fields using a 3x3 core with a stride distance of 1. In this example, a 5x5 feature map can be produced as the output of this layer. As shown, input segments 1 to 25 can be processed via corresponding processing elements in computational units 1 to 16 of the architecture. However, since the architecture may not support depth-wise parallelism, computational resources that could be used to process inputs at another depth may go unused.

[0051] Therefore, in Example 400, a significant amount of computational resources may be unused, and performance may be unadjustable. For example, resource utilization can be calculated as the product of the number of computational units on which the problem is executed, the number of inputs, the number of starts, and the number of filter channels. In this example, as discussed, the problem can be defined with the following parameters: depth = 512, input X = 7, input Y = 7, filter X = 3, filter Y = 3, stride X = 1, and stride Y = 1. As shown, there are four starts and 32 filter channels for each PE, and 25 input segments are processed on 16 of the 32 computational units because the 512 filter channels are mapped onto 16 computational units, each with 32 filter channels; and the 25 input segments are mapped onto 25 of the 128 start inputs. The remaining 16 of the 32 computational units were not further utilized because the architecture used in instance 400 does not support depth-wise parallelism. The number of depth cycles executed to accomplish this problem can be calculated as FilterX * FilterY * Depth / Activations = 3 * 3 * 512 / 4 = 1152 depth cycles, and the utilization efficiency compared to the maximum theoretical utilization can be (16 * 25 * 4 * 32) / (32 * 64 * 4 * 32) = 20%. Furthermore, due to the low utilization efficiency, the number of operations supported in a given time period may be similarly less than the maximum theoretical number of operations supported in that given time period assuming full utilization of the architecture's resources.

[0052] Various forms of this disclosure provide techniques for performing depth-parallel processing of input portions in a neural network processing architecture. By allowing input parallelism and depth parallelism, various forms of this disclosure can allow increased utilization of computational resources (e.g., processing elements) in such an architecture. Thus, the techniques described herein can improve inference performance using a neural network processing architecture because the increased utilization of available computational resources in such an architecture can result in an increase in the number of operations that can be performed by that architecture relative to an architecture designed to support deep sequential processing.

[0053] Furthermore, the various forms of this disclosure provide an adjustable processing architecture that allows for various types of parallel processing to be implemented for any given workload. The processing architecture described herein allows for selection regarding the use of parallel inputs, parallel kernels, or parallel depths to process the workload. Moreover, since (parts) of the input can be processed in parallel at different depths, the various forms of this disclosure can reduce the number of processing cycles required to process a certain number of deep cycles in a neural network.

[0054] Figure 5 illustrates an example of a neural network processing architecture 500, wherein, according to certain patterns of this disclosure, multiple processing element groups are configured to process multiple portions of the input in parallel at multiple depths. Typically, portions of the input can be defined by a tiling pattern used to partition the input. A tiling pattern with fewer tiles can typically divide the input into larger sub-parts, while a tiling pattern with more tiles can typically divide the input into smaller sub-parts. The choice of tiling pattern used when dividing the input into different parts for processing can represent a trade-off between the performance and accuracy of inference generated by the neural network. Using a larger number of tiles allows data to be processed at a finer level in the neural network, but with a greater computational management burden; however, using a smaller number of tiles (and correspondingly, larger sub-parts of the input) allows data to be processed in the neural network with a smaller computational management burden, but with a higher degree of versatility.

[0055] As shown in the figure, the neural network processing architecture 500 may include a plurality of processing element groups 510 configured to process a plurality of inputs (or portions of inputs) in parallel at a plurality of depths. In this example architecture, two processing element groups 510A and 510B are illustrated, but the reader should understand that there may be more than two processing element groups. Each processing element group 510 may include a plurality of processing elements 512, and the output of each processing element may be output to an associated tap register 514. In this example, processing element group 510A includes four processing elements 512A-512D, and processing element group 510B includes four processing elements 512E-512H, but the reader should understand that each processing element group may include more or fewer than four processing elements. As shown in the figure, each processing element 512 includes a multiply-accumulate (MAC) circuit 516 and a local accumulator 518 having an input coupled to the output of the MAC circuit. The output of the local accumulator 518 in processing element 512 can be coupled to the input of the corresponding tap register 514. In some cases, MAC circuit 516 may include a plurality of multiplier circuits configured to produce partial products based on multiplying the input value by a weight value, and local accumulator 518 may be implemented as adder circuits configured to combine the partial products produced by the independent multipliers in MAC circuit 516 into a local sum.

[0056] Typically, the tap registers 514 within the processing element group 510 can be coupled such that the output of one tap register 514 is used as the input of another tap register 514. The output of the last tap register (e.g., tap register 514D or 514H) within the processing element group 510 can be coupled to the input of the global accumulator circuit 520 (also referred to as the "last accumulator" and labeled "FINAL-ACC"). For each depth cycle processed in parallel, a given tap register 514 can be configured to shift the data provided as input from the previous tap register 514, while multiplication and accumulation operations are performed by the processing element 512 corresponding to the given tap register 514. For example, tap register 514B can be configured to shift data provided as input from tap register 514A simultaneously with the operation performed by processing element 512B.

[0057] The output of the global accumulator circuit 520 can be coupled to bus 530, which connects the processing element group 510 to the digital post-processing logic 540. Typically, the digital post-processing logic 540 can use the value of the global accumulator circuit 520 for further processing (e.g., biasing, batch normalization (BN), linear / nonlinear thresholding, quantization, etc.) based on the input data from which the value of the global accumulator circuit 520 was generated. In some cases, the output of the tap register 514 can also be coupled to bus 530. In this case, bus 530 can be an addressable bus to allow the digital post-processing logic 540 to selectively access data placed on bus 530 by the global accumulator circuit 520 (and tap register 514).

[0058] Figure 6 illustrates an example data flow timing diagram 600 for parallel processing of multiple inputs at different depths, according to certain patterns of this disclosure. In this example, the input can be defined as a portion of a large set of input data, such as data values ​​at a set of coordinates in a three-dimensional space with X, Y, and Z axes, where the depth dimension corresponds to the Z-axis in the three-dimensional space. As shown in timing diagram 600, the processing architecture 500 supports parallel processing of multiple inputs at multiple depth periods along the Z-axis. Typically, each depth period can be associated with a set of weights defined in the model that are used to compute the results at that depth period.

[0059] As shown in the figure, portions of the input can be processed simultaneously over a number of depth cycles E. For example, portions of the input located at the same position in the first and second dimensions and at different positions in the third dimension can be processed simultaneously over E depth cycles. While processing these inputs, the values ​​for the inputs stored in the tap register can be accumulated via a global accumulator (e.g., via serial shift). After executing a given depth cycle, the values ​​stored in the global accumulator can be output to digital post-processing logic for further use. To minimize the negative impact on the number of operations supported by the neural network processing architecture over a time period, the number of taps T (e.g., calculated or selected) can be determined (e.g., calculated or selected) based on the number of depth cycles E on which the portions of the input are provided and processed in parallel. For example, in the case where E = total depth cycles D / number of taps T, when E > T + 2, the negative impact on the processing power of the neural network processing architecture can be small or nonexistent (e.g., few or no unused processing elements). When E < T + 2, virtual cycles can be added to minimize the negative impact on the number of operations within a time period supported by the neural network processing architecture.

[0060] FIG7 is a block diagram of an example circuit for depth-by-depth parallel processing of an input portion according to certain aspects of this disclosure. The example circuit in FIG7 includes a more detailed version of the processing element 512 in FIG5. As shown in FIG7, a group of processing elements 510 of a neural network processing circuit configured to process the input portion at multiple depths simultaneously may include a plurality of processing elements 512 and a global accumulator circuit 520. The neural network processing circuit may include multiple groups of processing elements 510, and each group of processing elements 540 in the neural network processing circuit may be configured to process different portions of the input at multiple depths.

[0061] As shown in the figure, each processing element 512 typically includes a multiplication and accumulation (MAC) circuit 516 (e.g., a MAC adder tree) and a local accumulator 518. Typically, the MAC circuit 516 can produce the result of a mathematical operation on a given input and output the result of the mathematical operation to the local accumulator 518. The output of the local accumulator 518 can be provided as input to the selection circuit 712 along with the value stored in the corresponding tap register 514. For example, the selection circuit 711 can be a multiplexer circuit (e.g., a 2:1 multiplexer, where one of the two values ​​input to the multiplexer is output based on the value of the selector control signal), a tri-state buffer, multiple switches, or other circuitry (which allows selection of the output value 716 from the local accumulator 518 or selection of the output value 718 from the tap register 514 for output to the input of the tap register 514). Typically, when the local accumulator 518 completes the calculation of the final accumulator result at the end of E depth cycles (e.g., after E depth cycles have been processed), the selector control signal can control the selection circuit 712 to select the output of the local accumulator 518 as the input to the tap register 514. That is, when the number of depth cycles reaches E, the selector control signal can become high. Otherwise, the selection circuit 712 can select the current value of the tap register 514 as the input to the tap register 514 to save the data currently stored in the tap register.

[0062] At some processing elements 512, the second selection circuit 714 can be used to influence the value stored in the tap register 514. For example, the second selection circuit 714 can be used in processing elements following the initial processing element at the first depth cycle. That is, for the input at the first depth cycle processed by the initial processing element 512A, subsequent processing elements 512B and 512C (and other processing elements after processing element 512A) can use the corresponding second selection circuit 714 to influence the value stored in the corresponding tap register 514 for the processing element. Similar to the selection circuit 712, the second selection circuit 714 can be implemented, for example, by a multiplexer circuit (e.g., a 2:1 multiplexer, where one of the two values ​​input to the multiplexer is output based on the value of the selector input), a tri-state register, multiple switches, or other circuitry (which allows selection of one of multiple values). Control signals (such as signals used to shift out the value of tap register 514 over a certain number of depth cycles equal to the number of taps T) can be used to control the value stored in tap register 512. The value selected by the second selection circuit 714 can be the current value of tap register 514, or it can be the value of tap register 514 in the previous processing element (e.g., the output of tap register 514A of processing element 512A at processing element 512B, as shown). Thus, for processing elements 512B and 512C, tap register 514B (and 514C) can have a first input coupled to the output of selection circuit 712B (and 712C), and a second input coupled to the output of second selection circuit 714B (and 714C).

[0063] The global accumulator circuit 520 includes a selection circuit 722 and a global accumulator 724. The selection circuit 722 can take the current value in the global accumulator 724 and the output of a tap register 514 as inputs, which is associated with the last processing element used to process the input at a defined number of depths (e.g., as shown, tap register 514C is associated with processing element 512C). The selector input to the selection circuit 722 can be used to set the value of the global accumulator 724, such that the value stored in the global accumulator 724 is accumulated over multiple depth cycles. Typically, after E cycles have been processed (e.g., at depth cycle E+1), the value stored in the global accumulator 724 can be output to bus 530, and the global accumulator circuit 520 can be reset.

[0064] Bus 530 is typically coupled to the outputs of one or more tap registers 514 associated with the same or more processing elements 512 and the output of the global accumulator 724 in accumulator circuitry 520. Typically, bus 530 can arbitrate data requests from and to digital post-processing logic 540. For example, bus 530 can select which outputs are available to digital post-processing logic 540 based on blocks defined for a neural network executed on neural network processing circuitry. Thus, values ​​available to digital post-processing logic 540 via bus 530 can include values ​​output by processing elements 512, outputs from multiple processing elements 512, and / or values ​​stored in the global accumulator 724 and output to bus 530.

[0065] Figure 8 illustrates an example 800 of processing element utilization in a neural network processing architecture, where different depths of the input are processed in parallel. As shown, multiple computational units can be aggregated to support processing of the input at multiple depths. In this example, for a computational unit supporting 64 input portions, two computational units can be aggregated to form a virtual computational unit supporting 128 inputs. Each processing element can use four startups and 32 filter channels to process the input. As shown, 25 portions of the input (e.g., 25 different segments of an image) can be processed in parallel at four different depths (e.g., based on a ResNet 50 layer with a 7x7x512 input tensor and a number of corresponding 3x3x512 kernel tensors, and cyclotrons on 25 different fields). Because the input portions can be processed in parallel at different depths of the neural network, the number of unused processing element circuits can be much smaller relative to the number of unused processing element circuits shown in Figure 4. In this example, the problem processed by the neural network processing architecture described herein can be defined based on the parameters discussed above regarding Figure 4 (e.g., depth=512, InputX=7, InputY=7, FilterX=3, FilterY=3, StrideX=1, and StrideY=1), where different depths of the input can be processed in parallel. The number of clock cycles for processing the problem can be defined as: FilterX*FilterY*Depth / Activations=3*3*512 / 4 / 5=231 cycles, or approximately 20% of the number of cycles used in the depth-continuous processing discussed above. The utilization efficiency compared to the maximum theoretical utilization can be (32*50*4*32) / (32*64*4*31)=78%. Furthermore, the number of operations per second can be significantly increased relative to the number of operations per second that can be performed when performing depth-continuous processing of the input via a neural processing architecture operating at the same clock rate. Therefore, as shown in the figure, using deep parallelism to process input can waste fewer computational resources, and the overall performance of the neural processing architecture can be improved compared to neural processing architectures that use deep sequential processing to process input. Example operation

[0066] FIG9 is a flowchart illustrating an example operation 900 for neural network processing according to certain aspects of this disclosure. For example, operation 900 may be performed by neural network processing circuitry of a neural network processing architecture 500 such as FIG5 or FIG7.

[0067] Operation 900 may begin at block 910: receiving input for processing. The input may be a multidimensional data array or a multidimensional tensor, which may be segmented for parallel processing in a neural network.

[0068] In some cases, the input may include data corresponding to a three-dimensional space. The first dimension in the three-dimensional space may correspond to a horizontal dimension. The second dimension in the three-dimensional space may correspond to a vertical dimension. The third dimension in the three-dimensional space may correspond to a depth dimension. For example, the three-dimensional space may be Euclidean space, where the data is represented by dimensions of height, width, and depth, such as data from a three-dimensional image of an object. In another instance, the three-dimensional space may include spatial data on a two-dimensional plane, where the third dimension corresponds to a time channel in the input. For example, this three-dimensional data may include video data, audio data, or other information with a time dimension.

[0069] In some forms, the input may include a plurality of segments. For example, the size of the segments may allow the same portion of the input to be processed in parallel or substantially in parallel at different depths of the neural network. For example, each of the plurality of segments may be a block of the same primitive in different images corresponding to different depths in a 3D image or to different timestamps in temporal data (e.g., different frames in video) (e.g., in terms of horizontal and vertical coordinates). In some forms, the plurality of segments may be based on blocks used to define how the neural network will process the input. Each of the plurality of segments may represent a sub-part or block of the input, the size of which may be defined by the number of blocks into which the input is divided. For example, for a number of i blocks, an image input having primitive X on the horizontal axis and primitive Y on the vertical axis may be divided into blocks of size (X*Y) / i primitives.

[0070] In block 920, for one segment of a plurality of segments of the input, intermediate outputs are generated substantially in parallel at each of a plurality of depths in the neural network. The intermediate outputs may be generated based on weights in the neural network associated with a filter or kernel used to process portions of the input at a given depth in the neural network.

[0071] In some states, generating an intermediate output may include generating the intermediate output via a multiplication and accumulation (MAC) circuit. The value generated by the MAC circuit may be stored in a tap register. In some states, generating an intermediate output may further include shifting the value in the tap register during each of a plurality of processing cycles.

[0072] In block 930, for each segment of the input in a plurality of segments, each intermediate output is accumulated into the final output. In some states, accumulating intermediate outputs into the final output may include accumulating shifted values ​​from the tap registers associated with each corresponding value over a plurality of processing cycles. As discussed, the plurality of processing cycles may correspond to the number of depths processed in parallel within the neural processing unit.

[0073] In block 940, at least the last output is output to the memory bus. In some cases, the last output may be output to the memory bus based on a signal indicating that processing has been completed for a threshold number of depth cycles. By outputting the last output to the memory bus based on this signal, incomplete data may not be placed on the bus.

[0074] As discussed, the memory bus can be a selective bus, allowing data to be selectively sent or provided to digital post-processing blocks for further use. In some cases, each intermediate output can also be output to the memory bus. The memory bus can selectively provide values ​​to digital post-processing blocks based on, for example, information about how the input is segmented in the neural network for processing. Example device with deep parallel processing.

[0075] Figure 10 illustrates an example electronic device 1000. The electronic device 1000 can be configured to perform the methods described herein, including operation 900 described with respect to Figure 9.

[0076] Electronic device 1000 includes a central processing unit (CPU) 1002, which in some cases may be a multi-core CPU. Instructions executed at CPU 1002 may be loaded from program memory associated with CPU 1002 or from memory 1024.

[0077] The electronic device 1000 also includes additional processing blocks customized for specific functions, such as a graphics processing unit (GPU) 1004, a digital signal processor (DSP) 1006, a neural network circuit 1007 with a set of PEs 1009 (for implementing deep parallel processing of inputs to the neural network circuit), a multimedia processing block 1010, and a wireless connectivity processing block 1012. In one implementation, the neural network circuit 1007 is implemented in one or more of the CPU 1002, GPU 1004, and / or DSP 1006.

[0078] In some embodiments, the wireless connectivity processing block 1012 may include components, for example, for third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and / or wireless data transmission standards. The wireless connectivity processing block 1012 is further connected to one or more antennas 1014 to facilitate wireless communication.

[0079] The electronic device 1000 may also include one or more sensor processors 1016 associated with any type of sensor, one or more image signal processors (ISP) 1018 associated with any type of image sensor, and / or navigation processors 1020, which may include satellite-based positioning system components (e.g., Global Positioning System (GPS) or Global Navigation Satellite System (GLONASS)) and inertial positioning system components.

[0080] Electronic device 1000 may also include one or more input and / or output devices 1022, such as a screen, a touch-sensitive surface (including a touch-sensitive display), a physical button, a speaker, a microphone, etc. In some embodiments, one or more processors of electronic device 1000 may be based on an advanced RISC machine (ARM) instruction set, where RISC stands for "Reduced Instruction Set Computing".

[0081] Electronic device 1000 also includes memory 1024, which represents one or more static and / or dynamic memories, such as dynamic random access memory (DRAM), flash memory-based static memory, etc. In this example, memory 1024 includes computer-executable components that can be executed by one or more of the processors described above in electronic device 1000 (including neural network circuitry 1007). The depicted components and other undepicted components can be configured to perform various states of the methods described herein.

[0082] In some configurations, such as when electronic device 1000 is a server device, one or more of the following configurations may be omitted from the example depicted in FIG10: multimedia processing block 1010, wireless connection processing block 1012, antenna 1014, sensor processor 1016, ISP 1018, or navigation processor 1020. Example Terms

[0083] In addition to the individual states described above, specific combinations of states are also within the scope of this disclosure, some of which are described in detail in the following clauses:

[0084] Clause 1: A processing circuit comprising a complex array of processing element (PE) circuits, wherein: each array of PE circuits comprises a plurality of PE circuits configured to process inputs in parallel at a plurality of depths, and each PE circuit comprises: one or more multiplication circuits, each multiplication circuit configured to compute a partial product, and a local accumulator having an input coupled to the output of one or more multiplication circuits, the local accumulator being configured to generate a sum based on the partial product generated by each of the one or more multiplication circuits.

[0085] Clause 2: The processing circuit as described in Clause 1, wherein each PE circuit further includes: a register having an input coupled to the output of a local accumulator.

[0086] Clause 3: The processing circuit as described in Clause 2 further includes a plurality of global accumulators, wherein the output of a register in each PE circuit is coupled to the input of another register in another PE circuit in the same group of PE circuits, or is coupled to the input of one of the global accumulators.

[0087] Clause 4: The processing circuitry as described in Clause 3 further includes a bus, wherein another output of the register in each PE circuit is coupled to the bus.

[0088] Clause 5: The processing circuit as described in Clause 4, wherein the output of each global accumulator in the global accumulator is further coupled to a bus.

[0089] Clause 6: The processing circuit as described in any of Clauses 2 to 5, wherein each PE circuit further includes a first selection circuit having a first input coupled to the output of a local accumulator and an output coupled to the input of a temporary register.

[0090] Clause 7: The processing circuit as described in Clause 6, wherein the first selection circuit in each PE circuit has a second input coupled to the output of the register.

[0091] Clause 8: The processing circuit as described in Clause 7, wherein the first selection circuit includes a 2:1 multiplexer, a tri-state buffer, or a plurality of switches.

[0092] Clause 9: The processing circuit as described in any of Clauses 6 to 8, wherein at least some of the PE circuits in each group of PE circuits further include a second selection circuit having an output coupled to an input of a register, a first input having an output coupled to an output of a register, and a second input having an output coupled to an output of another register in another PE circuit in the same group of PE circuits.

[0093] Clause 10: The processing circuit as described in Clause 9, wherein the second selection circuit includes a 2:1 multiplexer, a tri-state buffer, or a plurality of switches.

[0094] Clause 11: The processing circuitry described in any of Clauses 1 to 10 further includes a plurality of global accumulators, wherein each global accumulator has an input that is coupled to the output of one of the PE circuits in the group of PE circuits.

[0095] Clause 12: The processing circuitry described in any of Clauses 1 to 11, wherein the plurality of PE circuits are further configured to process the plurality of depths of the plurality of inputs in parallel.

[0096] Clause 13: A method of neural network processing, comprising: receiving an input for processing; and for each of a plurality of segments of the received input: generating, substantially in parallel, intermediate outputs for a corresponding depth in the neural network based on weights in the neural network associated with a corresponding depth of a plurality of depths in the neural network; accumulating the intermediate outputs for each corresponding depth into a final output; and outputting the final output to a memory bus.

[0097] Item 14: The method as described in Item 13, wherein: the input includes data from a three-dimensional space, the first dimension of the three-dimensional space corresponding to the horizontal dimension, the second dimension of the three-dimensional space corresponding to the vertical dimension, and the third dimension of the three-dimensional space corresponding to the depth dimension.

[0098] Clause 15: The method as described in Clause 14, wherein: the data from the three-dimensional space includes video data, and the depth dimension corresponds to the time channel in the video data.

[0099] Clause 16: The method described in any of Clauses 13 to 15, wherein generating intermediate outputs for a corresponding depth in the neural network comprises: generating intermediate outputs via a multiplication and accumulation (MAC) circuit; and storing the value of the MAC circuit in a tap register.

[0100] Item 17: The method as described in Item 16, wherein generating intermediate outputs for a corresponding depth in the neural network further includes shifting values ​​in the tap register during each of a plurality of processing cycles.

[0101] Item 18: The method described in any of the items 13 to 17, wherein accumulating intermediate outputs for each corresponding depth into the final output comprises: accumulating shifted values ​​associated with each of the corresponding depths over a plurality of processing cycles stored in one or more tap registers.

[0102] Item 19: The method described in any of the items 13 to 18, wherein outputting the final output to the memory bus includes outputting each intermediate output to the memory bus.

[0103] Item 20: The method described in any of the items 13 to 19, wherein the final output is sent to the memory bus based on a signal indicating that processing has been completed for a number of depth cycles for a threshold.

[0104] Clause 21: An apparatus comprising: a memory bus, a digital post-processing block, and a processor configured to perform operations as described in any of Clauses 13 to 20.

[0105] Clause 22: An apparatus comprising: a component for performing the operation described in any of Clauses 13 to 20. Additional Notes

[0106] The above description is provided to enable those skilled in the art to practice the various embodiments described herein. The examples discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to such embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. For example, the function and arrangement of the discussed elements may be changed without departing from the scope of this disclosure. Various procedures or components may be omitted, substituted, or added as appropriate in various embodiments. For example, the described method may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some embodiments may be combined in other embodiments. For example, any number of embodiments set forth herein may be used to implement an apparatus or practice. Moreover, the scope of this disclosure is intended to cover such apparatus or methods practiced using structures, functions, or structures and functions other than or different from the various embodiments set forth herein. It should be understood that any embodiment of this disclosure may be embodied by one or more elements of the claims.

[0107] As used herein, the term “exemplary” means “serving as an example, instance or illustration.” Any state described herein as “exemplary” is not necessarily to be construed as superior to or advantageous to other states.

[0108] As used herein, the phrase “at least one of” in a list of items means any combination of such items, including a single member. For example, “at least one of a, b, or c” is used to cover any combination of a, b, c, ab, ac, bc, and abc, and any combination of multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0109] As used herein, the term "decision" encompasses a wide variety of actions. For example, "decision" can include calculation, estimation, processing, deduction, investigation, review (e.g., reviewing in a table, database, or other data structure), approval, etc. Furthermore, "decision" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. In addition, "decision" can include resolving, selecting, choosing, establishing, etc.

[0110] The methods disclosed herein include one or more steps or actions for implementing such methods. The method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, various operations of the above methods can be performed by any suitable component capable of performing the corresponding function. Components may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, if the operations shown in the figures are present, such operations may have corresponding equivalent components plus functional components with similar numbering. Example components plus functional components may include components for receiving, components for generating, components for accumulating, and components for outputting, etc. Components for receiving may include input / output blocks (such as input / output block 1012 depicted in FIG. 10), wireless communication blocks (e.g., transceivers) (such as wireless communication block 1012 depicted in FIG. 10), etc. The components used for generation may include one or more processors or processing circuits, such as neural network circuit 1007 and / or processing element 1009 shown in FIG. 10. The components used for accumulation may include one or more processors or processing circuits, such as neural network circuit 100 and / or processing element 1009 shown in FIG. 10. The components used for output may include input / output blocks (such as input / output block 1012 depicted in FIG. 10), wireless communication blocks (e.g., transceivers) (such as wireless communication block 1012 described in FIG. 10), etc.

[0111] The following claims are not limited to the forms shown herein, but should be construed as conforming to the full scope of the language consistent with the claims. In the claims, unless otherwise stated, references to singular elements are not intended to mean "one and only one," but rather "one or more." Unless otherwise stated, the term "some" means one or more. No element of any claim shall be interpreted in accordance with the provisions of § 112(f) of the Patent Act, unless the element is explicitly described using the phrase "component for..." or, in the case of a method claim, the element is described using the phrase "step for...". All structural and functional equivalents of the elements of the various forms described in this disclosure that are known or hereafter known to one of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered in the claims. Furthermore, nothing disclosed herein is intended to be made public, whether or not such disclosure is expressly described in the claims. [Simplified Explanation of the Diagram]

[0010] In order to achieve the manner in which the above-described features of this disclosure are understood in detail, a more specific description briefly outlined above can be obtained by referring to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the drawings only show some typical embodiments of this disclosure and are therefore not to be considered as limiting its scope, as the description may be adapted to other equivalent embodiments.

[0011] Figures 1A to 1D depict examples of various types of neural networks that can be implemented using the various forms of the present disclosure.

[0012] Figure 2 depicts an example of a conventional spin operation that can be implemented using the various forms disclosed herein.

[0013] Figure 3 is a block diagram of an example neural network processing architecture with tightly coupled memory (TCM) and processing element (PE) in which some of the contents of this disclosure can be implemented.

[0014] Figure 4 illustrates the utilization of processing elements in a neural network processing architecture, where inputs are processed serially at different depths.

[0015] Figure 5 illustrates an example neural network processing architecture of some form according to the present disclosure, in which multiple groups of processing elements are configured to process multiple depths of multiple inputs in parallel.

[0016] Figure 6 shows an example data flow timing diagram of different depths for parallel processing of multiple inputs according to certain patterns of the present disclosure.

[0017] Figure 7 is a block diagram of an example circuit for depth-by-depth parallel processing of inputs according to certain states of this disclosure.

[0018] Figure 8 illustrates the utilization of processing elements in a neural network processing architecture, where inputs are processed in parallel at different depths.

[0019] Figure 9 illustrates some patterns according to this disclosure, example operations that a processing system can perform to process multiple depths of input in parallel.

[0020] Figure 10 is a block diagram of an example electronic device having neural network processing circuitry of multiple depths in which inputs are processed in parallel, according to certain aspects of this disclosure.

[0021] For ease of understanding, the same element symbols have been used where possible to indicate the same elements common to all figures. It is anticipated that elements and features of one pattern can be advantageously incorporated into other patterns without further description. [Biomaterial Storage]

[0113] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. A processing circuit comprising a complex array of processing element (PE) circuits, wherein: Each group of PE circuits includes a plurality of PE circuits and a global accumulator. The plurality of PE circuits are configured to process an input in parallel at a plurality of depths. Each PE circuit includes: one or more multiplication circuits, each configured to compute a portion of a product; a local accumulator having an input coupled to an output of the one or more multiplication circuits, the local accumulator being configured to generate a sum based on the portion of the product computed by each of the one or more multiplication circuits; and a tap register coupled to the output of the local accumulator and one or more tap registers of other PE circuits in the group of PE circuits, such that the tap registers in each group of PE circuits are connected in series; wherein the output of a final tap register among the series-connected tap registers in each group of PE circuits is coupled to an input of the global accumulator of the group of PE circuits. Each tap register is configured to synchronously shift the output received from the coupled local accumulator and any output received from the coupled tap register through the serially connected equal tap registers; and the global accumulator circuit accumulates the value received from the serially connected equal tap registers and outputs the accumulated value when performing a given depth period.

2. The processing circuitry as claimed in claim 1 further includes a bus, wherein another output of the tap register in each PE circuit is coupled to the bus.

3. The processing circuit as described in claim 2, wherein an output of the global accumulator is further coupled to the bus.

4. The processing circuitry as claimed in claim 1, wherein the plurality of PE circuits are further configured to process the plurality of inputs in parallel at the plurality of depths.

5. A method for a processing circuit including a complex array of processing elements (PE) circuitry, each PE circuitry including a plurality of PE circuits and a global accumulator, the plurality of PE circuits being configured to process an input in parallel at a plurality of depths, comprising the steps of: calculating a partial product by means of one or more multiplication circuits of one of the plurality of PE circuits; generating a sum by means of the partial product calculated by each of the one or more multiplication circuits; receiving the generated sum and an output from a previous tap register of a plurality of tap registers serially connected to the PE circuitry at a tap register; and synchronously shifting the received generated sum and the output received from the tap register to a subsequent tap register of the plurality of tap registers serially connected to the PE circuitry. The method further includes: at a global accumulator circuit coupled to a final tap register among the serially connected plurality of tap registers, accumulating the value received from the serially connected tap registers; and outputting the accumulated value when performing a given depth period.

Images