Apparatus, method, and computer-readable medium for performing machine learning tasks

Abstract

Some embodiments provide an apparatus for signal processing in a neural network. The apparatus typically includes a first set of in-memory computing (CIM) units configured as a first core for neural network computation and a second set of CIM units configured as a second core for neural network computation. The first set of CIM units includes one or more first columns and a first plurality of rows of a CIM array, and the second set of CIM units includes one or more second columns and a second plurality of rows of a CIM array. In some embodiments, the one or more first columns differ from the one or more second columns, and the first plurality of rows differ from the second plurality of rows.

Description

[Technical Field]

[0001] Cross-reference to related applications

[0002] This patent application claims priority to U.S. Application No. 17 / 361,784, filed June 29, 2021, which has been assigned to the assignee of this application and is incorporated herein by reference in its entirety.

[0003] The nature of this case relates to performing machine learning tasks, specifically to in-memory computing architectures and data streaming. [Previous Technology]

[0004] Machine learning is generally the process of generating trained models (e.g., artificial neural networks, trees, or other structures) that represent a generalized fit to a set of prior known training data. Applying a trained model to new data produces inference that can be used to gain insights into the new data. In some cases, applying a model to new data is described as "running inference" on the new data.

[0005] Due to the surge in the use of machine learning (or artificial intelligence) for various tasks, there has been a demand for more efficient processing of machine learning model data. In some cases, dedicated hardware, such as machine learning accelerators, can be used to enhance the ability of processing systems to process machine learning model data. However, such hardware requires space and power, which are not always available on processing devices. For example, "edge processing" devices (such as mobile devices, always-on devices, Internet of Things (IoT) devices, etc.) must strike a balance between processing power and power and packaging constraints. Furthermore, accelerators may need to move data on a shared data bus, which can result in significant power consumption and latency for other programs on the shared data bus. Therefore, other forms of processing systems are being considered for processing machine learning model data.

[0006] Memory devices are another example of a processing system that can be used to process machine learning model data via so-called computation in memory (CIM) programs. Unfortunately, without additional hardware components such as digital multiply-and-accumulate circuits (DMACs) and associated peripherals, conventional CIM programs may be unable to process complex model structures such as depthwise separable spiral neural networks. These additional hardware components use additional space, power, and complexity in their implementation, which often diminishes the advantages of using memory devices as additional computing resources. Even when auxiliary states of the processing system have DMACs that can be used to perform processing that cannot be performed directly in memory, moving data into and out of these auxiliary states requires time and power, thus reducing the benefits of CIM programs.

[0007] Accordingly, there is a need for systems and methods for performing in-memory computations for a wider range of machine learning model architectures, such as deep separable spiral neural networks. [Summary of the Invention]

[0008] Certain embodiments provide an apparatus for signal processing in a neural network. The apparatus generally includes a first set of in-memory computing (CIM) units configured as a first core for neural network computation and a second set of CIM units configured as a second core for neural network computation. The first set of CIM units includes one or more first columns and a first plurality of rows of a CIM array, and the second set of CIM units includes one or more second columns and a second plurality of rows of a CIM array. In some embodiments, the one or more first columns differ from the one or more second columns, and the first plurality of rows differ from the second plurality of rows.

[0009] Certain forms provide a method for signal processing in a neural network. The method generally includes performing neural network computation by loading a first set of CIM units having a first plurality of weight parameters with a first core via one or more first columns, the first CIM unit set comprising one or more first columns and a first plurality of rows of a CIM array. The method may also include performing neural network computation by loading a second set of CIM units having a second plurality of weight parameters with a second core via one or more second columns, the second CIM unit set comprising one or more second columns and a second plurality of rows of a CIM array. The one or more first columns may differ from the one or more second columns, and the first plurality of rows may differ from the second plurality of rows. The method may also include performing neural network computation by applying a first start input to the first plurality of rows and a second start input to the second plurality of rows.

[0010] Certain forms provide a non-transitory computer-readable medium including instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a signal processing method in a neural network. The method generally includes performing neural network computation by loading a first set of CIM cells having a first plurality of weight parameters with a first core via one or more first columns, the first CIM cell set including one or more first columns and a first plurality of rows of a CIM array. The method may also include performing neural network computation by loading a second set of CIM cells having a second plurality of weight parameters with a second core via one or more second columns, the second CIM cell set including one or more second columns and a second plurality of rows of a CIM array. The one or more first columns may differ from the one or more second columns, and the first plurality of rows may differ from the second plurality of rows. The method may also include performing neural network computation by applying a first start input to the first plurality of rows and a second start input to the second plurality of rows.

[0011] Other embodiments include: a processing system configured to perform the foregoing methods and the methods described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and the methods described herein; a computer program product embodied on a computer-readable storage medium, including code for performing the foregoing methods and the methods further described herein; and a processing system including components for performing the foregoing methods and the methods further described herein.

[0012] The following description and related figures illustrate in detail certain illustrative features of one or more states.

Implementation Method

[0029] This invention provides apparatus, methods, processing systems, and computer-readable media for performing computation in memory (CIM) of machine learning models, including depthwise separable spiral neural networks. Some embodiments involve CIM cells of CIM arrays configured for different cores, wherein the CIM cells are implemented on different rows and columns of the CIM array to facilitate parallel operation of a first core and a second core. For example, a first core may be implemented on the first row and first column of the CIM array, and a second core may be implemented on the second row and second column of the CIM array, wherein the first row and second column are different from the second row and second column. Each core implemented on different rows and columns may be coupled to an analog-to-digital converter (ADC), thereby allowing parallel depth-wise (DW) computation and analog-to-digital conversion via the cores. The results of the DW computation may be input to a nonlinear startup circuit for further processing and to another CIM array for point-wise (PW) computation, as described in more detail herein.

[0030] CIM-based machine learning (ML) / artificial intelligence (AI) task accelerators can be used for a wide variety of tasks, including image and audio processing. Furthermore, CIM can be based on various types of memory architectures, such as DRAM, SRAM (e.g., based on SRAM cells in Figure 5), MRAM, and ReRAM, and can be attached to various types of processing units, including central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), AI accelerators, etc. Generally, CIM can beneficially reduce the "memory wall" problem, where data moving into and out of memory consumes more power than data computation. Therefore, significant power savings can be achieved by performing computations within memory. This is particularly useful for various types of electronic devices, such as low-power edge processing devices, mobile devices, etc.

[0031] For example, a mobile device may include a memory device configured for storing data and performing computational operations within memory. The mobile device may be configured to perform ML / AI operations based on data generated by the mobile device, such as image data generated by the mobile device's camera sensor. The mobile device's memory controller unit (MCU) can thus load weights from another onboard memory (e.g., flash memory or RAM) into the CIM array of the memory device and allocate input feature buffers and output (e.g., start) buffers. The processing device can then begin processing the image data by loading, for example, a layer into the input buffer and processing that layer with the weights loaded into the CIM array. This processing can be repeated for each layer of the image data, and the output (e.g., start) can be stored in the output buffer, which the mobile device can then use for ML / AI tasks such as face recognition. A brief background on neural networks, deep neural networks, and deep learning.

[0032] Neural networks are organized into multiple layers of interconnected nodes. Generally, nodes (or neurons) are where computation occurs. For example, a node can combine input data with a set of weights (or coefficients) that amplify or attenuate the input data. Thus, the amplification or attenuation of the input signal can be thought of as an assignment of the relative importance of various inputs to the task the network is attempting to learn. Generally, the product of the input weights is summed (or accumulated), and the sum is then passed through the node's initiation function to determine whether the signal should proceed further in the network and to what extent.

[0033] In the most basic implementation, a neural network can have an input layer, a hidden layer, and an output layer. "Deep" neural networks typically have more than one hidden layer.

[0034] Deep learning is a method for training deep neural networks. In essence, deep learning maps the network's inputs to its outputs, and is therefore sometimes called a "universal approximator" because it can learn to approximate an unknown function between any input x and any output y. In other words, deep learning finds the correct f that transforms x into y.

[0035] More specifically, deep learning trains each layer node based on a set of different features that are the output of the previous layer. Therefore, the features become more complex for each successive layer of a deep neural network. Thus, deep learning is powerful because it can progressively extract higher-level features from the input data and perform complex tasks (such as object recognition) by learning to represent the input with continuously higher-level abstractions in each layer, thereby building useful feature representations of the input data.

[0036] For example, if visual data is used for presentation, the first layer of a deep neural network can learn to identify relatively simple features in the input data, such as edges. In another instance, if auditory data is used for presentation, the first layer of a deep neural network can learn to identify spectral power at specific frequencies in the input data. The second layer of the deep neural network can then learn to identify combinations of features based on the output of the first layer, such as simple shapes in visual data or combinations of sounds in auditory data. Higher layers can then learn to identify complex shapes in visual data or words in auditory data. Higher layers can learn to identify common visual objects or spoken phrases. Therefore, deep learning architectures can perform particularly well when applied to problems with a natural hierarchical structure. The layers in a neural network are linearly connected.

[0037] Neural networks such as deep neural networks can be designed to have multiple connectivity patterns between layers.

[0038] Figure 1A illustrates an example of a fully connected neural network 102. In the fully connected neural network 102, nodes in the first layer transmit their outputs to each node in the second layer, such that each node in the second layer receives inputs from each node in the first layer.

[0039] Figure 1B illustrates an example of a locally connected neural network 104. In the locally connected neural network 104, nodes in the first layer can be connected to a finite number of nodes in the second layer. More generally, the locally connected layers of the locally connected neural network 104 can be configured such that each node in the layer will have the same or similar connection patterns, but the connection strengths (or weights) can have different values ​​(e.g., 110, 112, 114, and 116). The locally connected connection patterns can generate spatially distinct receptive fields in higher layers because higher-level nodes in a given region can receive inputs that are tuned to the properties of a restricted portion of the network's total input through training.

[0040] One type of locally connected neural network is the spiral neural network. Figure 1C illustrates an example of a spiral neural network 106. The spiral neural network 106 can be configured such that the connection strength associated with the input of each node in the second layer is shared (e.g., 108). Spiral neural networks are well-suited for problems where the spatial location of the input is meaningful.

[0041] One type of spiral neural network is the deep convolutional network (DCN). A deep convolutional network is a network with multiple convolutional layers, which can be further configured with, for example, pooling layers and normalization layers.

[0042] Figure 1D illustrates an example of DCN 100, which is designed to identify visual features in image 126 generated by image capturing device 130. For example, if image capturing device 130 is a camera mounted in a vehicle, DCN 100 can be trained using various supervised learning techniques to recognize traffic signs and even numbers on traffic signs. DCN 100 can also be trained for other tasks, such as recognizing lane markings or traffic lights. These are just some example tasks, and many other tasks are possible.

[0043] In this example, DCN 100 includes a feature extraction part and a classification part. Upon receiving image 126, cyclotron layer 132 applies a cyclotron core (e.g., as illustrated and described in FIG2) to image 126 to generate a first set of feature maps (or intermediate start) 118. Generally, a "core" or "filter" comprises a multidimensional array of weights designed to emphasize different states of the input data channels. In various instances, "core" and "filter" can be used interchangeably to represent multiple sets of weights applied in a cyclotron neural network.

[0044] Subsequently, the first set of feature maps 118 can be resampled via a pooling layer (e.g., a max pooling layer, not shown) to generate a second set of feature maps 120. The pooling layer can reduce the size of the first set of feature maps 118 while retaining most of the information to improve model performance. For example, the second set of feature maps 120 can be sampled from 28×28 to 14×14 via a pooling layer.

[0045] This procedure can be repeated through many layers. In other words, the second set of feature maps 120 can be further spun through one or more subsequent spun layers (not shown) to produce one or more subsequent sets of feature maps (not shown).

[0046] In the example of Figure 1D, the second set of feature maps 120 is provided to the fully connected layer 124, which in turn produces an output feature vector 128. Each feature in the output feature vector 128 may include a number corresponding to a possible feature of the image 126 (such as "symbol", "60", and "100"). In some cases, a softmax function (not shown) can convert the numbers in the output feature vector 128 into probabilities. In this case, the output 122 of the DCN 100 is the probability that the image 126 includes one or more features.

[0047] The softmax function (not shown) can convert the elements of the output feature vector 128 into probabilities so that the output 122 of DCN 100 is one or more probabilities that the image 126 includes one or more features (such as the symbol with the number "60" in the input image 126). Therefore, in this example, the probabilities of "symbol" and "60" in the output 122 should be higher than the probabilities of other outputs in the output 122 (such as "30", "40", "50", "70", "80", "90" and "100").

[0048] Before training DCN 100, the output 122 generated by DCN 100 may be incorrect. Therefore, the error between output 122 and the prior known target output can be calculated. For example, the target output here is an indication of the image 126 including the "symbol" and the digit "60". Subsequently, using the known target output, the weights of DCN 100 can be adjusted via training so that the subsequent output 122 of DCN 100 reaches the target output.

[0049] To adjust the weights of DCN 100, the learning algorithm can compute the gradient vector of the weights. The gradient indicates the amount by which the error will increase or decrease when the weights are adjusted in a particular way. The weights can then be adjusted to reduce the error. This method of adjusting the weights can be called "backpropagation" because it involves "back-passing" through the layers of DCN 100.

[0050] In practice, the error gradient of the weights can be calculated using a small number of examples, making the calculated gradient approximate the true error gradient. This approximation method can be called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system has stopped decreasing, or until the error rate has reached the target level.

[0051] After training, DCN 100 can be used to represent new images, and DCN 100 can generate inferences (such as classification), or the probabilities of various features being located in the new images. Rotation technique for rotational neural networks.

[0052] Rotary networks are generally used to extract useful features from an input dataset. For example, in a rotary neural network as described above, a rotary network can use its weights to automatically learn cores and / or filters during training to extract different features. The extracted features are then combined for inference.

[0053] A startup function can be applied before and / or after each layer of a spiral neural network. A startup function is generally a mathematical function (e.g., an equation) that determines the output of a node in the neural network. Therefore, the startup function determines whether a node should pass information based on whether the node's input is relevant to the model's prediction. In one instance, in the case of a spiral (i.e., y = x), both x and y can generally be considered "startup". However, for a specific spiral operation, x can also be called "pre-startup" or "input startup" because it exists before the specific spiral, and y can be called output startup or feature map.

[0054] Figure 2 illustrates an example of conventional rotation in which a 5×5×3 rotation core 204 and a step size (or stride) 1 are used to rotate a 12-pixel × 12-pixel × 3-channel input image. The resulting feature map 206 is 8-pixel × 8-pixel × 1-channel. As can be seen in this example, conventional rotation can change the dimensions of the input data (here, from 12×12 pixels to 8×8 pixels) compared to the output data, including the channel dimension (here, from 3 channels to 1 channel).

[0055] One method to reduce computational burden (e.g., measured in floating point operations per second, FLOPs) and the number of parameters associated with neural networks including spiral layers is to factorize the spiral layers. For example, spatially separable spirals, such as those illustrated in Figure 2, can be factored into two components: (1) deep spirals, where each spatial channel is spiraled independently via deep spirals (e.g., spatial fusion); and (2) point spirals, where all spatial channels are linearly combined (e.g., channel fusion). Examples of deeply separable spirals are illustrated in Figures 3A and 3B. Typically, during spatial fusion, the network learns features from the spatial plane, while during channel fusion, the network learns relationships between such features across channels.

[0056] In one example, separable depth-based spin can be implemented using a 3×3 core for spatial fusion and a 1×1 core for channel fusion. Specifically, channel fusion can be implemented using a 1×1×d core that iterates through each point of the input image at a depth of d, where the depth d of the core roughly matches the number of channels in the input image. Channel fusion via point-based spin is useful for efficient dimensionality reduction. Applying 1×1×d cores and adding a startup layer after the cores can give the network increased depth, which can improve its performance.

[0057] Figures 3A and 3B illustrate examples of depth-separable spin operations.

[0058] Specifically, in FIG3A, the 12-pixel × 12-pixel × 3-channel input image 302 is cyclotronically processed with a filter comprising three separate cores 304A-304C (each core having a dimension of 5×5×1) to produce an 8-pixel × 8-pixel × 3-channel feature map 306, wherein each channel is generated by the respective cores among 304A-304C.

[0059] Subsequently, the feature map 306 is further rotated using a point rotation operation with a dimension of 1×1×3 for the core 308 (e.g., the core) to produce an 8-pixel × 8-pixel × 1-channel feature map 310. As illustrated in this example, the feature map 310 has a reduced dimension (1 channel versus 3 channels), which allows for more efficient computation using the feature map 310.

[0060] Although the results of the deep separable spinners in Figures 3A and 3B are basically similar to those of the conventional spinners in Figure 2, the number of computations is significantly reduced, and therefore the deep separable spinners provide a significant efficiency gain where the network design allows.

[0061] Although not illustrated in Figure 3B, multiple (e.g., m) point-based cyclotron cores 308 (e.g., individual elements of a filter) can be used to increase the channel dimension of the cyclotron output. Thus, for example, m = 256 1×1×3 cores 308 can be generated, each core 308 outputting an 8-pixel × 8-pixel × 1-channel feature map (e.g., 310), and these feature maps can be stacked to obtain a resulting 8-pixel × 8-pixel × 256-channel feature map. The resulting increase in channel dimension provides more parameters for training, which can improve the ability of the cyclotron neural network (e.g., in the input image 302) to recognize features. Example of a computation within memory (CIM) architecture.

[0062] Figure 4 illustrates an exemplary in-memory computing (CIM) array 400 configured to perform machine learning model computation according to the present invention. In this example, the CIM array 400 is configured to use hybrid analog / digital operations for artificial neural networks to analogize MAC operations. Accordingly, as used herein, the terms multiplication and addition may refer to such analog operations. The CIM array 400 can be used to implement the processing techniques described herein.

[0063] In the illustrated configuration, the CIM array 400 includes precharge word lines (PCWL) 425a, 425b and 425c (collectively referred to as 425), read word lines (RWL) 427a, 427b and 427c (collectively referred to as 427), analog-to-digital converters (ADC) 410a, 410b and 410c (collectively referred to as 410), digital processing unit 413, bit lines 418a, 418b and 418c (collectively referred to as 418), PMOS transistors 411a-111i (collectively referred to as 411), NMOS transistors 413a-413i (collectively referred to as 413), and capacitors 423a-423i (collectively referred to as 423).

[0064] The weights associated with the neural network layers can be stored in the SRAM cells of the CIM array 400. In this example, binary weights are illustrated in the SRAM bit cells 405a-405i of the CIM array 400. Input initiation (e.g., input values ​​of an input vector) is provided on the PCWL 425a-c.

[0065] Multiplication occurs in each bit cell 405a-405i of the CIM array 400 associated with a bit line, and the summation (addition) of the results of multiplication in all bit cells occurs on the same bit line for a column. The multiplication in each bit cell 405a-405i is an equivalent operation to the AND operation of the corresponding start and weight, where the result is stored as charge on the corresponding capacitor 423. For example, a charge of 1 is produced only when start is 1 (here, PCWL is 0 for start 1 because PMOS is used) and weight is 1, and thus charge is produced on capacitor 423.

[0066] For example, during the accumulation phase, RWL 427 is switched to high, allowing any charge on capacitor 423 (based on the corresponding bit cell (weight) and PCWL (start-up) value) to be accumulated on the corresponding bit line 418. The voltage value of the accumulated charge is then converted into a digital value by ADC 410 (where, for example, the output value may be a binary value indicating whether the total charge is greater than a reference voltage). This digital value (output) can be provided as input to another state of the machine learning model, such as the next layer.

[0067] When the activation on the precharge word lines (PCWL) 425a, 425b, and 425c is, for example, 1, 0, and 1, the sum of bit lines 418a-418c corresponds to 0+0+1=1, 1+0+0=1, and 1+0+1=2, respectively. The outputs of ADCs 410a, 410b, and 410c are passed to the digital processing unit 413 for further processing. For example, if the CIM 100 is processing multi-bit weight values, the digital outputs of ADC 110 can be summed to produce the final output.

[0068] An exemplary 3×3 CIM circuit 400 can be used, for example, to perform efficient 3-channel cyclotron for a three-element core (or filter), wherein the weight of each core corresponds to an element in each of the three columns, such that for a given three-element receptive field (or input data patch), the output for each of the three channels is computed in parallel.

[0069] Note that although Figure 4 depicts an example of a CIM using SRAM cells, other memory types can also be used. For example, Dynamic Random Access Memory (DRAM), Magnetoresistive Random Access Memory (MRAM), and Resistive Random Access Memory (ReRAM or RRAM) can also be used in other embodiments.

[0070] Figure 5A illustrates additional details of the exemplary bit cell 500.

[0071] The state pattern of FIG5A can be an example of the state pattern of FIG4 or related to the state pattern of FIG4. Specifically, bit line 521 is similar to bit line 418a, capacitor 523 is similar to capacitor 423 of FIG4, read word line 527 is similar to read word line 427a of FIG4, and precharge word line 525 is similar to precharge word line 425a of FIG4, PMOS transistor 511 is similar to PMOS transistor 411a of FIG1, and NMOS transistor 513 is similar to NMOS transistor 413 of FIG1.

[0072] Bit cell 500 includes a static random access memory (SRAM) cell 501 (which may represent SRAM bit cell 405a of FIG. 4), a transistor 511 (e.g., a PMOS transistor), a transistor 513 (e.g., an NMOS transistor), and a capacitor 523 coupled to ground. Although a PMOS transistor is used for transistor 511, other transistors (e.g., NMOS transistors) can be used instead of the PMOS transistor with corresponding adjustments (e.g., inversion) to their respective control signals. This also applies to other transistors described herein. According to a configuration of the invention, additional transistors 511 and 513 are included to implement a computational array within the memory. In one configuration, SRAM cell 501 is a conventional six-transistor (6T) SRAM cell.

[0073] For multiple startups, the program for weighting the bit cells can be executed once. For example, during operation, SRAM cell 501 receives only one bit of information at nodes 517 and 519 via write word line (WWL) 516. For example, during a write operation (when WWL 216 is high), if write bit line (WBL) 229 is high (e.g., "1"), then node 217 is set high and node 219 is set low (e.g., "0"); or if WBL 229 is low, then node 217 is set low and node 219 is set high. Conversely, during a write operation (when WWL 216 is high), if write bit bar line (WBBL) 231 is high, then node 217 is set low and node 219 is set high; or if WBBL 229 is low, then node 217 is set high and node 219 is set low.

[0074] The programming of the weights can be followed by an input and multiplication step to charge the capacitors according to the corresponding products. For example, transistor 511 is activated by a start signal (PCWL signal) from the precharge word line (PCWL) 525 of the computation array in memory to perform the multiplication step. Subsequently, transistor 513 is activated by a signal through another word line of the computation array in memory (e.g., read word line (RWL) 527) to perform the accumulation of multiplication values ​​from bit cell 500 with other bit cells of the array, as described above with respect to FIG4.

[0075] If node 517 is "0" (e.g., when the stored weight value is "0"), capacitor 523 will not be charged when the low PCWL indicates a start-up "1" at the gate of transistor 511. Accordingly, no charge is supplied to bit line 521. However, if node 517 corresponding to the weight value is "1" and PCWL is set low (e.g., when the start-up input is high), this turns on PMOS transistor 511, which acts as a short circuit, thereby allowing charging of capacitor 523. After charging capacitor 523, transistor 511 is turned off, thus storing charge in capacitor 523. To move charge from capacitor 523 to bit line 521, NMOS transistor 513 is turned on via RWL 527, thereby causing NMOS transistor 513 to act as a short circuit.

[0076] Table 1 shows an example of a computation array operation within a memory cell set according to the AND operation, such as that which can be implemented by bit cell 500 in Figure 5A. start up PCWL Unit node (weight) Capacitor Node 1 0 1 1 1 0 0 0 0 1 1 0 0 1 0 0 Table 1: AND Operations

[0077] The first column (Start) of Table 1 includes the possible values ​​of the input start signal.

[0078] The second column (PCWL) of Table 1 includes PCWL values, which, according to the present invention, activate the transistor designed to perform in-memory computing functions. Since transistor 511 in this example is a PMOS transistor, the PCWL value is the reciprocal of the activation value. For example, the in-memory computing array includes transistor 511 activated by an activation signal (PCWL signal) via the precharge word line (PCWL) 525.

[0079] The third column (cell node) of Table 1 includes the weight values ​​stored in the SRAM cell node (e.g., corresponding to the weights in a weight tensor that can be used in a spin operation).

[0080] The fourth column (capacitor nodes) of Table 1 illustrates the product of the results of charge being stored on the capacitors. For example, charge can be stored at a node of capacitor 523 or at a node of one of capacitors 423a-423i. When transistor 513 is activated, charge from capacitor 523 is moved to bit line 521. For example, referring to transistor 511, capacitor 523 is charged (e.g., capacitor node is "1") when the weight at cell node 517 is "1" (e.g., high voltage) and the input activation is "1" (therefore PCWL is "0"). For all other combinations, the value of the capacitor node will be 0.

[0081] Figure 5B illustrates additional details of another exemplary bit cell 550.

[0082] Primarily based on an additional precharge word line 552 coupled to an additional transistor 554, bit cell 550 is different from bit cell 500 in FIG5A.

[0083] Table 2 shows examples of memory array operations similar to those in Table 1, except for those configured according to the XNOR operation, such as those that can be implemented by bit cell 550 in Figure 5B. start up PCWL1 PCWL2 Unit node (weight) Capacitor Node 1 0 1 1 1 1 0 1 0 0 0 1 0 1 0 0 1 0 0 1 Table 2: XNOR Operations

[0084] The first column (Start) of Table 2 includes the possible values ​​of the input start signal.

[0085] The second column (PCWL1) of Table 2 includes the PCWL1 value, which, according to the present invention, activates the transistor designed to perform computational functions within the memory. Here, similarly, transistor 511 is a PMOS transistor, and the PCWL1 value is the reciprocal of the activation value.

[0086] The third column (PCWL2) of Table 2 includes PCWL2 values, which, according to the case specifications, activate another transistor designed to perform computational functions within the memory.

[0087] The fourth column (cell node) of Table 2 includes the weight values ​​stored in the SRAM cell node (e.g., corresponding to the weights in a weight tensor that can be used in a spin operation).

[0088] The fifth column (capacitor node) of Table 2 is illustrated as the product of the result of charge being stored in capacitors (such as capacitor 523).

[0089] Figure 6 illustrates an example timing diagram 600 of various signals during the operation of a compute-in-memory (CIM) array.

[0090] In the illustrated example, the first row of timing diagram 600 illustrates the precharge word line PCWL (e.g., 425a in Figure 4 or 525 in Figure 5A) going low. In this example, a low PCWL indicates an activation "1". When PCWL is low, the PMOS transistor is turned on, which allows charging of the capacitor (if the weight is "1"). The second row illustrates the read word line RWL (e.g., read word line 427a in Figure 4 or read word line 527 in Figure 5A). The third row illustrates the read bit line RBL (e.g., 418 in Figure 4 or 521 in Figure 5A), the fourth row illustrates the analog-to-digital converter (ADC) readout signal, and the fifth row illustrates the reset signal.

[0091] For example, referring to transistor 511 in FIG5A, when the read word line RWL is high, the charge from capacitor 523 is gradually transferred to read bit line RBL.

[0092] The total charge / current / voltage (e.g., 403 in Figure 4 or the total charge from bit line 521 in Figure 5A) is passed to a comparator or ADC (e.g., ADC 411 in Figure 4), at which the total charge is converted into a digital output (e.g., a digital signal / value). The summation of the charge can occur in the accumulation region of timing diagram 600, and the readout from the ADC can be associated with the ADC readout region of timing diagram 600. After obtaining the ADC readout, a reset signal discharges all capacitors (e.g., capacitors 423a-423i) to prepare for processing the next set of start-up inputs.

[0093] The parallel processing techniques of this case can facilitate any type of edge computing involving artificial neural networks. These techniques can be applied to the inference stage or any other stage of neural network processing. Although the illustrated example is based on a binary network (which can be used when high precision is not required), the same concept applies to networks using multi-bit weights. Example of in-memory spin processing.

[0094] Figure 7 illustrates an exemplary spiral layer architecture 700 implemented by a computation in memory (CIM) array 708. The spiral layer architecture 700 may be part of a spiral neural network (e.g., as described above with respect to Figure 1D) and is designed to process multidimensional data (such as tensor data).

[0095] In the illustrated example, the input 702 of the swirl layer architecture 700 has dimensions of 38 (height) × 11 (width) × 1 (depth). The output 704 of the swirl layer has dimensions of 34 × 10 × 64, and includes 64 output channels corresponding to the 64 cores of the filter tensor 714 applied as part of the swirl procedure. Furthermore, in this example, each of the 64 cores of the filter tensor 714 (e.g., exemplary core 712) has a size of 5 × 2 × 1 (the cores of the filter tensor 714 together correspond to a 5 × 2 × 64 filter).

[0096] During the spin procedure, each 5×2×1 core spins with input 702 to produce a 34×10×1 layer of output 704. During spin, the 640 weights of filter tensor 714 (5×2×64) can be stored in a memory computation (CIM) array 708, which in this example includes one column for each core (i.e., 64 columns). Subsequently, the initiation of each 5×2 receptive field (e.g., receptive field input 706) is input to CIM array 708 using word lines (e.g., 716) and multiplied by the corresponding weights to produce a 1×1×64 output tensor (e.g., output tensor 710). Output tensor 704 represents the sum of the 1×1×64 individual output tensors for all receptive fields (e.g., receptive field input 706) of input 702. For simplicity, Figure 7 only illustrates a few example lines for the inputs and outputs of the CIM array 708.

[0097] In the illustrated example, the CIM array 708 includes word lines 716 (via which the CIM array 708 receives a receptive field (e.g., receptive field input 706)) and bit lines 718 (corresponding to columns of the CIM array 708). Although not illustrated, the CIM array 708 may also include precharged word lines (PCWL) and read word lines RWL (as described above with respect to Figures 4 and 5).

[0098] In this example, word line 716 is used for initial weight definition. However, once the initial weight definition is established, the startup input initiates a specially designed line in the CIM bit cell to perform MAC operation. Therefore, each intersection of bit line 718 and word line 716 represents a filter weight value, which is multiplied by the input startup on word line 716 to produce a product. The individual products along each bit line 718 are then summed to produce the corresponding output value of the output tensor 710. The sum can be charge, current, or voltage. In this example, the size of the output tensor 704 after processing the entire input 702 of the cyclotron layer is 34 × 10 × 64, although the CIM array 708 only produces 64 filter outputs at a time. Therefore, the processing of the entire input 702 can be completed within 34 × 10 or 340 cycles. CIM Architecture for Deeply Separable Cyclotrons

[0099] Computation in Memory (CIM)-based artificial intelligence (AI) hardware (HW) accelerators can be used for a variety of tasks, including image, sensor, and audio processing AI tasks. CIM can help reduce the problems associated with power consumption when moving data from memory. In some cases, data movement may consume more power than computation. Due to the weight-invariant nature of CIM, using CIM can lead to power savings. In other words, weights used for neural network computation can be stored in, for example, random access memory (RAM) (such as static random access memory (SRAM) memory cells), thereby allowing computation to be performed within memory, resulting in reduced power consumption.

[0100] Vector-matrix multiplication blocks implemented in the memory of a CIM architecture typically perform well with conventional spiral neural network processing, but they are not efficient for supporting deep separable spiral neural networks, as can be found in many existing machine learning architectures. For example, existing CIM architectures generally cannot perform deep separable spiral neural network processing in a single stage because each multidimensional filter uses a different input channel. Therefore, filter weights in the same row may not share the same start input for different channels. Consequently, a matrix-matrix multiplication (MxM) architecture is generally required to support deep separable spiral processing within a single phase period.

[0101] Traditional solutions to address this drawback include adding separate digital MAC blocks to handle the processing of deep, separable spinner parts, while CIM arrays can handle point, separable spinner parts. However, this hybrid approach leads to increased data movement, which can offset the memory efficiency advantage of the CIM architecture. Furthermore, the hybrid approach generally involves additional hardware (e.g., digital multiply-and-accumulate (DMAC) components), which increases space and power requirements and processing latency. In addition, the use of DMAC can affect the timing of processing operations and cause delays beyond model output timing constraints (or other dependencies). To address this problem, various trade-offs may be necessary, such as reducing the frame rate of the input data, increasing the clock rate of processing system components (including CIM arrays), reducing the size of input features, etc.

[0102] The CIM architecture described herein improves timing performance for deep, separable spinner processing operations. Compared to conventional architectures that use more hardware (e.g., DMAC) and / or more data movement, these improvements advantageously result in fewer cycle times for deep, separable spinner operations and achieve higher total operations per second (TOPS), i.e., TOPS / W.

[0103] Figure 8 illustrates a CIM system 800 comprising multiple CIM arrays according to certain aspects of this case.

[0104] As shown in the figure, the CIM system 800 includes a CIM array 802 configured for depthwise (DW) rotation and a CIM array 804 configured for pointwise (PW) rotation. In some configurations, cores (e.g., 3×3 cores) can be implemented diagonally on different columns of the CIM array 802. For example, core 806 can be implemented using CIM cells on columns 810, 812 (e.g., bit lines) and nine rows 814-1, 814-2 to 814-8 and 814-9 (e.g., word lines (WL), collectively referred to as row 814) to implement a 3×3 filter with 2-bit weight parameters. Another core 808 can be implemented on columns 816, 818 and nine rows 820-1 to 820-9 (collectively referred to as row 820) to implement another 3×3 filter. Therefore, cores 806 and 808 are implemented on different rows and columns to facilitate parallel spin operations for DW. That is, starting a row and column for one core (806 or 808) does not affect the row and column for the other. Different start inputs can be provided to each of cores 806 and 808, thus allowing cores 806 and 808 to operate in parallel.

[0105] The input start buffer for each core can be filled (e.g., stored) with the corresponding output channel patch from the previous layer. For example, a row for core 806 (e.g., row 814) can be coupled to start buffers 830-1, 830-2 to 830-8 and 830-9 (collectively referred to as start buffer 830), and a row for core 808 (e.g., row 820) can be coupled to start buffers 832-1 to 832-9 (collectively referred to as start buffer 832).

[0106] The output of core 806 (e.g., at columns 810, 812) can be coupled to analog-to-digital converter (ADC) 840, and the output of core 808 (e.g., at columns 816, 818) can be coupled to ADC 842. For example, each input of ADC 840 can receive the accumulated charge of row 814 from each of columns 810, 812, and each input of ADC 842 can receive the accumulated charge of row 820 from each of columns 816, 818, thereby generating a digital output signal for each of ADC 840, 842. Taking into account that the bits stored in column 812 represent lower importance with corresponding weights than the bits stored in column 810, ADC 840 takes the signals from columns 810, 812 as inputs and generates a digital representation of the signal. Similarly, taking into account that the bits stored in column 818 are of lower importance than the bits stored in column 816, the ADC 842 takes the signals from columns 816 and 818 as inputs and produces a digital representation of the signals.

[0107] Although the ADCs 840 and 842 are implemented to receive signals from two columns for analog-to-digital conversion of a core with a 2-bit weight parameter, the configuration described herein can be implemented as an ADC that is configured to receive signals from any number of columns (e.g., three columns to perform analog-to-digital conversion of a core with a 3-bit weight parameter).

[0108] The outputs of ADCs 840 and 842 can be coupled to a nonlinear operation circuit 850 (and a buffer) to implement nonlinear operations such as rectified linear unit (ReLU) and average pooling (AvePool). Nonlinear operations allow for the establishment of complex mappings between inputs and outputs, thereby enabling the learning and modeling of complex data such as images, video, audio, and nonlinear or high-dimensional datasets. The output of the nonlinear operation circuit 850 can be coupled to an input start buffer 860 for a CIM array 804 configured for PW cyclotron. The output of the CIM array 804 can be coupled to ADC 870, and the output of ADC 870 can be provided to the nonlinear operation circuit 880, as shown. Although a single ADC 870 is illustrated, multiple ADCs can be implemented for different columns of the CIM array 804.

[0109] Although each of cores 806 and 808 includes two columns, allowing 2-bit weights to be stored in each row of the core, cores 806 and 808 can be implemented using any number of suitable columns, such as one column for 1-bit binary weights, or two or more columns for multi-bit weights. For example, each of cores 806 and 808 can be implemented using three columns to store three-bit weight parameters in each row of the core, or using a single column to store one-bit weights in each row of the core. Furthermore, although each of cores 806 and 808 is implemented as nine rows for a 3×3 core for ease of understanding, cores 806 and 808 can be implemented using any number of rows to achieve a suitable core size. Additionally, more than two cores can be implemented using a subset of the cells of the CIM array. For example, CIM array 802 can include one or more other cores, wherein the cores of CIM array 802 are implemented on different rows and columns to facilitate parallel spin operations. For example, core 806 may correspond to core 304A as described in Figure 3A, and core 808 may correspond to core 304B as described in Figure 3A. Another core (not shown in Figure 8) corresponding to core 304C may also be implemented on a different row and column than cores 806 and 808.

[0110] Figure 9 illustrates an example operation 900 of signal processing via the CIM system 800 of Figure 8 according to certain states of this invention. Operation 900 can begin by processing a DW-CNN layer. For example, in block 904, DW spin weights can be loaded into CIM cells of the CIM array (e.g., for cores 806, 808), as described herein. For example, in block 904, DW 3×3 core weights can be grouped and written into the CIM array 802 of Figure 8. That is, 2-bit core weights can be provided to columns 810, 812, and the transmission gate switches of memory cells (e.g., memory cells b01 and b11 shown in Figure 8) can be closed to store the 2-bit core weights in the memory cells. Filter weights can be stored in a similar manner in each row of the CIM cells of each of cores 806, 808.

[0111] Weights that may have been previously stored in memory cells in the same column but different row as the active kernel can be zeroed. For example, logic 0 can be stored in memory cells (not shown) at columns 816, 818 and row 820, and in memory cells at columns 810, 812 and row 814. In some cases, CIM array 802 can be zeroed first before storing weights for kernels 806 and 808.

[0112] In some implementations, the CIM array can be divided into tiles. For example, a tile in the same column as the active core can be configured in a tri-state mode. In tri-state mode, the output of the memory cell in the tile can be configured to have relatively high impedance, thereby effectively eliminating the influence of the cell on the output. As described herein, DW cyclotron cores in different columns and rows can be stacked. Both the DW cyclotron weights and PW cyclotron weights can be updated for each subsequent layer.

[0113] In block 906, (e.g., at start buffers 830, 832) the DW spin start input can be applied to each set of rows of cores 806, 808 during the same cycle to generate DW spin outputs in parallel using two cores.

[0114] In block 908, ADCs 840 and 842 can convert the cyclotron outputs of cores 806 and 808 (e.g., at columns 810, 812 and 816, 818) from the analog domain to the digital domain. Nonlinear (NL) operation can be performed via nonlinear operation circuitry 850 based on the outputs of ADCs 840 and 842 used for DW cyclotron.

[0115] In block 910, the output from the nonlinear operation circuit 850 can be applied to the PW input of the CIM array 804 (e.g., stored in the input start buffer 860) to perform PW spin. In block 912, the ADC 870 can convert the PW spin output from the CIM array 804 from the analog domain to the digital domain. Nonlinear operation can be performed via the nonlinear operation circuit 880 based on the output of the ADC 870 used for PW spin.

[0116] By implementing cores on different rows and columns, spin operations can be performed in parallel, facilitating faster processing time and lower dynamic power compared to conventional implementations. In other words, performing parallel spin operations allows multiple filters to be processed in one cycle, rather than each filter in different cycles, thus saving processing time and reducing dynamic power. In some cases, each core can be repeated multiple times to increase row utilization and reduce ADC range compression, as described in more detail herein. Techniques for reducing power consumption and improving CIM array utilization.

[0117] Figure 10 illustrates a CIM array 1000 that, according to certain states of this case, is divided into tiles (also called sub-banks) to save power. As an example, the CIM array 1000 may have 1024 rows and 256 columns. Individual tiles (e.g., sub-banks) in rows and columns (r×c) can be enabled or disabled. For example, a tile may include 128 rows and 23 columns. As an example, tile array 1002 (e.g., including multiple tiles, such as tile 1004) may be active for DW-CNN rotation, while the remaining tiles may be disabled. In other words, the remaining tiles can be configured in a tri-state mode.

[0118] In some implementations, row and column filler cells can be implemented in the CIM array 1000. Filler circuitry (e.g., buffers or switches) can be used to enable or disable the tiles of the CIM array to save power. Column filler cells can be AND gated logic, and row filler cells can be buffers on write bit-lines (WBLs) and transfer switches on read bit-lines (RBLs). The size and type of transfer switches can be configured based on linearity specifications.

[0119] DW spinners can use relatively small core sizes (3×3, 5×5, ...), and underutilization of the CIM array can affect the output signal-to-noise ratio (SNR) due to range compression (e.g., the output of the neural network is distributed over a small range due to nonlinear initiation). Some aspects of this work address techniques for improving SNR. For example, fine-grained patchwork designs can be used to mitigate the impact on SNR, as described in more detail herein with respect to Figure 11.

[0120] Figure 11 illustrates a CIM array 802 with diagonally stacked cores according to certain configurations of this invention. Various diagonally stacked cores can be implemented in the CIM array 802. For example, the CIM array 802 may include CIM cells for cores 806 and 808 (as discussed with respect to Figure 8), and CIM cells for cores 1108, 1110, 1112, 1114, and 1116 (each implemented in different rows and columns of the CIM array 802). As shown, the CIM array 802 can be divided into tiles, such as tiles 1104 and 1106. Each tile of the CIM array that does not include at least a portion of the cores (e.g., tile 1106) can be deactivated to save power.

[0121] In some configurations, for a CIM bank running a DW spinner (e.g., 16 rows and 32 columns), a relatively small tile size (e.g., selected during wafer design) can be used to improve CIM array utilization and save power. Using a smaller tile size improves the utilization of active CIM cells, which are cells that are not part of a disabled tile.

[0122] As an example, three CIM unit groups can be designed to perform inverse bottleneck operations on neural network architectures. Inverse bottleneck operations generally refer to operations used to expand input features, followed by DW rotation and then PW rotation to reduce the DW output size. The first CIM unit group (CIM1) can be used for bottleneck operations, the second CIM unit group (CIM2) can be used for DW rotation operations, and the third CIM unit group (CIM3) can be used for bottleneck operations. In some cases, CIM2 used for DW rotation can have a finer tile arrangement (e.g., 16 rows for 3×3 cores, or 32 rows for 5×5 cores) to improve CIM array utilization and save power, while CIM1 and CIM3 can have coarse-grained tiles (e.g., 64 or 128 rows) to avoid the impact of filler cells on non-DW rotation operations (e.g., because using smaller tiles for the CIM array results in a larger number of filler cells for the CIM array). In this way, the reusability of the CIM array library can be doubled for both DW and non-DW operations.

[0123] As an example, in the case of coarse-grained tiling (e.g., where each tiling uses 64 rows and 32 columns of a CIM array with 1024 rows), the average (e.g., approximate) CIM utilization can be 13.08% for 3×3 cores and 31.44% for 5×5 cores. In other words, only 13.08% of the active memory cells in the CIM array can be used for 3×3 cores, and 31.44% of the active memory cells in the CIM array can be used for 5×5 cores. On the other hand, in the case of fine-grained tiling using 16 rows and 32 columns per tiling, and in the case of a CIM array with 1024 rows, the average CIM utilization can be 40.46% for 3×3 cores and 47.64% for 5×5 cores. With each tile of a 1024-row CIM array using fine-grained tiles of 32 rows and 32 columns, the average CIM utilization can be 24.18% for a 3×3 core and 47.64% for a 5×5 core. Therefore, fine-grained tiles improve CIM array utilization for filters with smaller core sizes (e.g., for DW recursion). Increased CIM array utilization results in a higher percentage of active memory cells being utilized, thereby reducing power loss caused by unutilized active memory cells.

[0124] In some morphologies, utilization can be improved by selecting a piece size that is closer to the core size. For example, as shown in Figure 11, a piece size of 16 (e.g., as shown for piece 1104) can be used for a core size of 9 (e.g., 9 rows, as shown for core 806). The piece size can be a power of 2 (logarithmic scale) of the core size to improve flexibility in handling different neural network models. In some morphologies, the core can be repeated to improve row utilization and increase ADC SNR, as described in more detail with respect to Figure 12.

[0125] Figure 12 illustrates a CIM array implemented with a repeating core according to certain patterns of this case.

[0126] As shown in the figure, multiple cores can be repeated to form a core group. For example, multiple cores, such as cores 806, 1204 or cores 808, 1208, can be implemented on the same column. The same weighting parameters can be stored in the repeated cores (e.g., cores 806, 1204) of the core group on the same column, and the same start input can be provided to the repeated cores. Therefore, repeated cores can produce the same output signal combined at each column (output), resulting in an increase in dynamic range at the output of the repeated cores. For example, using two repeated cores may result in doubling the dynamic range at the output of the repeated cores provided to the ADC (e.g., ADC 840). Increasing the dynamic range at the core output facilitates analog-to-digital conversion with higher accuracy because it can utilize a wider range of the ADC. In other words, using the entire range of the ADC input allows the ADC's digital output to more accurately identify the ADC's analog input and improves the ADC's SNR.

[0127] In some cases, the number of DW cyclotron channels that can be implemented in a CIM array may be limited by the size of the CIM array. For example, if a 3×3 filter is implemented, 113 channels can be implemented for a CIM array with 1024 rows (e.g., because 113×9 is less than 1024). In other words, the DW core used for DW cyclotron may not be suitable for a CIM array due to row or column limitations associated with the CIM array. Therefore, the input start and DW cyclotron weights can be arranged by a sequencer so that the sum of some DW cyclotron channels can be computed.

[0128] In some cases, the maximum number of cores achievable in a CIM array may be less than the total number of cores for all channels. The maximum number of cores can be achieved in the CIM array. Then, all corresponding channel inputs can be processed to produce partial channel outputs. Subsequently, the array can be loaded with the next batch of cores, and this partial output can be processed until all cores are processed. As another example, the DW cyclotron input batch size can be determined based on subsequent PW layer size information. Cores can be loaded multiple times to process the input batch size. Subsequently, a portion of the DW output can be fed to the next PW cyclotron layer to produce partial bottleneck outputs. Example operation for performing neural network processing in a CIM array.

[0129] Figure 13 is a flowchart illustrating an example operation 1300 for signal processing in a neural network according to certain forms of this invention. Operation 1300 can be performed by a controller (such as the CIM controller 1432 described relative to Figure 14).

[0130] Operation 1300 begins at block 1305, whereby the controller loads a first in-memory computation (CIM) unit set having a first plurality of weight parameters for a first core (e.g., core 806 of Figure 8) via one or more first columns (e.g., 810, 812 of Figure 8) to perform neural network computation (e.g., DW neural network computation). The first CIM unit set has one or more first columns and a first plurality of rows (e.g., row 814 of Figure 8) of a CIM array (e.g., CIM array 802 of Figure 8). At block 1310, the controller loads a second CIM unit set having a second plurality of weight parameters for a second core (e.g., core 808 of Figure 8) via one or more second columns (e.g., columns 816, 818 of Figure 8) to perform neural network computation. The second CIM unit set has one or more second columns and a second plurality of rows (e.g., row 820 of Figure 8) of a CIM array. For example, the first CIM cell set may include a subset of the cells of the CIM array, and the second CIM cell set may include a subset of another cell of the CIM array. In some patterns, one or more first columns may differ from one or more second columns, and the first plurality of rows may differ from the second plurality of rows. In block 1315, the controller may perform neural network computation by applying a first start input to the first plurality of rows and a second start input to the second plurality of rows.

[0131] In some configurations, operation 1300 may also include loading another CIM array (e.g., CIM array 804 of FIG. 8) with a third plurality of weights for the third core to perform point neural network computation. The controller may also generate input signals for the second CIM array (e.g., provided via input start buffer 860 of FIG. 8) based on the output signals from the deep neural network computation.

[0132] In some embodiments, operation 1300 may also include generating a first digital signal by converting the voltages at one or more first columns from the analog domain to the digital domain (e.g., via ADC 840 of FIG8), and generating a second digital signal by converting the voltages at one or more second columns from the analog domain to the digital domain (e.g., via ADC 842 of FIG8). Operation 1300 may also include performing a nonlinear startup operation based on the first and second digital signals (e.g., via nonlinear startup circuit (ACT) 850).

[0133] In some configurations, the core can be repeated to improve CIM array utilization and increase input range compression for the ADC. For example, the controller can also load a third CIM unit with a first plurality of weight parameters for a third core (e.g., core 1204 of FIG. 12) via one or more first columns to perform neural network computation. The third CIM unit can be on one or more first columns and a third plurality of rows of the CIM array. The controller can perform neural network computation by applying at least a first start input (e.g., the same start input provided to the first core) to the third plurality of rows. As described herein, each bit of the weight parameter can be stored via columns of the core. For example, the number of one or more first columns can be associated with the number of one or more bits of each of the first plurality of weight parameters, and the number of one or more second columns can be associated with the number of one or more bits of each of the second plurality of weight parameters. Example processing system for performing phase-selective spin

[0134] Figure 14 illustrates an example electronic device 1400. Electronic device 1400 can be configured to perform the methods described herein, including operation 1300 relative to that described in Figure 13.

[0135] Electronic device 1400 includes a central processing unit (CPU) 1402, which may be a multi-core CPU in some cases. Instructions executed at CPU 1402 may be loaded, for example, from program memory associated with CPU 1402, or may be loaded from memory 1424.

[0136] The electronic device 1400 also includes additional processing blocks customized for specific functions, such as a graphics processing unit (GPU) 1404, a digital signal processor (DSP) 1406, a neural processing unit (NPU) 1408, a multimedia processing block 1410, and a wireless connectivity processing block 1412. In one implementation, the NPU 1408 is implemented in one or more of the CPU 1402, GPU 1404, and / or DSP 1406.

[0137] In some embodiments, the wireless connectivity processing block 1412 may include elements for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and wireless data transmission standards. The wireless connectivity processing block 1412 is also connected to one or more antennas 1414 to facilitate wireless communication.

[0138] Electronic device 1400 may also include one or more sensor processors 1416 associated with any type of sensor, one or more image signal processors (ISP) 1418 associated with any type of image sensor and / or may include a navigation processor 1420 that may include satellite-based positioning system elements (e.g., GPS or GLONASS) and inertial positioning system elements.

[0139] Electronic device 1400 may also include one or more input and / or output devices 1422, such as a screen, a touch-sensitive surface (including a touch-sensitive display), a physical button, a speaker, a microphone, etc. In some embodiments, one or more processors of electronic device 1400 may be based on the ARM instruction set.

[0140] Electronic device 1400 also includes memory 1424, which represents one or more static and / or dynamic storage devices, such as dynamic random access memory, static memory based on flash memory, etc. In this example, memory 1424 includes computer executable elements that can be executed by one or more of the aforementioned processors or CIM controllers 1432 of electronic device 1400. For example, electronic device 1400 may include CIM circuitry 1426, which includes one or more CIM arrays, such as CIM arrays 802 and 804 described herein. CIM circuitry 1426 may be controlled via CIM controller 1432. For example, in some cases, memory 1424 may include code 1424A for loading (e.g., loading CIM cells with weight parameters) and code 1424B for computation (e.g., performing neural network computation via application start input). As shown in the figure, the CIM controller 1432 may include circuitry 1428A for loading (e.g., loading CIM cells with weight parameters) and circuitry 1428B for computation (e.g., performing neural network computation by applying start input). The illustrated elements, as well as other elements not shown, may be configured to perform various states of the methods described herein.

[0141] In some configurations, such as when electronic device 1400 is a server device, various configurations, such as one or more of multimedia element 1410, wireless connection element 1412, antenna 1414, sensor 1416, ISP 1418, or navigation 1420, can be omitted from the configurations illustrated in FIG14. Example Terms

[0142] State 1. An apparatus for signal processing in a neural network, comprising: a first in-memory computing (CIM) unit configured as a first core for neural network computing, the first CIM unit set including one or more first columns and a first plurality of rows of a CIM array; and a second CIM unit set configured as a second core for neural network computing, the second CIM unit set including one or more second columns and a second plurality of rows of a CIM array, wherein: one or more first columns are different from one or more second columns, and the first plurality of rows are different from the second plurality of rows.

[0143] State 2. The apparatus according to State 1, wherein the first set of CIM units includes a subset of the units of the CIM array, and the second set of CIM units includes a subset of another set of the CIM array.

[0144] State 3. The apparatus according to any one of states 1 to 2, wherein the neural network computation includes deep (DW) neural network computation.

[0145] State 4. The apparatus according to State 3 further includes: another CIM array configured as a third core for point-of-war (PW) neural network computation, wherein the input signal to the other CIM array is generated based on the output signal from the CIM array.

[0146] State 5. The apparatus according to any one of states 1 to 4 further includes: a first analog-to-digital converter (ADC) coupled to one or more first columns; and a second ADC coupled to one or more second columns.

[0147] State 6. The apparatus according to State 5 further includes a nonlinear start-up circuit coupled to the outputs of the first ADC and the second ADC.

[0148] State 7. The apparatus according to any one of states 1 to 6 further includes a third CIM unit configured as a third core for neural network computing, the third CIM unit being located on one or more first columns and a third plurality of rows of the CIM array.

[0149] State 8. The apparatus according to State 7, wherein the same weight parameters are configured to be stored in the first CIM unit set and the third CIM unit.

[0150] State 9. The apparatus according to any one of States 1 to 8, wherein: one or more CIM cells in a first CIM cell set on each of the first plurality of rows are configured to store a first weight parameter; and one or more CIM cells in a second CIM cell set on each of the second plurality of rows are configured to store a second weight parameter.

[0151] State 10. The apparatus according to State 9, wherein: the number of one or more first columns is associated with the number of one or more bits of a first weighting parameter; and the number of one or more second columns is associated with the number of one or more bits of a second weighting parameter.

[0152] State 11. A method for signal processing in a neural network, comprising: loading a first in-memory computation (CIM) unit having a first plurality of weight parameters for a first core via one or more first columns to perform neural network computation, the first CIM unit set including one or more first columns and a first plurality of rows of a CIM array; loading a second CIM unit set having a second plurality of weight parameters for a second core via one or more second columns to perform neural network computation, the second CIM unit set including one or more second columns and a second plurality of rows of a CIM array, wherein one or more first columns are different from one or more second columns, and wherein the first plurality of rows are different from the second plurality of rows; and performing neural network computation by applying a first start input to the first plurality of rows and applying a second start input to the second plurality of rows.

[0153] State 12. According to the method of State 11, wherein the first CIM cell set includes a subset of the cells of the CIM array, and the second CIM cell set includes a subset of another cell of the CIM array.

[0154] State 13. The method according to any one of states 11 to 12, wherein the neural network computation includes deep (DW) neural network computation.

[0155] State 14. The method according to State 13 further includes: loading another CIM array having a third plurality of weights for the third core to perform point-wise (PW) neural network computation; and generating an input signal for the other CIM array based on the output signal from the DW neural network computation.

[0156] State 15. The method according to any one of states 11 to 14 further includes: generating a first digital signal by converting the voltages at one or more first columns from an analog domain to a digital domain; and generating a second digital signal by converting the voltages at one or more second columns from an analog domain to a digital domain.

[0157] State 16. The method according to State 15 further includes performing a nonlinear start operation based on the first digital signal and the second digital signal.

[0158] State 17. The method according to any one of states 11 to 16, further comprising: loading a third CIM unit having a first plurality of weight parameters for a third core via one or more first columns to perform neural network computation, the third CIM unit being located on one or more first columns and a third plurality of rows in memory, wherein performing neural network computation further comprises applying a first initiation input to the third plurality of rows.

[0159] State 18. The method according to any one of states 11 to 17, wherein: the number of one or more first columns is associated with the number of one or more bits of each of the first plurality of weight parameters; and the number of one or more second columns is associated with the number of one or more bits of each of the second plurality of weight parameters.

[0160] State 19. A non-transitory computer-readable medium, including instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a signal processing method in a neural network, the method comprising: loading a first in-memory computation (CIM) unit having a first plurality of weight parameters for a first core via one or more first columns to perform neural network computation, the first CIM unit set including one or more first columns and a first plurality of rows of a CIM array; loading a second CIM unit set having a second plurality of weight parameters for a second core via one or more second columns to perform neural network computation, the second CIM unit set including one or more second columns and a second plurality of rows of a CIM array, wherein one or more first columns are different from one or more second columns, and wherein the first plurality of rows are different from the second plurality of rows; and performing neural network computation by applying a first start input to the first plurality of rows and applying a second start input to the second plurality of rows. Additional considerations

[0161] The foregoing description is provided to enable any person skilled in the art to practice the various forms described herein. The examples discussed herein do not limit the scope, applicability, or forms set forth in the claims. Various modifications to such forms will be apparent to those skilled in the art, and the general principles defined herein can be applied to other forms. For example, the function and arrangement of the discussed elements can be changed without departing from the scope of this application. Various procedures or elements can be appropriately omitted, substituted, or added to the various examples. For example, the described methods can be performed in a different order than that described, and various steps can be added, omitted, or combined. Furthermore, features described relative to some examples can be combined in some other examples. For example, any number of forms set forth herein can be used to implement an apparatus or method of practice. Moreover, the scope of this application is intended to cover an apparatus or method of practice that uses a structure, function, or structure and function other than or different from the various forms of this application set forth herein. It should be understood that any form of this application disclosed herein can be embodied by one or more elements of the claims.

[0162] As used herein, the word “exemplary” means “used as an example, instance or illustration”. Any state described herein as “exemplary” is not necessarily to be construed as being more preferred or advantageous than other states.

[0163] As used herein, the phrase “at least one of the items” refers to any combination of those items, including a single member. As an example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0164] As used herein, the term "decision" encompasses a wide variety of actions. For example, "decision" can include operation, calculation, processing, deduction, investigation, examination (e.g., examining in a table, database, or other data structure), ascertainment, etc. Furthermore, "decision" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. In addition, "decision" can include parsing, selecting, choosing, building, etc.

[0165] The methods disclosed herein include one or more steps or actions for implementing the method. The method steps and / or actions may be interchanged without departing from the scope of the claim. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claim. Furthermore, the various operations of the methods described above can be performed by any suitable component capable of performing the corresponding function. Such components may include various hardware and / or software elements and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, in the cases of operations illustrated in the figures, such operations may have corresponding paired components with similar numbering plus functional elements.

[0166] The following claims are not intended to be limited to the forms shown herein, but to encompass the full scope consistent with the language of the claims. In the claims, unless otherwise stated, elements referred to in the singular do not mean "one and only one," but rather "one or more." Unless otherwise stated, the term "some" means one or more. Elements of the claims are not to be interpreted in accordance with the provisions of patent law unless the element is explicitly described using the phrase "component for..." or, in the case of a method claim, using the phrase "for step...". All structural and functional equivalents of the elements throughout the various forms described herein, known or hereafter known to a person generally skilled in the art, are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is expressly stated in the claims. [Simplified Explanation of the Diagram]

[0013] The accompanying drawings illustrate some of the states among one or more states, and therefore should not be considered as a limitation on the scope of this case.

[0014] Figures 1A-1D illustrate examples of various types of neural networks.

[0015] Figure 2 illustrates an example of a conventional rotary operation.

[0016] Figures 3A and 3B illustrate examples of depth-separable spin operations.

[0017] Figure 4 illustrates an example in-memory computation (CIM) array configured to perform machine learning model computations.

[0018] Figures 5A-5B illustrate additional details of an exemplary bit cell that may represent the bit cell in Figure 4.

[0019] Figure 6 illustrates an example timing diagram of various signals during the operation of a compute-in-memory (CIM) array.

[0020] Figure 7 illustrates an exemplary swirl layer architecture implemented by a computation in memory (CIM) array.

[0021] Figure 8 illustrates a CIM architecture including multiple CIM arrays according to certain aspects of this case.

[0022] Figure 9 illustrates example operations for signal transmission processing via the CIM architecture according to certain states of this case.

[0023] Figure 10 illustrates a CIM array that is divided into sub-libraries according to certain patterns in this case to save power.

[0024] Figure 11 illustrates a CIM array with diagonally stacked cores according to certain patterns of this case.

[0025] Figure 12 illustrates a CIM array implemented with a repeating core according to certain states of this case.

[0026] Figure 13 is a flowchart illustrating example operations of signal processing in neural networks according to certain states of this case.

[0027] Figure 14 illustrates an example electronic device configured to perform operations for signal processing in a neural network, according to certain states of this case.

[0028] For ease of understanding, the same element symbols are used to represent common elements in the figures where possible. It is anticipated that elements and features of one pattern can be beneficially incorporated into other patterns without further description. [Biomaterial Storage]

[0168] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. An apparatus for performing machine learning tasks, comprising: A first in-memory computation (CIM) unit set configured as a first core for neural network computation, the first CIM unit set including one or more first columns and a first plurality of rows of a first CIM array; and a second CIM unit set configured as a second core for neural network computation, the second CIM unit set including one or more second columns and a second plurality of rows of the first CIM array, wherein: the one or more first columns are different from the one or more second columns, the first plurality of rows are different from the second plurality of rows, all of the first core and all of the second core are stored in different rows and columns, the first core and the second core are part of the same filter; a first analog-to-digital converter (ADC) coupled to the one or more first columns configured to store the first core; a second ADC coupled to the one or more second columns configured to store the second core, the first ADC and the second ADC being separate to perform at least partially parallel analog-to-digital conversion on the one or more first columns and the one or more second columns respectively; A nonlinear startup circuit is coupled to the outputs of the first ADC and the second ADC, wherein the neural network computation includes a deep (DW) neural network computation based on the same filter; and a second CIM array is configured as a third core for point-of-use (PW) neural network computation, the second CIM array for the PW neural network computation being separate from the first CIM array for the DW neural network computation, wherein the input signal to the second CIM array is generated based on the output signal from the first CIM array.

2. The apparatus according to claim 1, wherein the first CIM unit set includes a subset of the units of the first CIM array, and wherein the second CIM unit set includes another subset of the units of the first CIM array.

3. The apparatus according to claim 1 further includes a third set of CIM units configured as a fourth core for neural network computation, the third set of CIM units being located on one or more first columns and a third plurality of rows of the first CIM array.

4. The apparatus according to claim 3, wherein the same weight parameters are configured to be stored in the first CIM cell set and the third CIM cell set.

5. The apparatus according to claim 1, wherein: One or more CIM cells in the first CIM cell set on each of the first plurality of rows are configured to store a first weight parameter; and one or more CIM cells in the second CIM cell set on each of the second plurality of rows are configured to store a second weight parameter.

6. The apparatus according to claim 5, wherein: The quantity of one or more first columns is associated with the quantity of one or more bits of the first weight parameter; and the quantity of one or more second columns is associated with the quantity of one or more bits of the second weight parameter.

7. A method for performing a machine learning task, comprising: A neural network computation is performed by loading a first in-memory computation (CIM) unit set having a first plurality of weight parameters for a first core via one or more first columns, the first CIM unit set comprising one or more first columns and a first plurality of rows of a first CIM array; the neural network computation is performed by loading a second CIM unit set having a second plurality of weight parameters for a second core via one or more second columns, the second CIM unit set comprising one or more second columns and a second plurality of rows of the first CIM array, wherein the one or more first columns are different from the one or more second columns, and wherein the first plurality of rows are different from the second plurality of rows, wherein all of the first core and all of the second core are stored in different rows and columns, the first core and the second core are part of the same filter; and the neural network computation is performed by applying a first start input to the first plurality of rows and applying a second start input to the second plurality of rows. A first digital signal is generated by converting a voltage at one or more first columns storing the first core from an analog domain to a digital domain via a first analog-to-digital converter (ADC); a second digital signal is generated by converting a voltage at one or more second columns storing the second core from the analog domain to the digital domain via a second ADC, wherein the first ADC and the second ADC are separate to generate the first digital signal and the second digital signal in at least partially parallel operation; a nonlinear startup operation is performed based on the first digital signal and the second digital signal, wherein the neural network computation includes a deep (DW) neural network computation based on the same filter; a second CIM array having a third plurality of weights for a third core is loaded to perform a point-of-use (PW) neural network computation, wherein the second CIM array used for the PW neural network computation is separate from the first CIM array used for the DW neural network computation; and an input signal to the second CIM array is generated based on the output signal from the DW neural network computation.

8. The method of claim 7, wherein the first CIM cell set includes a subset of cells of the first CIM array, and wherein the second CIM cell set includes another subset of cells of the first CIM array.

9. The method according to claim 7 also includes: The neural network computation is performed by loading a third set of CIM cells with the first plurality of weight parameters for a fourth core via one or more first columns, the third set of CIM cells being located on one or more first columns and a third plurality of rows of the memory, wherein performing the neural network computation further includes applying the first start input to the third plurality of rows.

10. According to the method of request item 7, wherein: The quantity of one or more first columns is associated with the quantity of one or more bits of each of the first plurality of weight parameters; and the quantity of one or more second columns is associated with the quantity of one or more bits of each of the second plurality of weight parameters.

11. A non-transitory computer-readable medium, comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method comprising: A neural network computation is performed by loading a first in-memory computation (CIM) unit set having a first plurality of weight parameters for a first core via one or more first columns, the first CIM unit set comprising one or more first columns and a first plurality of rows of a first CIM array; the neural network computation is performed by loading a second CIM unit set having a second plurality of weight parameters for a second core via one or more second columns, the second CIM unit set comprising one or more second columns and a second plurality of rows of the first CIM array, wherein the one or more first columns are different from the one or more second columns, and wherein the first plurality of rows are different from the second plurality of rows, wherein all of the first core and all of the second core are stored in different rows and columns, the first core and the second core are part of the same filter; and the neural network computation is performed by applying a first start input to the first plurality of rows and applying a second start input to the second plurality of rows. A first digital signal is generated by converting a voltage at one or more first columns of the first core from an analog domain to a digital domain via a first analog-to-digital converter (ADC); a second digital signal is generated by converting a voltage at one or more second columns of the second core from the analog domain to the digital domain via a second ADC, the first ADC and the second ADC being separate to generate the first digital signal and the second digital signal in at least partially parallel operation; a nonlinear activation operation is performed based on the first digital signal and the second digital signal, wherein the neural network computation includes a deep (DW) neural network computation based on the same filter; A second CIM array having a third plurality of weights for a third core is loaded to perform point-of-use (PW) neural network computation, wherein the second CIM array for the PW neural network computation is separate from the first CIM array for the DW neural network computation; and an input signal to the second CIM array is generated based on the output signal from the DW neural network computation.

Images