Apparatus, method, and computer-readable medium for computation in memory architecture for phased depth-wise convolutional

Abstract

Some forms provide means for signal processing in neural networks. In general, the means includes: a first in-memory computing (CIM) unit configured for neural network computation in a first core, the first CIM unit set including one or more first columns and a first plurality of rows of a CIM array. The means also includes: a second CIM unit set configured for neural network computation in a second core, the second CIM unit set including one or more first columns and a second plurality of rows of a CIM array. The first plurality of rows may differ from the second plurality of rows.

Description

Technical Field

[0001] This application claims priority to U.S. Application No. 17 / 361,807, filed on June 29, 2021, which has been assigned to the assignee of this application and is incorporated herein by reference in its entirety.

[0002] The variants of this disclosure relate to performing machine learning tasks, and more specifically, the variants of this disclosure relate to intra-memory computing architectures and data flows for performing depth-separable spinners in memory. Prior Technology

[0003] Machine learning is typically the process of generating trained models (e.g., artificial neural networks, trees, or other structures) that represent a generalized fit to a previously known training dataset. Applying the trained model to new data produces inferences, which can then be used to gain insights into the new data. In some cases, applying a model to new data is described as "running inferences" on the new data.

[0004] With the surge in the use of machine learning for various machine learning (or artificial intelligence) tasks, there has been a demand for more efficient processing of machine learning model data. In some cases, dedicated hardware, such as machine learning accelerators, can be used to enhance the processing system's ability to process machine learning model data. However, such hardware requires space and power and is not always available on processing devices. For example, "edge processing" devices (such as mobile devices, always-on devices, Internet of Things (IoT) devices, etc.) must balance processing power with power and packaging constraints. Furthermore, accelerators may need to move data across a common data bus, which can result in significant power consumption and introduce latency into other processes on the shared data bus. Therefore, alternative forms of processing systems are being considered for processing machine learning model data.

[0005] Memory devices are another example of a processing system that can be used to process machine learning model data via a process known as computation in memory (CIM). Unfortunately, without additional hardware components such as digital multiplication and accumulation circuitry (DMAC) and associated peripherals, CIM processes may be unable to process complex model architectures, such as depthwise separable spiral neural networks. These additional hardware components consume extra space, power, and complexity in their implementation, which often diminishes the advantages of using memory devices as additional computational resources. Even when auxiliary states of the processing system have DMAC (which can be used to perform processing that cannot be directly executed in memory), moving data into and out of these auxiliary states requires time and power, thus reducing the benefits of CIM processes.

[0006] Therefore, there is a need for systems and methods to perform computations within the memory of a wider range of machine learning model architectures, such as depthwise separable spiral neural networks. Summary of the Invention

[0007] Some embodiments provide an apparatus for signal processing in a neural network. In general, the apparatus includes: a first in-memory computation (CIM) unit configured for depth-by-depth (DW) neural network computation, the first CIM unit set including one or more first columns and a first plurality of rows of a CIM array; and a second CIM unit set configured for a second core for the neural network computation, the second CIM unit set including the one or more first columns and the second plurality of rows of the CIM array, wherein the first plurality of rows are different from the second plurality of rows. The apparatus may also include: a third CIM unit set of the CIM array configured for a third core for point-by-point (PW) neural network computation.

[0008] Some approaches provide a method for signal processing in neural networks. In general, the method includes: performing a plurality of DW spin operations via a plurality of cores implemented using a plurality of CIM unit groups on one or more first columns of a CIM array; and generating input signals for PW spin operations based on the outputs from the plurality of DW spin operations. The method also includes: performing PW spin operations based on the input signals, the PW spin operations being performed via cores implemented using CIM unit groups on one or more second columns of the CIM array.

[0009] Some forms provide a non-transitory computer-readable medium with instructions that, when executed by one or more processors of a processing system, cause the processing system to perform signal processing in a neural network. In summary, the method includes: performing a plurality of DW spin operations via a plurality of cores implemented using a plurality of CIM cell groups on one or more first columns of a CIM array; and generating input signals for a PW spin operation based on the outputs from the plurality of DW spin operations. The method also includes: performing a PW spin operation based on the input signals, the PW spin operation being performed via cores implemented using CIM cell groups on one or more second columns of the CIM array.

[0010] Other forms provide: a processing system configured to perform the methods described above and herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the methods described above and herein; a computer program product embodied on a computer-readable storage medium, including code for performing the methods described above and herein; and a processing system including components for performing the methods described above and herein.

[0011] The following description and related figures illustrate certain illustrative features of one or more states. Simple Explanation of the Diagram

[0012] The accompanying drawings illustrate some of the states of one or more states, and should therefore not be considered as limiting the scope of this disclosure.

[0013] Figures 1A to 1D illustrate examples of various types of neural networks.

[0014] Figure 2 illustrates an example of a familiar spin operation.

[0015] Figures 3A and 3B illustrate examples of depth-separable spin operations.

[0016] Figure 4 shows an example in-memory computation (CIM) array configured to perform machine learning model computations.

[0017] Figures 5A and 5B illustrate additional details of an exemplary bit cell that may represent the bit cell in Figure 4.

[0018] Figure 6 illustrates example timing diagrams of various signals during CIM array operation.

[0019] Figure 7 illustrates an exemplary swirl layer architecture implemented by a CIM array.

[0020] Figures 8A and 8B illustrate certain forms of CIM architectures including CIM arrays according to this disclosure.

[0021] Figure 9 illustrates example operations of signal processing for the CIM architecture of Figure 8B in accordance with certain forms of this disclosure.

[0022] Figure 10 illustrates a CIM array that, according to this disclosure, is divided into sub-storages to improve processing efficiency.

[0023] Figure 11 illustrates a CIM array implemented using a repeating core in accordance with certain patterns of this disclosure to improve processing accuracy.

[0024] Figure 12 is a flowchart illustrating some of the example operations for signal processing in neural networks according to the present disclosure.

[0025] Figure 13 illustrates an example electronic device configured to perform signal processing operations in a neural network, according to certain states of this disclosure.

[0026] To aid understanding, where possible, the same element symbols have been used to designate common elements for the purposes of the accompanying drawings. It is anticipated that elements and features of one pattern can be beneficially incorporated into other patterns without further description. Implementation

[0027] Various embodiments of this disclosure provide apparatus, methods, processing systems, and computer-readable media for performing in-memory computation (CIM) of machine learning models, including depth-wise (DW) separable spiral neural networks. Some embodiments provide a two-stage spiral technique implemented on a CIM array. For example, one of the two stages may include a DW spiral operation using a core implemented on the CIM array, while the other of the two stages may include a point-wise (PW) spiral operation using a core implemented on the CIM array.

[0028] For example, some patterns involve CIM cells configured for different cores to be used in DW spin operations, where the cores are implemented on different rows and the same columns of the CIM array. As described herein, a staged approach can be used to process the cores. The output of the cell implementing the core can be coupled to an analog-to-digital converter (ADC). The results of the DW calculations can be fed into a nonlinear startup circuit for further processing and fed back into the same CIM array for point-by-point calculations, as described in more detail herein. The patterns described herein provide flexibility in configuring any CIM array for DW spin operations as needed, while increasing the number of cores that can be implemented on the CIM array compared to conventional implementations, as described in more detail herein.

[0029] CIM-based machine learning (ML) / artificial intelligence (AI) task accelerators can be used for a wide variety of tasks, including image and audio processing. Furthermore, CIM can be based on various types of memory architectures (such as Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM) (e.g., based on SRAM cells as shown in Figure 5), Magnetoresistive Random Access Memory (MRAM), and Resistive Random Access Memory (ReRAM)) and can be attached to various types of processing units, including Central Processing Units (CPUs), Digital Signal Processors (DSPs), Graphics Processing Units (GPUs), Field-Programmable Gate Arrays (FPGAs), AI accelerators, and more. Typically, CIM can beneficially reduce the "memory wall" problem, where moving data into and out of memory consumes more power than the computation itself. Therefore, significant power savings can be achieved by performing computations in memory. This is particularly useful for various types of electronic devices, such as low-power edge processing devices, mobile devices, etc.

[0030] For example, a mobile device may include a memory device configured for storing data and performing computational operations within memory. The mobile device may be configured to perform ML / AI operations based on data generated by the mobile device, such as image data generated by the mobile device's camera sensor. Therefore, the mobile device's memory controller unit (MCU) can load weights from another onboard memory (e.g., flash memory or RAM) into the CIM array of the memory device and allocate input feature buffers and output (e.g., start) buffers. The processing device can then begin processing the image data by loading layers, for example, into the input buffers and processing those layers using the weights loaded into the CIM array. This processing can be repeated for each layer of the image data, and the outputs (e.g., start) can be stored in the output buffers and subsequently used by the mobile device for ML / AI tasks, such as face recognition. A brief background on neural networks, deep neural networks, and deep learning.

[0031] Neural networks are organized into layers of interconnected nodes. Typically, nodes (or neurons) are where computations are performed. For example, a node can combine input data with a set of weights (or coefficients) that amplify or suppress the input data. Thus, the amplification or suppression of an input signal can be viewed as an assignment of the relative importance of various inputs to the task the network is attempting to learn. Typically, the products of the input weights are summed (or accumulated), and this sum is then passed via the node's activation function to determine whether the signal should proceed further in the network and to what extent.

[0032] In its most basic implementation, a neural network can have an input layer, hidden layers, and an output layer. "Deep" neural networks typically have more than one hidden layer.

[0033] Deep learning is a method for training deep neural networks. Typically, deep learning maps the network's inputs to its outputs, and is therefore sometimes called a "universal approximator" because it can learn an unknown function f(x) = y that approximates any input x and any output y. In other words, deep learning finds the correct f to transform x into y.

[0034] More specifically, deep learning trains nodes in each layer based on different feature sets derived from the output of the previous layer. Therefore, with each successive layer of a deep neural network, the features become increasingly complex. Deep learning is powerful because it can progressively extract higher-level features from input data and perform complex tasks (such as object recognition) by learning to represent the input at progressively higher levels of abstraction in each layer, thus building useful feature representations of the input data.

[0035] For example, when using visual data for presentation, the first layer of a deep neural network can learn to recognize relatively simple features in the input data, such as edges. In another example, when using auditory data for presentation, the first layer of a deep neural network can learn to recognize the spectral power of specific frequencies in the input data. Subsequently, the second layer of the deep neural network can learn to recognize combinations of features based on the output of the first layer, such as simple shapes for visual data or sound combinations for auditory data. Higher layers can then learn to recognize complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases. Therefore, deep learning architectures can perform particularly well when applied to problems with a natural hierarchical structure. Layer connectivity in neural networks

[0036] Neural networks (such as deep neural networks) can be designed with various connectivity patterns between layers.

[0037] Figure 1A illustrates an example of a fully connected neural network 102. In the fully connected neural network 102, nodes in the first layer transmit their outputs to each node in the second layer, such that each node in the second layer receives inputs from each node in the first layer.

[0038] Figure 1B illustrates an example of a locally connected neural network 104. In the locally connected neural network 104, nodes in the first layer can connect to a finite number of nodes in the second layer. More typically, the locally connected layers of the locally connected neural network 104 can be configured such that each node in a layer will have the same or similar connectivity pattern, but with connectivity strengths (or weights) that can have different values ​​(e.g., 110, 112, 114, and 116). The connectivity patterns of locally connected networks can create spatially different receptive fields in higher layers because higher-layer nodes in a given region can receive inputs of properties that have been trained to a restricted portion of the network's total input.

[0039] One type of locally connected neural network is the spiral neural network. Figure 1C shows an example of a spiral neural network 106. The spiral neural network 106 can be configured such that the connection strength associated with the input for each node in the second layer is shared (e.g., 108). Spiral neural networks are well-suited for problems in which the spatial location of the input is meaningful.

[0040] One type of spiral neural network is the deep spiral network (DCN). A deep spiral network is a network with multiple spiral layers, which can be further configured with, for example, pooling and regularization layers.

[0041] Figure 1D illustrates an example of DCN 100, which is designed to recognize visual features in image 126 generated by image capturing device 130. For example, if image capturing device 130 is a camera mounted in a vehicle, DCN 100 can be trained using various supervised learning techniques to recognize traffic signs and even numbers on traffic signs. DCN 100 can also be trained for other tasks, such as recognizing lane markings or traffic lights. These are just some example tasks, and many other tasks are possible.

[0042] In this example, DCN 100 includes a feature extraction section and a classification section. After receiving image 126, cyclotron layer 132 applies a cyclotron core (e.g., as illustrated and described in Figure 2) to image 126 to generate a first feature map set (or intermediate start) 118. Typically, a "core" or "filter" comprises a multidimensional array of weights designed to emphasize different modes of the input data channels. In various examples, "core" and "filter" can be used interchangeably to represent the set of weights applied in the cyclotron neural network.

[0043] Subsequently, the first feature set 118 can be resampled by a pooling layer (e.g., a max pooling layer, not shown) to generate a second feature set 120. The pooling layer can reduce the size of the first feature set 118 while retaining most of the information, thereby improving model performance. For example, the second feature set 120 can be downsampled from 28x28 to 14x14 by the pooling layer.

[0044] This process can be repeated through multiple layers. In other words, the second feature set 120 can be further spun through one or more subsequent spun layers (not shown) to generate one or more subsequent feature sets (not shown).

[0045] In the example of Figure 1D, the second feature set 120 is provided to the fully connected layer 124, which in turn produces an output feature vector 128. Each feature in the output feature vector 128 may include a number corresponding to a possible feature of the image 126 (e.g., "sign", "60", and "100"). In some cases, a softmax function (not shown) can convert the numbers in the output feature vector 128 into probabilities. In this case, the output 122 of the DCN 100 is the probability that the image 126 includes one or more features.

[0046] Before training DCN 100, the output 122 generated by DCN 100 may be incorrect. Therefore, the error between output 122 and the prior known target output can be calculated. For example, here the target output is an indication that image 126 includes a "sign" and the number "60". Using the known target output, the weights of DCN 100 can then be adjusted via training so that subsequent outputs 122 of DCN 100 reach the target output.

[0047] To adjust the weights of the DCN 100, the learning algorithm can compute gradient vectors for the weights. The gradient indicates how much the error will increase or decrease if the weights are adjusted in a particular way. The weights can then be adjusted to reduce the error. This method of adjusting weights can be called "backpropagation" because it involves "passing backward" through the layers of the DCN 100.

[0048] In practice, the error gradient of the weights can be calculated using a small number of instances, making the calculated gradient approximate the true error gradient. This approximation method is called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system has stopped decreasing or until the error rate has reached the target level.

[0049] After learning, new images can be presented to the DCN 100, and the DCN 100 can generate inferences (such as classification) or probabilities of various features in the new images. Rotation technology for rotational neural networks

[0050] Rotation is commonly used to extract useful features from an input dataset. For example, in a rotation neural network such as the one described above, rotation enables the extraction of different features using cores and / or filters whose weights are automatically learned during training. The extracted features are then combined for inference.

[0051] A priming function can be applied before and / or after each layer of a spiral neural network. The priming function is typically a mathematical function (e.g., an equation) that determines the output of a node in the neural network. Therefore, the priming function determines whether a node should pass information based on whether its input is relevant to the model's prediction. In one instance, in y = conv(x) (i.e., a spiral of y = x), both x and y can generally be considered as the "priming". However, for a specific spiral operation, x can also be called the "pre-priming" or "input priming" because it exists before that specific spiral, and y can be called the output priming or feature map.

[0052] Figure 2 illustrates an example of conventional spiral, where a 12-particle x 12-particle x 3-channel input image is spiraled using a 5x5x3 spiral core 204 and a stride (or stride size) of 1. The resulting feature map 206 is an 8-particle x 8-particle x 1-channel image. As can be seen in this example, conventional spiral can change the dimensions of the input data (here, from 12x12 to 8x8 particles) (compared to the output data), including the channel dimension (here, from 3 to 1 channel).

[0053] One way to reduce computational overhead (e.g., measured in floating-point operations per second (FLOPs)) and the number of parameters associated with neural networks that include spiral layers is to decompose the spiral layers. For example, spatially separable spirals, such as those illustrated in Figure 2, can be decomposed into two components: (1) depthwise spirals, where each spatial channel is spiraled independently by depthwise spirals (e.g., spatial fusion); and (2) pointwise spirals, where all spatial channels are linearly combined (e.g., channel fusion). Examples of depthwise separable spirals are illustrated in Figures 3A and 3B. Typically, during spatial fusion, the network learns features from the spatial plane, and during channel fusion, the network learns relationships between these features across channels.

[0054] In one example, separable depthwise spin can be implemented using a 3x3 core for spatial fusion and a 1x1 core for channel fusion. Specifically, channel fusion can use a 1x1xd core that iteratively passes through each single point in the input image at depth d, where the core depth d typically matches the number of channels in the input image. Channel fusion via pointwise spin is useful for dimensionality reduction for efficient computation. Applying a 1x1xd core and adding a startup layer after the core can provide increased network depth, which can improve its performance.

[0055] Figures 3A and 3B illustrate examples of depth-separable spin operations.

[0056] Specifically, in Figure 3A, a 12-particle x 12-particle x 3-channel input image 302 is cyclotronized using filters comprising three separate cores 304A-C, each having a 5x5x1 dimension, to produce an 8-particle x 8-particle x 3-channel feature map 306, where each channel is generated by a single core among 304A-C.

[0057] Subsequently, feature map 306 is further spiraled using a pointwise spiral operation, where core 308 (e.g., the core) with dimensions 1x1x3 produces feature map 310 with 8 primitives x 8 primitives x 1 channel. As illustrated in this example, feature map 310 has a reduced dimension (1 channel compared to 3 channels), which allows for more efficient computation using feature map 310. In some variations of this disclosure, cores 304A-C and core 308 can be implemented using the same computation in memory (CIM) array, as described in more detail herein.

[0058] Although the results of depth-separable spinners in Figures 3A and 3B are essentially similar to those of the conventional spinners in Figure 2, the computational cost is significantly reduced, and thus depth-separable spinners provide a significant efficiency gain where network design allows.

[0059] Although not illustrated in Figure 3B, multiple (e.g., m) pointwise cyclotron cores 308 (e.g., individual components of a filter) can be used to increase the channel dimension of the cyclotron output. Thus, for example, m = 256 1x1x3 cores 308 can be generated, each outputting an 8-primitive x 8-primitive x 1-channel feature map (e.g., 310), and these feature maps can be stacked to obtain a resulting 8-primitive x 8-primitive x 256-channel feature map. This increase in channel dimension provides more parameters for training, which can improve the ability of the cyclotron neural network (e.g., in the input image 302) to recognize features. Example of in-memory computing (CIM) architecture

[0060] Figure 4 illustrates an exemplary in-memory computing (CIM) array 400 configured to perform machine learning model computations according to various embodiments of this disclosure. In this example, the CIM array 400 is configured to simulate MAC operations using hybrid analog / digital operations for artificial neural networks. Therefore, as used herein, the terms multiplication and addition may refer to this type of simulated operation. The CIM array 100 can be used to implement various embodiments of the processing techniques described herein.

[0061] In the illustrated configuration, the CIM array 400 includes precharge word lines (PCWL) 425a, 425b, and 425c (collectively referred to as 425), read word lines (RWL) 427a, 427b, and 427c (collectively referred to as 427), analog-to-digital converters (ADC) 410a, 410b, and 410c (collectively referred to as 410), a digital processing unit 413, bit lines 418a, 418b, and 418c (collectively referred to as 418), PMOS transistors 411a-411i (collectively referred to as 411), NMOS transistors 413a-413i (collectively referred to as 413), and capacitors 423a-423i (collectively referred to as 423).

[0062] Weights associated with neural network layers can be stored in SRAM cells of the CIM array 400. In this example, binary weights are shown in SRAM bit cells 405a-405i of the CIM array 400. Input initiation (e.g., input values ​​of an input vector) is provided on the PCWL 425a-c.

[0063] Multiplication occurs in each bit cell 405a-405i of the CIM array 400, associated with a bit line, and the sum of the results of multiplication in all bit cells appears on the same bit line for a column. The multiplication in each bit cell 405a-405i has an equivalent operational form to the AND operation of the corresponding enable and weight, where the result is stored as a charge on the corresponding capacitor 423. For example, a product of 1 and therefore a charge on capacitor 423 is only produced when enable is 1 (here, PCWL is zero for enable 1 because PMOS is used) and weight is 1.

[0064] For example, during the accumulation phase, RWL 427 is switched high, allowing any charge on capacitor 423 (based on the corresponding bit cell (weight) and PCWL (start-up) value) to be accumulated on the corresponding bit line 418. The voltage value of the accumulated charge is then converted into a digital value by ADC 410 (where, for example, the output value could be a binary value indicating whether the total charge is greater than a reference voltage). This digital value (output) can then be provided as input to another state of the machine learning model, such as a later layer.

[0065] When the activation values ​​on the precharge word lines (PCWL) 425a, 425b, and 425c are, for example, 1, 0, and 1, the sums of bit lines 418a-c correspond to 0+0+1=1, 1+0+0=1, and 1+0+1=2, respectively. The outputs of ADCs 410a, 410b, and 410c are passed to the digital processing unit 413 for further processing. For example, if the CIM 100 is processing multi-bit weight values, the digital outputs of ADC 110 can be summed to produce the final output.

[0066] An exemplary 3x3 CIM circuit 400 can be used, for example, to perform efficient three-channel cyclotron for a three-element core (or filter), where the weight of each core corresponds to an element in each of the three columns, such that for a given three-element receptive field (or input data block), the output for each of the three channels is computed in parallel.

[0067] It is worth noting that although Figure 4 depicts an example of a CIM using SRAM cells, other memory types can also be used. For example, Dynamic Random Access Memory (DRAM), Magnetoresistive Random Access Memory (MRAM), and Resistive Random Access Memory (ReRAM or RRAM) can also be used in other embodiments.

[0068] Figure 5 illustrates additional details of the exemplary bit cell 500.

[0069] The states in Figure 5A may be exemplary or otherwise related to the states in Figure 4. Specifically, bit line 521 is similar to bit line 418a, capacitor 523 is similar to capacitor 423 in Figure 4, read word line 527 is similar to read word line 427a in Figure 4, and precharge word line 525 is similar to precharge word line 425a in Figure 4, PMOS transistor 511 is similar to PMOS transistor 411a in Figure 1, and NMOS transistor 513 is similar to NMOS transistor 413 in Figure 1.

[0070] Bit cell 500 includes a static random access memory (SRAM) cell 501 (which may represent SRAM bit cell 405a of FIG. 4), a transistor 511 (e.g., a PMOS transistor) and a transistor 513 (e.g., an NMOS transistor), and a capacitor 523 coupled to ground. Although a PMOS transistor is used for transistor 511, other transistors (e.g., NMOS transistors) can be used instead of the PMOS transistor, with corresponding adjustments (e.g., inversion) made to their respective control signals. This also applies to other transistors described herein. According to various embodiments of this disclosure, additional transistors 511 and 513 are included to implement a computational array within the memory. In one embodiment, SRAM cell 501 is a conventional six-transistor (6T) SRAM cell.

[0071] The programming of the weights in the bit cells can be executed once for multiple starts. For example, during operation, SRAM cell 501 receives only one bit of information at nodes 517 and 519 via write word line (WWL) 516. For example, during a write operation (when WWL 216 is high), if write bit line (WBL) 229 is high (e.g., "1"), then node 217 is set high and node 219 is set low (e.g., "0"); or, if WBL 229 is low, then node 217 is set low and node 219 is set high. Conversely, during a write operation (when WWL 216 is high), if write bit bar line (WBBL) 231 is high, then node 217 is set low and node 219 is set high; or, if WBBL 229 is low, then node 217 is set high and node 219 is set low.

[0072] Following the programming of the weights can be an input initiation and multiplication step to charge the capacitor according to the corresponding product. For example, transistor 511 is initiated by an initiation signal (PCWL signal) via the precharge word line (PCWL) 525 of the computational array in memory to perform the multiplication step. Subsequently, transistor 513 is initiated by a signal via another word line (e.g., read word line (RWL) 527) of the computational array in memory to accumulate the multiplication value from bit cell 500 with other bit cells of the array, as described above with respect to Figure 4.

[0073] If node 517 is "0" (e.g., when the stored weight value is "0"), then if the low PCWL indicates a start-up "1" at the gate of transistor 511, capacitor 523 will not be charged. Therefore, no charge is supplied to bit line 521. However, if node 517 corresponding to the weight value is "1" and PCWL is set low (e.g., when the start-up input is high) (which turns on PMOS transistor 511, acting as a short circuit), then charging of capacitor 523 is allowed. After capacitor 523 is charged, transistor 511 is turned off, and thus the charge is stored in capacitor 523. To move the charge from capacitor 523 to bit line 521, NMOS transistor 513 is turned on by RWL 527, thereby causing NMOS transistor 513 to act as a short circuit.

[0074] Table 1 shows examples of memory-based computation array operations (which can be implemented, for example, by bit cell 500 in Figure 5A) configured according to the AND operation. start up PCWL Unit node (weight) Capacitor Node 1 0 1 1 1 0 0 0 0 1 1 0 0 1 0 0 Table 1: AND Operations

[0075] The first column (Start) in Table 1 includes the possible values ​​for the incoming start signal.

[0076] The second column (PCWL) of Table 1 includes PCWL values, whose activation is designed to enable in-memory computing functions according to various types of the present disclosure. Since transistor 511 in this example is a PMOS transistor, the PCWL value is the opposite of the activation value. For example, an in-memory computing array includes transistor 511, which is activated by an activation signal (PCWL signal) via the precharge word line (PCWL) 525.

[0077] The third column (cell nodes) of Table 1 includes the weight values ​​stored in the SRAM cell nodes (e.g., corresponding to the weights in the weight tensor), which can be used in spin operations.

[0078] The product generated by the diagram in the fourth column (capacitor nodes) of Table 1 will be stored as charge on the capacitor. For example, charge can be stored at a node of capacitor 523 or at a node of one of capacitors 423a-423i. When transistor 513 is activated, the charge from capacitor 523 moves to bit line 521. For example, referring to transistor 511, capacitor 523 is charged (e.g., capacitor node is "1") when the weight at cell node 517 is "1" (e.g., high voltage) and the input activation is "1" (therefore PCWL is "0"). For all other combinations, the capacitor node will have a value of 0.

[0079] Figure 5B illustrates additional details of another exemplary bit cell 550.

[0080] The difference between bit cell 550 and bit cell 500 in FIG5A is mainly in that it includes an additional precharge word line 552 coupled to an additional transistor 554.

[0081] Table 2 shows examples of in-memory computation array operations similar to those in Table 1 (such as those that can be implemented by bit cell 550 in Figure 5B), except for the settings according to the XNOR operation. start up PCWL1 PCWL2 Unit node (weight) Capacitor Node 1 0 1 1 1 1 0 1 0 0 0 1 0 1 0 0 1 0 0 1 Table 2: XNOR Operation

[0082] The first column (Start) in Table 2 includes the possible values ​​for the incoming start signal.

[0083] The second column (PCWL1) of Table 2 includes PCWL1 values, whose activation is designed to enable computational functions within the memory cell according to various states disclosed herein. Similarly, transistor 511 is a PMOS transistor, and its PCWL1 value is the opposite of the activation value.

[0084] The third column (PCWL2) of Table 2 includes PCWL2 values, which are activated by additional transistors designed to enable computational functions within the memory according to various types of this disclosure.

[0085] The fourth column (cell nodes) of Table 2 includes weight values ​​stored in SRAM cell nodes, such as weights that can be used in the spin operation, for example, weights corresponding to the weight tensor.

[0086] The product generated by the fifth column (capacitor nodes) of Table 2 will be stored as charge on capacitors (such as capacitor 523).

[0087] Figure 6 illustrates an example timing diagram 600 of various signals during the operation of a compute-in-memory (CIM) array.

[0088] In the illustrated example, the first row of timing diagram 600 shows the precharge word line PCWL going low (e.g., 425a in Figure 4 or 525 in Figure 5A). In this example, a low PCWL indicates an activation "1". When PCWL is low, the PMOS transistor is turned on, which allows the capacitor to be charged (if the weight is "1"). The second row shows the read word line RWL (e.g., read word line 427a in Figure 4 or 527 in Figure 5A). The third row shows the read bit line RBL (e.g., 418 in Figure 4 or 521 in Figure 5A), the fourth row shows the analog-to-digital converter (ADC) readout signal, and the fifth row shows the reset signal.

[0089] For example, referring to transistor 511 in Figure 5A, when the read word line RWL is high, the charge from capacitor 523 is gradually transferred to the read bit line RBL.

[0090] The summed charge / current / voltage (e.g., 403 in Figure 4 or the charge summed from bit line 521 in Figure 5A) is passed to a comparator or ADC (e.g., ADC 411 in Figure 4), where the summed charge is converted into a digital output (e.g., a digital signal / digital). The summation of the charge can occur in the accumulation region of timing diagram 600, and the readout from the ADC can be associated with the ADC readout region of timing diagram 600. After obtaining the ADC readout, a reset signal discharges all capacitors (e.g., capacitors 423a-423i) to prepare for processing the next set of start-up inputs. An example of spin processing in memory

[0091] Figure 7 illustrates an exemplary spiral layer architecture 700 implemented by a computation within memory (CIM) array 708. The spiral layer architecture 700 may be part of a spiral neural network (e.g., as described above with respect to Figure 1D) and is designed to process multidimensional data, such as tensor data.

[0092] In the illustrated example, the input 702 of the swirl layer architecture 700 has dimensions of 38 (height) × 11 (width) × 1 (depth). The output 704 of the swirl layer has dimensions of 34 x 10 x 64, comprising 64 output channels corresponding to the 64 cores of the core tensor 714 applied as part of the swirl process. Further, in this example, each of the 64 cores of the core tensor 714 (e.g., exemplary core 712) has a dimension of 5 × 2 × 1 (in general, the cores of the filter tensor 714 are equivalent to a 5 × 2 × 64 core).

[0093] During the spin process, each 5×2×1 core is spinned using input 702 to produce a 34×10×1 layer of output 704. During spin, the 640 weights of the core tensor 714 (5×2×64) can be stored in a memory computation (CIM) array 708, which in this instance comprises one column (i.e., 64 columns) for each core. Subsequently, the initiation input of each receptive field (e.g., receptive field input 706) in the 5x2 receptive fields is fed into the CIM array 708 using word lines (e.g., 716) and multiplied by the corresponding weights to produce a 1×1×64 output tensor (e.g., output tensor 710). Output tensor 704 represents the sum of the 1×1×64 individual output tensors of all receptive fields (e.g., receptive field input 706) of input 702. For simplicity, Figure 7 only shows a few illustrative lines of the inputs and outputs of the in-memory computing array 708.

[0094] In the illustrated example, the CIM array 708 includes word lines 716 and bit lines 718 (corresponding to columns of the CIM array 708) through which the CIM array 708 receives a sensing field (e.g., sensing field input 706). Although not illustrated, the CIM array 708 may also include precharged word lines (PCWL) and read word lines RWL (as described above with respect to Figures 4 and 5).

[0095] In this example, word line 716 is used for initial weight definition. However, once the initial weight definition is established, the initiation input initiates a specially designed row in the CIM bit cell to perform a MAC operation. Therefore, each intersection of bit line 718 and word line 716 represents a core weight value, which is multiplied by the input initiation on word line 716 to produce a product. The individual products along each bit line 718 are then summed to produce the corresponding output value of the output tensor 710. The summed value can be charge, current, or voltage. In this example, after processing the entire input 702 of the cyclotron layer, the output tensor 704 has a dimension of 34x10x64, but the CIM array 708 only produces 64 core outputs at a time. Therefore, the processing of the entire input 702 can be completed within 34x10 or 340 cycles. CIM architecture for depth-separable spin

[0096] Vector-matrix multiplication blocks implemented in memory for CIM architectures generally perform well with familiar spiral neural network processing, but they are not efficient in supporting depthwise separable spiral neural networks present in many state-of-the-art machine learning architectures.

[0097] Conventional solutions for improving efficiency include adding separate digital MAC blocks to handle processing for depth-wise portions of separable spinners, while the CIM array can handle point-wise portions of separable spinners. However, this hybrid approach results in increased data movement, which may offset the memory efficiency advantage of the CIM architecture. Furthermore, hybrid approaches typically involve additional hardware (e.g., digital multiplication and accumulation (DMAC) elements), increasing space and power requirements and processing latency. Additionally, the use of DMAC can affect the timing of processing operations and cause deviations from model output timing constraints (or other dependencies). To address this issue, various compromises may be made, such as reducing the frame rate of incoming data, increasing the clock rate of processing system elements (including the CIM array), reducing the input feature size, and so on.

[0098] The CIM architecture described in this paper improves the timing performance of processing operations for depth-separable spinner. Compared to conventional architectures that require more hardware (e.g., DMAC) and / or more data movement, these improvements advantageously result in reduced cycle time for depth-separable spinner operations and achieve higher total operations per watt per second (TOPS), i.e., TOPS / W.

[0099] Figures 8A and 8B illustrate certain configurations of a CIM system 800 including a CIM array 802 according to this disclosure. As shown in Figure 8A, the CIM array 802 can be used to implement cores 806, 808, and 809 for DW spin operations and core 890 for PW spin operations. For example, cores 806, 808, and 809 can correspond to cores 304A, 304B, and 304C, respectively, and core 890 can correspond to core 308, as described with respect to Figures 3A and 3B. DW spin operations can be performed sequentially during a first phase (phase 1). For example, core 806 can be processed during phase 1-1, core 808 can be processed during phase 1-2, and core 809 can be processed during phase 1-3. The outputs of the DW spin operations for cores 806, 808, and 809 can be used to generate inputs for core 890 to perform PW spin operations in a second phase. In this way, DW and PW spin operations can be performed using cores implemented on a single CIM array. DW cores can be implemented on the same columns of the CIM array, thus allowing for a greater number of DW cores to be implemented on the CIM array (compared to conventional implementations).

[0100] As shown in Figure 8B, the CIM system 800 includes a CIM array 802 configured for DW-rotation neural network computation and pointwise (PW)-CNN computation (e.g., CNN 1x1). Cores for DW-rotation and PW-rotation operations can be implemented on different column groups and initiated individually during different phases, as described with respect to Figure 8A. In some cases, cores (e.g., 3x3 cores) can be implemented on the same columns (also referred to herein as bit lines) of the CIM array 802. For example, a 3x3 core 806 with two-bit weights (i.e., nine two-bit values, including the first two-bit values ​​b01, b11, the second two-bit values ​​b02, b12, and so on) can be implemented using CIM cells on columns 810, 812 (e.g., one column for each bit width of the weights) and nine rows 814-1, 814-2 to 814-8 and 814-9 (e.g., also referred to herein as word lines (WL), collectively referred to as row 814, where one row exists for each value in the core)). Another core 808 can be implemented on columns 810, 812 and nine rows 820-1 to 820-9 (collectively referred to as row 820) to implement another 3x3 filter. Therefore, cores 806 and 808 are implemented on different rows but on the same columns. As a result, cores 806 and 808 can operate sequentially. In other words, starting a row for one of the cores 806 and 808 does not affect the row for the other core. However, starting a column for one core affects the column for the other core. Therefore, cores 806 and 808 can be operated sequentially. Although only two cores 806 and 808 are illustrated, more than two cores can be implemented in some configurations. For example, cores 806, 808, and 809 shown in Figure 8A can be implemented in CIM array 802.

[0101] In some configurations, the input start buffer for each core is filled (e.g., stored) with the corresponding output from the previous layer. Each core can be operated sequentially to produce a DW (Dual-Wave) output. The inputs of inactive cores can be filled with zeros (e.g., logic low) such that the read BL (RBL) output of the inactive core is zero (e.g., as supported in ternary mode bit cells). In this way, inactive cores do not affect the output from active cores implemented on columns (BLs).

[0102] In some configurations, rows for core 806 (e.g., row 814) may be coupled to start buffers 830-1, 830-2 through 830-8, and 830-9 (collectively referred to as start buffer 830), and rows for core 808 (e.g., row 820) may be coupled to start buffers 832-1 through 832-9 (collectively referred to as start buffer 832). The outputs of core 806 (e.g., at columns 810, 812) may be coupled to an analog-to-digital converter (ADC) 840. ADC 840 takes the signals from columns 810, 812 as inputs and produces a digital representation of the signals, taking into account the lower importance of the bits stored in column 812 compared to the bits stored in column 810.

[0103] CIM array 802 may also include PW cyclotron units 890 for PW cyclotron calculations on columns 816, 818, as shown. The output of PW cyclotron unit 890 (e.g., at columns 816, 818) may be coupled to ADC 842. For example, each input of ADC 840 may receive accumulated charge from row 814 from each column in columns 810, 812, and each input of ADC 842 may receive accumulated charge from each column in columns 816, 818, and each of ADCs 840, 842 generates a digital output signal based on the accumulated charge. For example, ADC 842 takes the signal from columns 816, 818 as input and generates a digital representation of the signal, taking into account the lower importance of bits stored in column 818 compared to bits stored in column 816. Although the ADCs 840 and 842 are illustrated as receiving signals from two columns to facilitate analog-to-digital conversion for a core with a 2-bit weighting parameter, the configurations described herein can be implemented for ADCs configured to receive signals from any number of columns (e.g., three columns to perform analog-to-digital conversion for a core with a 3-bit weighting parameter). In some configurations, an ADC such as the ADC 840 or 842 can be coupled to eight columns. Furthermore, in some configurations, the accumulation can be distributed across more than one ADC.

[0104] The outputs of ADCs 840 and 842 can be coupled to a nonlinear operation circuit 850 (and a buffer) to perform one or more nonlinear operations (e.g., sequentially), such as Rectified Linear Unit (ReLU) and Average Pooling (AvePool), to name a few. Nonlinear operations allow for complex mappings between inputs and outputs, and thus allow for learning and modeling of complex data, such as nonlinear or high-dimensional images, video, audio, and datasets. The output of the nonlinear operation circuit 850 can be coupled to a start-up output buffer circuit 860. The start-up output buffer circuit 860 can store the output from the nonlinear operation circuit 850 for use as a PW spin input for PW spin calculations via the PW spin unit 890. For example, the output of the start-up output buffer circuit 860 can be provided to a start-up buffer 830. The corresponding start-up input stored in the start-up buffer 830 can be provided to the PW spin unit 890 to perform PW spin calculations.

[0105] Although each of cores 806 and 808 includes two columns, allowing 2-bit weights to be stored in each row of the core, cores 806 and 808 can be implemented using any number of suitable columns, such as one column for 1-bit binary weights, or two or more columns for multi-bit weights. For example, each of cores 806 and 808 can be implemented using three columns to facilitate storing 3-bit weight parameters in each row of the core, or using a single column to facilitate storing 1-bit weights in each row of the core. Furthermore, although each of cores 806 and 808 utilizes nine rows for a 3x3 core for ease of understanding, cores 806 and 808 can be implemented using any number of rows to achieve an appropriate core size. Additionally, two or more cores can be implemented using a subset of the cells of the CIM array. For example, CIM array 802 can include one or more other cores where the cores of CIM array 802 are implemented in different rows and the same columns.

[0106] The patterns described herein offer flexibility in configuring any CIM array for DW spin operations as needed. For example, the number of rows used to implement each of the 806 and 808 cores can be increased to increase the size of each corresponding core (e.g., implementing a 5x5 core). Furthermore, some patterns allow for an increase in the number of cores that can be implemented on the CIM array compared to conventional implementations. In other words, some patterns of this disclosure reduce the area consumed on the CIM array for DW spin operations by implementing the cores for DW spin operations on the same columns. In this way, the number of cores that can be implemented for DW spin operations on the CIM array can be increased compared to conventional implementations. For example, a total of 113 3x3 filters can be implemented on a CIM array with 1024 rows. Therefore, the area consumed for implementing DW spin operations can be reduced compared to conventional implementations that can use DMAC hardware.

[0107] Figure 9 illustrates example operation 900 of signal processing via the CIM architecture 800 of Figure 8B, according to certain forms of this disclosure. A CIM array can be used for both DW spin computation and PW spin computation. The core used for DW spin operates in two stages on the same CIM array hardware.

[0108] During the first stage of the DW revolve, columns 810 and 812 used by the DW revolve core are active. Operation 900 can begin processing of the DW revolve layer. For example, at block 904, DW revolve weights can be loaded into the CIM cells used for the core. That is, at block 904, the DW 3x3 core weights can be grouped into rows and written into the CIM cells of cores 806 and 808 used for the CIM array 802 of FIG. 8. That is, 2-bit core weights can be provided to columns 810 and 812, and the pass gate switches of memory cells (e.g., memory cells b11 and b01 shown in FIG. 8) can be turned off to store the 2-bit core weights in the memory cells. Filter weights can be stored in each row of each of cores 806 and 808. The remaining CIM columns can be used to write PW revolve weights into PW revolve cell 890. Both the DW and PW spin weights will be updated for each subsequent layer. In some implementations, the CIM array can be divided into slices that can be configured in a tri-state mode, as described in more detail herein. In some cases, slices on the same column as the active core can be configured in a tri-state mode. In tri-state mode, the outputs of the slice's memory cells can be configured to have relatively high impedance, thereby effectively eliminating the cell's influence on the output.

[0109] At block 906, the DW spin-start input (e.g., at start buffer 830) can be sequentially applied to each set of rows of cores 806, 808 to produce a DW spin-start output for each core. Only one of cores 806, 808 can be active at any given time. Inactive filter rows can be placed in tri-state operating mode.

[0110] At block 908, ADC 840 can convert the spinner outputs of cores 806 and 808 (e.g., at columns 810 and 812) from the analog domain to the digital domain. Nonlinear operations can be performed via nonlinear operation circuit 850 based on the DW spinner outputs of ADC 840. The output from nonlinear operation circuit 850 can be applied to PW spinner inputs (e.g., stored in start-up output buffer circuit 860) to perform PW spinner operations. In other words, PW spinner inputs can be written to start-up buffer 830 and applied to PW spinner units 890 in rows 814 and 820 and columns 816 and 818.

[0111] Operation 900 can continue to stage 2 by processing the PW spin operation. For example, at block 912, the CIM array can be loaded using the core for PW spin. For example, PW spin columns (e.g., columns 816, 818) can be enabled, and DW spin columns (e.g., columns 810, 812) can be disabled. At block 914, PW spin can be performed, and the output of the PW spin can be converted to a digital signal via ADC 842. At block 916, ADC 842 can convert the output of PW spin unit 890 from the analog domain to the digital domain. A nonlinear start operation (e.g., ReLU) can be performed via nonlinear operation circuit 850 based on the PW spin output of ADC 842. Technologies for reducing power consumption and improving CIM array utilization

[0112] Figure 10 illustrates a CIM array 1000 divided into shards (also called sub-storages) according to certain states of this disclosure to save power. As an example, the CIM array 1000 may have 1024 rows and 256 columns. Individual shards of rows and columns can be enabled or disabled. For example, a shard may include 128 rows and 23 columns. As an example, shard 1002 (e.g., comprising multiple shards, such as shard 1004) may be active for revolving, while the remaining shards may be disabled. In other words, the remaining shards can be configured in a tri-state mode.

[0113] In some implementations, row and column filler cells can be implemented in the CIM array 1000. Filler circuitry (e.g., buffers or switches) can be used to enable or disable slicing of the CIM array to save power. For example, column filler cells can be implemented using AND gated logic, and row filler cells can be implemented using buffers on the write bit line (WBL) and transfer switches on the read bit line (RBL). The size and type of transfer switches can be configured based on linear specifications.

[0114] DW spirals can use relatively small core dimensions (3x3, 5x5, ...), and underutilization of the CIM array can affect the output signal-to-noise ratio (SNR) due to range compression (e.g., the neural network output is distributed over a small range due to nonlinear activation). Some aspects of this disclosure relate to techniques for improving SNR, as described in more detail with reference to Figure 11.

[0115] Figure 11 shows a CIM array implemented using a repeating core in some forms according to this disclosure.

[0116] As shown in the figure, each of cores 806 and 808 can be repeated to form a core group. For example, cores 806, 1104, and 1106 form core group 1102, and each of cores 806, 1104, and 1106 includes the same weight. Furthermore, multiple core groups, such as core groups 1102 and 1104, can be implemented on the same columns. Since the repeated cores 806, 1104, and 1106 in group 1102 have the same weight, the same starting input can be provided to each of the repeated cores in that group. The same applies to group 1104.

[0117] Repeating cores can produce the same output signal combined at each column (output), resulting in an increase in dynamic range at the output of the repeating core. For example, using three repeating cores can result in the dynamic range at the output of the repeating core provided to the ADC (e.g., ADC 840) becoming three times. This increased dynamic range at the core output promotes analog-to-digital conversion with higher accuracy because it can utilize a wider range of the ADC. In other words, using the entire range of the ADC input allows the ADC's digital output to more accurately identify the ADC's analog input and improves the ADC's signal-to-noise ratio (SNR).

[0118] In some configurations, relatively small slice sizes can be used for the CIM store (e.g., 16 rows and 32 columns), which runs DW spin, allowing for the deployment of a larger number of CIM cells to save power. For example, a group of three CIM cells (e.g., with multiple slices) can be designed to run a neural network architecture with a reverse bottleneck. The reverse bottleneck operation typically represents an operation to expand the input features, followed by a DW spin and a reduction of the DW output dimensionality via a PW spin.

[0119] As an example, the first CIM cell group (CIM1) can be used for bottleneck operations, the second CIM cell group (CIM2) can be used for DW spin operations, and the third CIM cell group (CIM3) can be used for bottleneck operations. In some configurations, CIM2 used for DW spin operations can have a finer sharding arrangement (e.g., 16 rows for 3x3 cores or 32 rows for 5x5 cores) to improve CIM array utilization, while CIM1 and CIM3 can have coarse-fine sharding (e.g., 64 or 128 rows) to avoid the impact of filler cells on non-DW spin operations. In this way, the reusability of the CIM array library can be doubled for both DW and non-DW operations.

[0120] The average (e.g., approximate) CIM utilization under coarse-to-fine sharding (e.g., where each shard uses 64 rows and 32 columns of a 1024-row CIM array) is 13.8% for 3x3 cores and 31.44% for 5x5 cores. In other words, only 13.8% of the active memory units in the CIM array can be used for 3x3 cores, while 31.44% of the active memory units in the CIM array can be used for 5x5 cores. Similarly, the average CIM utilization under fine-to-fine sharding (e.g., where each shard uses 16 rows and 32 columns in the case of a 1024-row CIM array) is 40.46% for 3x3 cores and 47.64% for 5x5 cores. The average CIM utilization under fine-to-fine sharding (e.g., where each shard uses 32 rows and 32 columns of a 1024-row CIM array) is 24.18% for 3x3 cores and 47.64% for 5x5 cores. Therefore, fine-grained slicing improves CIM array utilization for filters with smaller core sizes (e.g., filters used in many popular DW-CNN architectures). Improved CIM array utilization results in a higher percentage of active memory cells being utilized, thereby reducing power loss that would otherwise result from unutilized active memory cells.

[0121] Typically, utilization can be improved by choosing a slice size that is closer to the core size (e.g., during chip design). For example, a slice size of 16 can be used for a core size of 9. In some cases, the slice size can be determined to be a power of 2 (logarithmic scale) greater than the core size to improve flexibility in handling different neural network models. Example operation for performing neural network processing in a CIM array

[0122] Figure 12 is a flowchart illustrating an example operation 1200 for signal processing in a neural network according to certain forms of this disclosure. Operation 1200 can be performed by a neural network system, which may include a controller (such as, as described with respect to Figure 13, a CIM controller 1332) and a CIM system (such as, a CIM system 800).

[0123] Operation 1200 at block 1205 begins by the following operation: the neural network system performs a plurality of depth-wise (DW) spin operations via multiple cores (e.g., cores 806, 808, 809) implemented using multiple CIM cell groups on one or more first columns (e.g., columns 810, 812) of an in-memory computation (CIM) array (e.g., CIM array 802). As an example, performing a plurality of DW spin operations may include: loading a first set of CIM cells in multiple CIM cell groups via one or more first columns using a first plurality of weight parameters for a first core (e.g., core 806) among the multiple cores, the first set of CIM cells including a first plurality of rows (e.g., row 814) of the CIM array; and performing a first DW spin operation in the plurality of DW spin operations via the first cores, wherein performing the first DW spin operation includes applying a first start input (e.g., via start buffer 830) to the first plurality of rows. Performing a plurality of DW spin operations may also include: loading a second CIM cell set in a plurality of CIM cell groups via one or more first columns, using a second plurality of weight parameters of a second core (e.g., core 808) among a plurality of cores, the second CIM cell set comprising one or more first columns and a second plurality of rows (e.g., row 820) of a CIM array, the first plurality of rows being different from the second plurality of rows; and performing a second DW spin operation in the plurality of DW spin operations via a second core, wherein performing the second DW spin operation includes applying a second start input (e.g., via start buffer 832) to the second plurality of rows. In some embodiments, the first CIM cell set comprises a subset of cells of the CIM array, and the second CIM cell set comprises another subset of cells of the CIM array.

[0124] At block 1210, the neural network system can (e.g., via ADC 840 and nonlinear operation circuit 850) generate input signals for a PW spin operation based on the outputs from a plurality of DW spin operations. At block 1215, the neural network system can perform a PW spin operation based on the input signals, the PW spin operation being performed via a core implemented using CIM cell groups on one or more second columns of a CIM array. For example, performing a PW spin operation may include loading CIM cell groups for cores on one or more second columns using a third plurality of weights. In some cases, the neural network system can generate digital signals by converting voltages at one or more first columns from the analog domain to the digital domain after performing a plurality of DW spin operations. The input signals to the CIM cell groups on one or more second columns can be generated based on digital signals.

[0125] In some cases, the core can be repeated to improve the utilization of the CIM array and the dynamic range of the ADC, as described herein. For example, a neural network system can load a third set of CIM cells from a plurality of CIM cell groups via one or more first columns using a first plurality of weight parameters to perform a first DW spin operation, the third set of CIM cells comprising one or more first columns and a third plurality of rows of the CIM array. Example processing system for selective spinning during execution phase

[0126] Figure 13 illustrates an example electronic device 1300. Electronic device 1300 can be configured to perform the methods described herein, including operation 1200 described with respect to Figure 12.

[0127] Electronic device 1300 includes a central processing unit (CPU) 1302, which in some cases may be a multi-core CPU. Instructions executed at CPU 1302 may be loaded, for example, from program memory associated with CPU 1302 or from memory 1324.

[0128] Electronic device 1300 also includes additional processing blocks tailored for specific functions, such as a graphics processing unit (GPU) 1304, a digital signal processor (DSP) 1306, a neural processing unit (NPU) 1308, a multimedia processing block 1310, and a wireless connectivity processing block 1312. In one implementation, the NPU 1308 is implemented in one or more of the CPU 1302, GPU 1304, and / or DSP 1306.

[0129] In some embodiments, the wireless connectivity processing block 1312 may include components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and wireless data transmission standards. The wireless connectivity processing block 1612 is further connected to one or more antennas 1314 to facilitate wireless communication.

[0130] Electronic device 1300 may also include one or more sensor processors 1316 associated with any type of sensor, one or more image signal processors (ISP) 1318 associated with any type of image sensor, and / or navigation processor 1320, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0131] Electronic device 1300 may also include one or more input and / or output devices 1322, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc. In some embodiments, one or more processors in the processor of electronic device 1300 may be based on the ARM instruction set.

[0132] Electronic device 1300 also includes memory 1324, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 1324 includes computer-executable components that can be executed by one or more processors of the aforementioned processors of electronic device 1300 or CIM controller 1332. For example, as described herein, electronic device 1300 may include CIM circuitry 1326, which includes one or more CIM arrays, such as CIM array 802 and CIM array 804. CIM circuitry 1326 may be controlled via CIM controller 1332. For example, in some cases, memory 1324 may include code 1324B for spin-around (e.g., performing DW or PW spin-around operations by applying a start input). Memory 1324 may also include code 1324C for generating input signals. Memory 1324 may optionally include code 1324A for loading (e.g., loading CIM cells using weight parameters). As shown, CIM controller 1332 may include circuitry 1328B for rotation (e.g., performing DW or PW rotation operations by applying a start input). CIM controller 1332 may also include circuitry 1328C for generating input signals. CIM controller 1332 may also optionally include circuitry 1328A for loading (e.g., loading CIM cells using weight parameters). The illustrated components and other components not illustrated may be configured to perform various states of the methods described herein.

[0133] In some configurations (such as when electronic device 1300 is a server device), one or more of the following configurations may be omitted from the configurations illustrated in FIG13: multimedia processing block 1310, wireless connection component 1312, antenna 1314, sensor processor 1316, ISP 1318, or navigation 1320. Example Terms

[0134] 1. An apparatus for signal processing in a neural network, comprising: a first in-memory computing (CIM) unit configured for depth-by-depth (DW) neural network computation, the first CIM unit set including one or more first columns and a first plurality of rows of a CIM array; a second CIM unit set configured for a second core of the neural network computation, the second CIM unit set including the one or more first columns and the second plurality of rows of the CIM array, wherein the first plurality of rows are different from the second plurality of rows; and a third CIM unit set of the CIM array configured for a third core of point-by-point (PW) neural network computation.

[0135] Version 2. The apparatus as described in Version 1, wherein the first CIM unit set includes a subset of the units of the CIM array, and the second CIM unit set includes another subset of the units of the CIM array.

[0136] State 3. The apparatus as described in State 2, wherein the third set of CIM units is a third subset of the CIM array.

[0137] 4. The apparatus of any one of 1-3, wherein the third CIM unit set includes one or more second columns of the CIM array and the first plurality of rows, the one or more second columns being different from the one or more first columns.

[0138] 5. The apparatus of any one of embodiments 1-4, further comprising: an analog-to-digital converter (ADC) coupled to the one or more first columns.

[0139] Version 6. The apparatus as described in Version 5 further includes: a nonlinear circuit coupled to the output of the ADC.

[0140] 7. The apparatus of any one of embodiments 1-6 further comprises: a third set of CIM units configured for a third core for computation of the neural network, the third set of CIM units comprising the one or more first columns and the third plurality of rows of the CIM array.

[0141] Version 8. The apparatus as described in Version 7, wherein the same weight parameters are configured to be stored in the first CIM unit set and the third CIM unit set when the neural network computation is performed.

[0142] 9. The apparatus of any one of embodiments 1-8, wherein: one or more CIM cells in each of the first plurality of rows of the first CIM cell set are configured to store a first weight parameter; and one or more CIM cells in each of the second plurality of rows of the second CIM cell set are configured to store a second weight parameter.

[0143] Version 10. The apparatus as described in Version 9, wherein the number of the one or more first columns is associated with the number of one or more bits of the first weighting parameter.

[0144] 11. A method for signal processing in a neural network, comprising: performing a plurality of depth-by-depth (DW) spin operations via a plurality of cores implemented using a plurality of CIM unit groups on one or more first columns of a memory computing (CIM) array; generating input signals for point-by-point (PW) spin operations based on outputs from the plurality of DW spin operations; and performing PW spin operations based on the input signals, the PW spin operations being performed via cores implemented using CIM unit groups on one or more second columns of the CIM array.

[0145] State 12. The method as described in State 11, wherein performing the plurality of DW spin operations comprises: loading a first CIM cell set of the plurality of CIM cell groups via the one or more first columns using a first plurality of weight parameters for a first core among the plurality of cores, the first CIM cell set comprising a first plurality of rows of the CIM array; performing a first DW spin operation in the plurality of DW spin operations via the first core, wherein performing the first DW spin operation comprises applying a first start input to the first plurality of... The second CIM cell set of the plurality of CIM cell groups is loaded via the one or more first columns using a second plurality of weight parameters of a second core among the plurality of cores. The second CIM cell set includes the one or more first columns and the second plurality of rows of the CIM array, wherein the first plurality of rows are different from the second plurality of rows; and the second DW spin operation of the plurality of DW spin operations is performed via the second core, wherein performing the second DW spin operation includes applying a second start input to the second plurality of rows.

[0146] Version 13. The method as described in Version 12, wherein the first CIM cell set includes a subset of the cells of the CIM array, and the second CIM cell set includes another subset of the cells of the CIM array.

[0147] State 14. The method as described in State 13, wherein performing the PW spin operation includes using a third complex number of weights to load the CIM cell group for the core on the one or more second columns.

[0148] Version 15. The method of Version 14 further includes: generating a digital signal by converting the voltages at one or more first columns from the analog domain to the digital domain after performing the plurality of DW spin operations, wherein the input signals of the CIM cell group on the one or more second columns are generated based on the digital signal.

[0149] Sample 16. The method of any one of Samples 12-15 further includes: loading a third CIM cell set of the plurality of CIM cell groups via the one or more first columns using the first plurality of weight parameters to perform the first DW spin operation, the third CIM cell set including the one or more first columns and the third plurality of rows of the CIM array.

[0150] 17. A non-transitory computer-readable medium including instructions that, when executed by one or more processors of a processing system, cause the processing system to perform signal processing in a neural network, the method comprising: performing a plurality of depthwise (DW) spin operations via a plurality of cores implemented using a plurality of CIM cell groups on one or more first columns of a memory computing (CIM) array; and generating input signals for pointwise (PW) spin operations based on outputs from the plurality of DW spin operations; and performing PW spin operations based on the input signals, the PW spin operations being performed via CIM cell groups on one or more second columns of the CIM array.

[0151] Version 18. The non-transitory computer-readable medium as described in Version 17, wherein performing the plurality of DW spin operations includes: loading a first CIM cell set of the plurality of CIM cell groups via the one or more first columns using a first plurality of weight parameters for a first core among the plurality of cores, the first CIM cell set including a first plurality of rows of the CIM array; performing a first DW spin operation of the plurality of DW spin operations via the first core, wherein performing the first DW spin operation includes applying a first start input to the... A first plurality of rows; a second set of CIM cells is loaded into the plurality of CIM cell groups via the one or more first columns using a second plurality of weight parameters of a second core among the plurality of cores, the second set of CIM cells including the one or more first columns and the second plurality of rows of the CIM array, wherein the first plurality of rows are different from the second plurality of rows; and a second DW spin operation is performed in the plurality of DW spin operations via the second core, wherein performing the second DW spin operation includes applying a second start input to the second plurality of rows.

[0152] Version 19. The non-transitory computer-readable medium as described in Version 18, wherein the first set of CIM cells includes a subset of cells of the CIM array, and the second set of CIM cells includes another subset of cells of the CIM array.

[0153] Sample 20. The non-transitory computer-readable medium as described in Sample 19, wherein performing the PW spin operation includes using a third complex number of weights to load the CIM cell group for the third core on the one or more second columns.

[0154] Sample 21. The non-transitory computer-readable medium as described in Sample 20, the method further comprising: generating a digital signal by converting the voltages at one or more first columns from the analog domain to the digital domain after performing the plurality of DW spin operations, wherein the input signals of the CIM cell group on the one or more second columns are generated based on the digital signal.

[0155] Sample 22. The non-transitory computer-readable medium as described in any one of Samples 18-21, the method further comprising: loading a third CIM cell set of the plurality of CIM cell groups via the one or more first columns using the first plurality of weight parameters to perform the first DW spin operation, the third CIM cell set including the one or more first columns and the third plurality of rows of the CIM array. Additional considerations

[0156] The preceding description is provided to enable those skilled in the art to practice the various forms described herein. The examples discussed herein do not limit the scope, applicability, or forms set forth in the claims. Various modifications to these forms will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other forms. For example, changes can be made to the function and arrangement of the elements discussed without departing from the scope of this disclosure. Various procedures or components can be omitted, substituted, or added as appropriate in the various examples. For example, the described method can be performed in a different order than that described, and steps can be added, omitted, or combined. Furthermore, features described with respect to some examples can be combined into some other examples. For example, an apparatus or a method can be implemented using any number of forms set forth herein. Moreover, the scope of this disclosure is intended to cover such apparatus or methods practiced using structures, functions, or structures and functions other than or different from the various forms of the disclosure set forth herein. It should be understood that any form of the disclosure herein can be embodied by one or more elements of the claims.

[0157] As used herein, the term "exemplary" means "serving as an example, instance, or illustration." Any state described as "exemplary" in this document is not necessarily to be construed as preferred or superior to other states.

[0158] As used in this article, the phrase "at least one of" in a list of items refers to any combination of those items, including a single member. As an example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination of multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, cc, and ccc, or any other ordering of a, b, and c).

[0159] As used herein, the term "decision" encompasses a wide variety of actions. For example, "decision" can include calculation, operation, processing, deduction, investigation, examination (e.g., examining a table, database, or other data structure), ascertainment, and so on. Furthermore, "decision" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and so on. Additionally, "decision" can include parsing, selecting, choosing, creating, and so on.

[0160] The methods disclosed herein include one or more steps or actions for implementing the methods. These method steps and / or actions can be interchanged without departing from the scope of the claim. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claim. Furthermore, the various operations of the above methods can be performed by any suitable component capable of performing the corresponding function. Such components can include various hardware and / or software components and / or modules, including but not limited to: circuits, application-specific integrated circuits (ASICs), or processors. Typically, where operations are shown in the figures, these operations can have corresponding paired components plus functional components with similar numbering.

[0161] The following claims are not intended to be limited to the forms illustrated herein, but are given the full range consistent with the language of the claims. Within a claim, unless expressly stated otherwise, references to singular elements do not mean "one and only one," but rather "one or more." Unless expressly stated otherwise, the term "some" means one or more. No claim element is to be interpreted under Patent Law §112(f) unless the element is explicitly described using the phrase "component for..." or, in the case of a method claim, using the frequency domain phrase "step for...". All structural and functional equivalents of the elements throughout the various forms described in this disclosure are expressly incorporated herein by reference, and are intended to be covered by the claims, such structural and functional equivalents being known to or later to those skilled in the art. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is expressly stated in a claim.

[0162] 100: Deep Spin Network (DCN) 102: Fully Connected Neural Networks 104: Local Connectivity Neural Networks 106: Rotational Neural Network 108: Shared 110:ADC 112: value 114: value 116: value 118: First Feature Atlas 120: Second Feature Atlas 122: Output 124: Fully Connected Layer 126: Image 128: Output feature vector 130: Image capturing device 132: Cyclone 204: Spinning Core 206: Feature Map 302: Input Image 304A-C: Core 306: Feature Map 308: Core 310: Feature Map 400: Computation in Memory (CIM) Array 403: Charge / Current / Voltage 405a: SRAM bit cell 405b: SRAM bit cell 405c: SRAM bit cell 405d: SRAM bit cell 405e: SRAM bit cell 405f: SRAM bit cell 405g: SRAM bit cell 405h: SRAM bit cell 405i: SRAM bit cell 410a: Analog-to-Digital Converter (ADC) 410b: Analog-to-Digital Converter (ADC) 410c: Analog-to-Digital Converter (ADC) 411a: PMOS transistor 411b: PMOS transistor 411c: PMOS transistor 411d: PMOS transistor 411e: PMOS transistor 411f: PMOS transistor 411g: PMOS transistor 411h: PMOS transistor 411i: PMOS transistor 413: NMOS transistor 413a: NMOS transistor 413b: NMOS transistor 413c: NMOS transistor 413d: NMOS transistor 413e: NMOS transistor 413f: NMOS transistor 413g: NMOS transistor 413h: NMOS transistor 413i: NMOS transistor 418a: Bitline 418b: Bitline 418c: Bit line 423a: Capacitor 423b: Capacitor 423c: Capacitor 423d: Capacitor 423e: Capacitor 423f: Capacitor 423g: Capacitor 423h: Capacitor 423i: Capacitor 425a: Precharged Word Line (PCWL) 425b: Precharged Word Line (PCWL) 425c: Precharged Word Cable (PCWL) 427a: Read Word Line (RWL) 427b: Read Word Line (RWL) 427c: Read Word Line (RWL) 500: Bit unit 501: Static Random Access Memory (SRAM) Unit 511: PMOS transistor 513: NMOS transistor 516: Write Word Line (WWL) 517: Node 519: Node 521: Bitline 523: Capacitor 525: Precharged digital cable 527: Read word lines 552: Extra pre-charged word cable 554: Additional Transistor 600: Timing Diagram 700: Spin Layer Architecture 702: Input 704: Output 706: Sensing Field Input 708: Computation in Memory (CIM) Array 710: Output Tensor 712: Core 714: Core Tensor 716: Word Line 718: Bitline 800:CIM System 802: CIM Array 806: Core 808: Core 809: Core 810: column 812: column 814-1: line 814-2: line 814-8: Line 814-9: Line 816: Column 818: Column 820-1: line 820-2: line 820-8: line 820-9: Line 830-1: Start Buffer 830-2: Start Buffer 830-8: Start the buffer 830-9: Start the buffer 832-1: Start Buffer 832-2: Start Buffer 832-8: Start the buffer 832-9: Start the buffer 840: Analog-to-Digital Converter (ADC) 842:ADC 850: Nonlinear Operational Circuit 860: Startup output buffer circuit 890:PW Cyclone Unit 900: Operation 904: Square 906: Square 908: Square 912: Square 914: Square 916: Square 1000:CIM Array 1002: Fragmentation 1004: Fragmentation 1102: Core Group 1104: Core Group 1106: Core 1200: Operation 1205: Square 1210: Square 1215: Square 1300: Electronic Equipment 1302: Central Processing Unit (CPU) 1304: Graphics Processing Unit (GPU) 1306: Digital Signal Processor (DSP) 1308: Neural Processing Unit (NPU) 1310: Multimedia Processing Block 1312: Wireless Connection Processing Block 1314: Antenna 1316: Sensor Processor 1318: Image Signal Processor (ISP) 1320: Navigation Processor 1322: Input and / or output devices 1324: Memory 1324A: Code 1324B: Code 1324C: Code 1326: CIM Circuit 1328A: Circuit 1328B: Circuit 1328C: Circuit 1332: CIM Controller ADC: Analog-to-Digital Converter b01: Two-dimensional value b02: Two-digit value b11:b08 b12:b08 PCWL: Precharged Word Cable RBL: Read Bit Line RWL: Read word line WBBL: Write bit bar WBL: Write Bit Line WWL: Write word line

Claims

1. An apparatus for signal processing in a neural network, comprising: A first in-memory computation (CIM) unit set configured as a first core for a depth-by-depth (DW) neural network computation, the first CIM unit set including one or more first columns and a first plurality of rows of a CIM array; a second CIM unit set configured as a second core for the DW neural network computation, the second CIM unit set including the one or more first columns and the second plurality of rows of the CIM array, wherein the first plurality of rows are different from the second plurality of rows; and a third CIM unit set of the CIM array configured as a third core for a point-by-point (PW) neural network computation, wherein the third CIM unit set includes one or more second columns and the first plurality of rows of the CIM array, the one or more second columns being different from the one or more first columns.

2. The apparatus of claim 1, wherein the first CIM cell set comprises a subset of cells of the CIM array, and the second CIM cell set comprises another subset of cells of the CIM array.

3. The apparatus as claimed in claim 2, wherein the third set of CIM cells is a third subset of the CIM array.

4. The apparatus as claimed in claim 1, further comprising: An analog-to-digital converter (ADC) coupled to one or more first columns.

5. The apparatus as claimed in claim 4, further comprising: A nonlinear circuit coupled to the output of the ADC.

6. The apparatus as claimed in claim 1, further comprising: A fourth set of CIM cells, configured as a fourth core for computation of the DW neural network, the fourth set of CIM cells comprising the one or more first columns and the third plurality of rows of the CIM array.

7. The apparatus of claim 6, wherein, during the execution of the DW neural network computation, the same weight parameters are configured to be stored in the first CIM cell set and the fourth CIM cell set.

8. The apparatus as claimed in claim 1, wherein: One or more CIM cells in each of the first plurality of rows in the first CIM cell set are configured to store a first weight parameter; and one or more CIM cells in each of the second plurality of rows in the second CIM cell set are configured to store a second weight parameter.

9. The apparatus of claim 8, wherein the quantity of the one or more first columns is associated with the quantity of the one or more bits of the first weight parameter.

10. A method for signal processing in a neural network, comprising the steps of: performing a plurality of depthwise (DW) spin operations via a plurality of cores implemented using a plurality of CIM cell groups on one or more first columns of a memory computation (CIM) array; generating an input signal for a pointwise (PW) spin operation based on an output from the plurality of DW spin operations; and performing the PW spin operation based on the input signal, the PW spin operation being performed via a core implemented using a CIM cell group on one or more second columns of the CIM array, wherein performing the plurality of DW spin operations comprises the step of: loading a first CIM cell set of the plurality of CIM cell groups via the one or more first columns using a first plurality of weight parameters for a first core among the plurality of cores, the first CIM cell set comprising a first plurality of rows of the CIM array; The first DW spin operation is performed via the first core, wherein performing the first DW spin operation includes applying a first start input to the first plurality of rows; a second CIM cell set of the plurality of CIM cell groups is loaded via the one or more first columns using a second plurality of weight parameters of a second core among the plurality of cores, the second CIM cell set including the one or more first columns and the second plurality of rows of the CIM array, wherein the first plurality of rows are different from the second plurality of rows; and a second DW spin operation is performed via the second core, wherein performing the second DW spin operation includes applying a second start input to the second plurality of rows.

11. The method of claim 10, wherein the first CIM cell set comprises a subset of cells of the CIM array, and the second CIM cell set comprises another subset of cells of the CIM array.

12. The method of claim 11, wherein performing the PW spin operation includes using a third complex number of weights to load the CIM cell group for the core on the one or more second columns.

13. The method of claim 12 further includes the step of: generating a digital signal by converting a voltage at one or more first columns from an analog domain to a digital domain after performing the plurality of DW spin operations, wherein the input signals of the CIM cell group on the one or more second columns are generated based on the digital signal.

14. The method of claim 10 further includes the step of: loading a third CIM cell set of the plurality of CIM cell groups via the one or more first columns using the first plurality of weight parameters to perform the first DW spin operation, the third CIM cell set including the one or more first columns and the third plurality of rows of the CIM array.

15. A non-transitory computer-readable medium including instructions that, when executed by one or more processors of a processing system, cause the processing system to perform signal processing in a neural network, the method comprising: A plurality of depth-by-depth (DW) spin operations are performed via a plurality of cores implemented using a plurality of CIM cell groups on one or more first columns of a memory computing (CIM) array; and an input signal for a point-by-point (PW) spin operation is generated based on an output from the plurality of DW spin operations; and the PW spin operation is performed based on the input signal, the PW spin operation being performed via a CIM cell group on one or more second columns of the CIM array, wherein performing the plurality of DW spin operations includes: loading a first CIM cell set of the plurality of CIM cell groups via the one or more first columns using a first plurality of weight parameters for a first core among the plurality of cores, the first CIM cell set including a first plurality of rows of the CIM array; and performing a first DW spin operation among the plurality of DW spin operations via the first core, wherein performing the first DW spin operation includes applying a first start input to the first plurality of rows; A second CIM cell set of the plurality of CIM cell groups is loaded via one or more first columns using a second plurality of weight parameters of a second core among the plurality of cores. The second CIM cell set includes the one or more first columns and a second plurality of rows of the CIM array, wherein the first plurality of rows are different from the second plurality of rows; and a second DW spin operation is performed via the second core among the plurality of DW spin operations, wherein performing the second DW spin operation includes applying a second start input to the second plurality of rows.

16. The non-transitory computer-readable medium as claimed in claim 15, wherein the first set of CIM cells includes a subset of cells of the CIM array, and the second set of CIM cells includes another subset of cells of the CIM array.

17. The non-transitory computer-readable medium as claimed in claim 16, wherein performing the PW spin operation includes using a third plurality of weights to load the CIM cell group for a third core on one or more second columns.

18. The non-transitory computer-readable medium as described in claim 17, the method further comprising: A digital signal is generated by converting a voltage at one or more first columns from an analog domain to a digital domain after performing the plurality of DW spin operations, wherein the input signals of the CIM cell group on one or more second columns are generated based on the digital signal.

19. The non-transitory computer-readable medium as described in claim 15, the method further comprising: A third CIM cell set is loaded from the plurality of CIM cell groups via the one or more first columns and the first plurality of weight parameters to perform the first DW spin operation. The third CIM cell set includes the one or more first columns and the third plurality of rows of the CIM array.

Images