In-memory computation (CIM) architecture and dataflow supporting depth convolutional neural networks (CNNs)

Abstract

Some aspects provide an apparatus for signal processing in a neural network. The apparatus generally includes a first set of computation in memory (CIM) cells configured as a first kernel for neural network computation, the first set of CIM cells including one or more first columns and a first plurality of rows of a CIM array, and a second set of CIM cells configured as a second kernel for neural network computation, the second set of CIM cells including one or more second columns and a second plurality of rows of the CIM array. In some aspects, the one or more first columns are distinct from the one or more second columns, and the first plurality of rows are distinct from the second plurality of rows.

Description

Technical Field

[0001] Cross - Reference to Related Applications

[0001] This application claims priority to U.S. Application No. 17 / 361,784, filed June 29, 2021, which is assigned to the assignee of this application and is hereby incorporated by reference in its entirety.

Background Art

[0002]

[0002] Aspects of the present disclosure relate to performing machine learning tasks, and more particularly, to in - memory computing architectures and data flows.

[0003]

[0003] Machine learning is generally a process of creating a trained model (e.g., an artificial neural network, a tree, or other structure) that represents a generalization fit to a set of a priori known training data. By applying the trained model to new data, inferences are generated, and the inferences can be used to gain insights into the new data. In some cases, applying the model to new data is described as "performing an inference" on the new data.

[0004]

[0004] As the use of machine learning has surged to enable a variety of machine learning (or artificial intelligence) tasks, the need for more efficient processing of machine learning model data has arisen. In some cases, dedicated hardware such as machine learning accelerators can be used to enhance the capabilities of processing systems that process machine learning model data. However, such hardware requires space and power, which are not always available on the processing device. For example, “edge processing” devices such as mobile devices, always-on devices, and Internet of Things (IoT) devices must balance processing power with power and packaging constraints. Furthermore, accelerators may need to move data across a common data bus, which can cause significant power consumption and introduce latency to other processes that share the data bus. Thus, other forms of processing systems are being considered for processing machine learning model data.

[0005]

[0005] Memory devices are an example of another aspect of a processing system that can be utilized to perform processing of machine learning model data through a so-called computation in memory (CIM) process. Unfortunately, conventional CIM processes may not be able to perform processing of complex model architectures such as depth-separable convolutional neural networks without additional hardware elements such as digital multiply-and-accumulate circuits (DMACs) and associated peripherals. These additional hardware elements use additional space, power, and complexity in their implementation, which tends to reduce the benefits of utilizing memory devices as additional computational resources. Even if an auxiliary aspect of the processing system has a DMAC available to perform processing that cannot be performed directly in memory, moving data between those auxiliary aspects requires time and power, and therefore reduces the benefits of the CIM process.

[0006]

[0006] Therefore, there is a need for systems and methods to perform in-memory computations of a wider variety of machine learning model architectures, such as depth separable convolutional neural networks. [Overview of the project]

[0007]

[0007] Several embodiments provide a device for signal processing in a neural network. The device generally includes a first set of in-memory computation (CIM) cells configured as a first kernel for neural network computation, the first set of CIM cells comprising one or more first columns and a first plurality of rows of a CIM array, and a second set of CIM cells configured as a second kernel for neural network computation, the second set of CIM cells comprising one or more second columns and a second plurality of rows of a CIM array. In some embodiments, one or more first columns are different from one or more second columns, and the first plurality of rows are different from the second plurality of rows.

[0008]

[0008] Several embodiments provide a method for signal processing in a neural network. This method generally involves loading a first plurality of weight parameters of a first kernel into a first set of CIM cells via one or more first columns, wherein the first set of CIM cells includes one or more first columns and a first plurality of rows of a CIM array, in order to perform a neural network computation. This method also includes loading a second plurality of weight parameters of a second kernel into a second set of CIM cells via one or more second columns, wherein the second set of CIM cells includes one or more second columns and a second plurality of rows of a CIM array, in order to perform a neural network computation. The one or more first columns may be different from the one or more second columns, and the first plurality of rows may be different from the second plurality of rows. This method also may involve performing a neural network computation by applying a first activation input to the first plurality of rows and a second activation input to the second plurality of rows.

[0009]

[0009] Some embodiments provide a non-temporary computer-readable medium comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method of signal processing in a neural network. The method generally includes loading a first plurality of weight parameters of a first kernel into a first set of CIM cells via one or more first columns, wherein the first set of CIM cells includes one or more first columns and a first plurality of rows of a CIM array, in order to perform a neural network computation. The method also includes loading a second plurality of weight parameters of a second kernel into a second set of CIM cells via one or more second columns, wherein the second set of CIM cells includes one or more second columns and a second plurality of rows of a CIM array, in order to perform a neural network computation. The one or more first columns may be different from the one or more second columns, and the first plurality of rows may be different from the second plurality of rows. This method may also involve performing neural network computations by applying a first activation input to a first set of rows and a second activation input to a second set of rows.

[0010]

[0010] Another embodiment provides a processing system configured to perform the methods described above and the methods described herein; a non-temporary computer-readable medium having instructions, when executed by one or more processors of the processing system, that cause the processing system to perform the methods described above and the methods described herein; a computer program product embodied on a computer-readable storage medium having code that performs the methods described above and the methods described herein; and a processing system having means for performing the methods described above and the methods described herein.

[0011]

[0011] The following description and related drawings detail some exemplary features of one or more embodiments. [Brief explanation of the drawing]

[0012]

[0012] The attached figures illustrate some of the one or more embodiments and should therefore not be considered as limiting the scope of the present disclosure. [Figure 1A]

[0013] This figure shows examples of various types of neural networks. [Figure 1B] This figure shows examples of various types of neural networks. [Figure 1C] This figure shows examples of various types of neural networks. [Figure 1D] This figure shows examples of various types of neural networks. [Figure 2]

[0014] This figure shows an example of a conventional convolution operation. [Figure 3A]

[0015] This figure shows an example of a depth-separable convolution operation. [Figure 3B] This figure shows an example of a depth-separable convolution operation. [Figure 4]

[0016] This figure shows an exemplary in-memory computation (CIM) array configured to perform machine learning model calculations. [Figure 5A]

[0017] Figure 4 shows additional details of an exemplary bit cell that can represent bitccells. [Figure 5B] Figure 4 shows additional details of an exemplary bit cell that can represent bitccells. [Figure 6]

[0018] This is an illustrative timing diagram of various signals during in-memory computation (CIM) array operations. [Figure 7]

[0019] This figure shows an exemplary convolutional layer architecture implemented using an in-memory computation (CIM) array. [Figure 8]

[0020] This figure shows a CIM architecture including multiple CIM arrays according to some aspects of the present disclosure. [Figure 9]

[0021] A diagram showing exemplary operations for signaling processing via a CIM architecture, according to some aspects of the present disclosure. [Figure 10]

[0022] A diagram showing a CIM array divided into sub-banks to save power, according to some aspects of the present disclosure. [Figure 11]

[0023] A diagram showing a CIM array having obliquely stacked kernels, according to some aspects of the present disclosure. [Figure 12]

[0024] A diagram showing a CIM array implemented using repeated kernels, according to some aspects of the present disclosure. [Figure 13]

[0025] A flowchart showing exemplary operations for signal processing in a neural network, according to some aspects of the present disclosure. [Figure 14]

[0026] A diagram showing an exemplary electronic device configured to perform operations for signal processing in a neural network, according to some aspects of the present disclosure.

[0013]

[0027] For ease of understanding, the same reference numbers are used to designate identical elements common to the drawings, where possible. It is contemplated that elements and features of one aspect may be beneficially incorporated into other aspects without further recitation.

Modes for Carrying Out the Invention

[0014]

[0028] Aspects of this disclosure provide apparatus, methods, processing systems, and computer-readable media for performing in-memory computation (CIM) of machine learning models including depth-separable convolutional neural networks. Some aspects relate to CIM cells of a CIM array configured for different kernels, where the CIM cells are implemented on different rows and columns of the CIM array to facilitate parallel operation of first and second kernels. For example, a first kernel may be implemented on a first row and column of the CIM array, and a second kernel may be implemented on a second row and column of the CIM array, where the first row and column is different from the second row and column. Each kernel implemented on a different row and column can be coupled to an analog-to-digital converter (ADC) to enable parallel depth-wise (DW) computation and analog-to-digital conversion via the kernel. The results of the DW computation may be input to a nonlinear activation circuit for further processing, as described in more detail herein, and may be input to another CIM array for point-by-point computation.

[0015]

[0029] CIM-based machine learning (ML) / artificial intelligence (AI) task accelerators can be used for a wide variety of tasks, including image and audio processing. Furthermore, CIM can be based on various types of memory architectures, such as DRAM, SRAM (e.g., based on SRAM cells as shown in Figure 5), MRAM, and ReRAM, and can be mounted on various types of processing units, including central processor units (CPUs), digital signal processors (DSPs), graphical processor units (GPUs), field-programmable gate arrays (FPGAs), and AI accelerators. Generally, CIM can advantageously mitigate the "memory wall" problem, where data movement in and out of memory consumes more power than data computation. Therefore, significant power savings can be achieved by performing computations within memory. This is particularly useful for various types of electronic devices, such as low-power edge processing devices and mobile devices.

[0016]

[0030] For example, a mobile device may include a memory device configured to store data and in-memory computation operations. The mobile device may be configured to perform ML / AI operations based on data generated by the mobile device, such as image data generated by the mobile device's camera sensor. Thus, the mobile device's memory controller unit (MCU) can load weights from another onboard memory (e.g., flash or RAM) into the memory device's CIM array and allocate input feature buffers and output (e.g., activation) buffers. The processing device can then begin processing the image data, for example, by loading layers in the input buffer and processing the layers using the weights loaded in the CIM array. This processing can be iterated over for each layer of image data, and the output (e.g., activation) can be stored in an output buffer and then used by the mobile device for ML / AI tasks such as face recognition.

[0017] A brief background on neural networks, deep neural networks, and deep learning.

[0031] A neural network consists of layers of interconnected nodes. Generally, a node (or neuron) is where computations are performed. For example, a node may combine input data with a set of weights (or coefficients) that either amplify or attenuate the input data. Thus, the amplification or attenuation of an input signal can be seen as assigning relative importance to various inputs with respect to the task the network is trying to learn. Generally, the input-weight product is added (or accumulated), and this sum is then passed through the node's activation function to determine whether and how far the signal should proceed within the network.

[0018]

[0032] In most basic implementations, a neural network can have an input layer, a hidden layer, and an output layer. "Deep" neural networks generally have two or more hidden layers.

[0019]

[0033] Deep learning is a method of training deep neural networks. Generally, deep learning is sometimes called a "universal approximator" because it can map inputs to a network to outputs from the network, and thus learn to approximate an unknown function f(x)=y between any input x and any output y. In other words, deep learning finds the correct f to transform x into y.

[0020]

[0034] More specifically, deep learning trains each layer of a node based on a different set of features, i.e., the output from the previous layer. Thus, with each successive layer of a deep neural network, the features become more complex. Therefore, deep learning is powerful because it can perform complex tasks such as object recognition by progressively extracting higher-level features from input data by learning to represent the input at a successively higher level of abstraction in each layer, thereby building a useful feature representation of the input data.

[0021]

[0035] For example, when visual data is presented, the first layer of a deep neural network can learn to recognize relatively simple features in the input data, such as edges. In another example, when auditory data is presented, the first layer of a deep neural network can learn to recognize spectral power at specific frequencies in the input data. Then, based on the output of the first layer, the second layer of the deep neural network can learn to recognize combinations of features, such as simple shapes in visual data, or combinations of sounds in auditory data. Subsequently, higher layers can learn to recognize complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or speech phrases. Thus, deep learning architectures can function particularly well when applied to problems with natural hierarchical structures.

[0022] Layered coupling in neural networks

[0036] Neural networks, such as deep neural networks, can be designed using various connectivity patterns between layers.

[0023]

[0037] Figure 1A shows an example of a fully connected neural network 102. In the fully connected neural network 102, each node in the first layer transmits its output to every node in the second layer, and as a result, each node in the second layer receives input from every node in the first layer.

[0024]

[0038] Figure 1B shows an example of a locally connected neural network 104. In the locally connected neural network 104, nodes in the first layer may be connected to a limited number of nodes in the second layer. More generally, the locally connected layers of the locally connected neural network 104 may be configured so that each node in a layer has the same or similar connectivity pattern, but with different connectivity strengths (or weights) (e.g., 110, 112, 114, and 116). Since higher layer nodes in a given region may receive inputs that are tuned through training to the characteristics of a constrained portion of all inputs to the network, the connectivity patterns of local connections may create spatially distinct receptive fields within the higher layers.

[0025]

[0039] One type of locally connected neural network is the convolutional neural network. Figure 1C shows an example of a convolutional neural network 106. The convolutional neural network 106 can be configured such that the connection strengths associated with the input to each node in the second layer are shared (e.g., 108). Convolutional neural networks are suitable for problems where the spatial location of the input is meaningful.

[0026]

[0040] One type of convolutional neural network is the deep convolutional network (DCN). A deep convolutional network is a network of multiple convolutional layers, which can be further constructed using, for example, pooling layers and normalization layers.

[0027]

[0041] Figure 1D shows an embodiment of the DCN100 designed to recognize visual features in an image 126 generated by an image capture device 130. For example, if the image capture device 130 is a camera mounted on a vehicle, the DCN100 can be trained using various supervised learning techniques to identify traffic signs and even numbers on traffic signs. Similarly, the DCN100 can be trained for other tasks, such as identifying lane markings or traffic signals. These are just a few exemplary tasks, and many other tasks are possible.

[0028]

[0042] In this embodiment, DCN100 includes a feature extraction section and a classification section. Upon receiving image 126, the convolutional layer 132 applies a convolutional kernel to image 126 (as illustrated, for example, in Figure 2) to generate a first set of feature maps (or intermediate activations) 118. Generally, a “kernel” or “filter” includes a multidimensional array of weights designed to highlight different aspects of the input data channel. In various embodiments, “kernel” and “filter” can be used interchangeably to refer to a set of weights applied in a convolutional neural network.

[0029]

[0043] Next, the first set 118 of feature maps may be subsampled by a pooling layer (e.g., a max pooling layer, not shown) to generate a second set 120 of feature maps. The pooling layer can reduce the size of the first set 118 of feature maps while retaining much of the information in order to improve model performance. For example, the second set 120 of feature maps can be downsampled by the pooling layer from 28×28 to 14×14.

[0030]

[0044] This process can be iterated through many layers. In other words, the second set of feature maps 120 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0031]

[0045] In the embodiment shown in Figure 1D, a second set 120 of feature maps is provided to a fully connected layer 124, which then generates an output feature vector 128. Each feature in the output feature vector 128 may include a number corresponding to a possible feature of image 126, such as "label", "60", and "100". In some cases, a softmax function (not shown) may be used to convert the numbers in the output feature vector 128 into probabilities. In that case, the output 122 of DCN 100 is the probability that image 126 contains one or more features.

[0032]

[0046] The softmax function (not shown) can convert the individual elements of the output feature vector 128 into probabilities such that the output 122 of DCN100 is one or more probabilities that image 126 contains one or more features, such as a marker with the digit "60" on it, as in the input image 126. Therefore, in this embodiment, the probabilities in output 122 for "marker" and "60" should be higher than the probabilities of other elements in output 122 such as "30", "40", "50", "70", "80", "90", and "100".

[0033]

[0047] Before training DCN100, the output 122 generated by DCN100 may be inaccurate. Therefore, an error may be calculated between output 122 and a priori known target output. For example, here the target output is the instruction that image 126 contains the label and the digit "60". Then, using the known target output, the weights of DCN100 can be adjusted throughout training so that subsequent outputs 122 of DCN100 achieve the target output.

[0034]

[0048] To adjust the weights of DCN100, the learning algorithm may calculate a gradient vector for the weights. The gradient can indicate the amount by which the error will increase or decrease if the weights are adjusted in a particular way. The weights may then be adjusted to reduce the error. This method of adjusting weights is sometimes called "backpropagation" because it involves a "backward path" through the layers of DCN100.

[0035]

[0049] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent may be iterated until the achievable error rate for the entire system stops decreasing, or until the error rate reaches a target level.

[0036]

[0050] After training, DCN100 may be presented with new images and can generate inferences such as classification or the probability that various features are present in the new images.

[0037] Convolutional techniques for convolutional neural networks

[0051] Convolution is generally used to extract useful features from an input dataset. For example, in a convolutional neural network as described above, convolution allows for the extraction of different features using kernels and / or filters whose weights are automatically learned during training. The extracted features are then combined to perform inference.

[0038]

[0052] Activation functions can be applied before and / or after each layer of a convolutional neural network. Generally, an activation function is a mathematical function (e.g., an equation) that determines the output of a node in a neural network. Thus, the activation function determines whether a node should pass information based on whether its input is relevant to the model's predictions. In one embodiment, if y = conv(x) (i.e., a convolution of y = x), both x and y can generally be considered "activations." However, for a particular convolution operation, x may be called a "pre-activation" or "input activation" because it exists before the particular convolution, and y may be called an output activation or feature map.

[0039]

[0053] Figure 2 shows an example of conventional convolution where a 12x12 pixel x 3 channel input image is convolved using a 5x5x3 convolution kernel 204 and a stride (or step size) of 1. The resulting feature map 206 is 8x8 pixel x 1 channel. As seen in this example, conventional convolution can vary the number of dimensions of the input data compared to the output data (here, from 12x12 to 8x8 pixels) and the number of channel dimensions (here, from 3 to 1 channel).

[0040]

[0054] One way to reduce the computational load (e.g., measured in floating-point operations per second, FLOPs) and numerical parameters associated with neural networks that have convolutional layers is to factorize the convolutional layers. For example, a spatially separable convolution as shown in Figure 2 may be factorized into two components: (1) a depth-method convolution (e.g., spatial fusion) in which each spatial channel is independently convolved by a depth-method convolution, and (2) a point-by-point convolution (e.g., channel fusion) in which all spatial channels are linearly combined. Examples of depth-separable convolutions are shown in Figures 3A and 3B. Generally, during spatial fusion, the network learns features from the spatial plane, and during channel fusion, the network learns the relationships between these features across channels.

[0041]

[0055] In one embodiment, separable depth-direction convolution can be implemented using a 3x3 kernel for spatial fusion and a 1x1 kernel for channel fusion. Specifically, channel fusion can use a 1x1xd kernel iterated through every single point in the input image at a depth of d, where the kernel depth d generally corresponds to the number of channels in the input image. Channel fusion by point-by-point convolution is useful for dimensionality reduction for efficient computation. By applying a 1x1xd kernel and adding an activation layer after the kernel, additional depth can be given to the network, thereby improving its performance.

[0042]

[0056] Figures 3A and 3B show examples of depth-separable convolution operations.

[0043]

[0057] Specifically, in Figure 3A, the input image 302, which is 12 pixels × 12 pixels × 3 channels, is convolved with a filter comprising three separate kernels 304A to C, each having dimensions of 5 × 5 × 1, to generate an 8 pixel × 8 pixel × 3 channel feature map 306, where each channel is generated by an individual kernel among 304A to C.

[0044]

[0058] Next, the feature map 306 is further convolved using a point-by-point convolution operation with kernel 308 (e.g., kernel) having dimensions of 1 × 1 × 3 to generate an 8-pixel × 8-pixel × 1-channel feature map 310. As shown in this example, the feature map 310 has a reduced dimensionality (1 channel vs. 3), which enables more efficient computation using the feature map 310.

[0045]

[0059] The results of depth-separable convolution in Figures 3A and 3B are substantially the same as those of conventional convolution in Figure 2, but the number of calculations is significantly reduced. Therefore, depth-separable convolution provides a significant efficiency gain when the network design allows it.

[0046]

[0060] Although not shown in Figure 3B, the channel dimension of the convolutional output can be increased by using multiple (e.g., m) point-by-point convolution kernels 308 (e.g., individual components of the filter). For example, m = 256 1 × 1 × 3 kernels 308 can be generated, each outputting an 8-pixel × 8-pixel × 1-channel feature map (e.g., 310), and these feature maps can be stacked to obtain a resulting 8-pixel × 8-pixel × 256-channel feature map. The resulting increase in channel dimension provides more parameters for training, thereby improving the ability of the convolutional neural network to identify features (e.g., in the input image 302).

[0047] Exemplary In-Memory Computation (CIM) Architecture

[0061] Figure 4 shows an exemplary in-memory computation (CIM) array 400 configured to perform machine learning model computations according to an aspect of this disclosure. In this embodiment, the CIM array 400 is configured to simulate MAC operations using mixed analog / digital operations for an artificial neural network. Thus, as used herein, the terms multiplication and addition may refer to such simulated operations. The CIM array 400 can be used to implement aspects of the processing techniques described herein.

[0048]

[0062] In the illustrated embodiment, the CIM array 400 includes precharge word lines (PCWL) 425a, 425b, and 425c (collectively 425), read word lines (RWL) 427a, 427b, and 427c (collectively 427), analog-to-digital converters (ADCs) 410a, 410b, and 410c (collectively 410), a digital processing unit 413, bit lines 418a, 418b, and 418c (collectively 418), PMOS transistors 411a to 111i (collectively 411), NMOS transistors 413a to 413i (collectively 413), and capacitors 423a to 423i (collectively 423).

[0049]

[0063] The weights associated with the neural network layers can be stored in the SRAM cells of the CIM array 400. In this embodiment, the binary weights are shown in the SRAM bit cells 405a-405i of the CIM array 400. Input activations (which may be input values, e.g., input vectors) are provided on PCWL425a-c.

[0050]

[0064] The multiplication is performed in each bit cell 405a-405i of the CIM array 400 associated with the bit line, and the cumulative (sum) of all bit cell multiplication results is performed on the same bit line for one column. The multiplication in each bit cell 405a-405i is in the form of an operation equivalent to the AND operation of the corresponding activation and weight, and the result is stored as a charge on the corresponding capacitor 423. For example, a product of 1, and therefore a charge on capacitor 423, is generated only when the activation is 1 (here, since a PMOS is used, the PCWL is 0 for an activation of 1) and the weight is 1.

[0051]

[0065] For example, during the accumulation phase, RWL427 is switched high so that any charge on capacitor 423 (based on the corresponding bit cell (weight) and PCWL (activation) value) can be accumulated on the corresponding bit line 418. The voltage value of the accumulated charge is then converted to a digital value by ADC410 (for example, the output value may be a binary value indicating whether the total charge is greater than the reference voltage). These digital values ​​(outputs) can be provided as input to another aspect of the machine learning model, such as the next layer.

[0052]

[0066] When the activations on the precharge word lines (PCWLs) 425a, 425b, and 425c are, for example, 1, 0, and 1, the sums of the bit lines 418a~c correspond to 0+0+1=1, 1+0+0=1, and 1+0+1=2, respectively. The outputs of ADCs 410a, 410b, and 410c are passed to the digital processing unit 413 for further processing. For example, if CIM100 is processing multi-bit weight values, the digital outputs of ADC110 can be summed to produce the final output.

[0053]

[0067] An exemplary 3x3 CIM circuit 400 can be used, for example, to perform efficient 3-channel convolution for a 3-element kernel (or filter), where the weights of each kernel correspond to each element of the three columns, and as a result, for a given 3-element receptive field (or input data patch), the output of each of the three channels is computed in parallel.

[0054]

[0068] In particular, Figure 4 illustrates an example of a CIM using SRAM cells, but other memory types can be used. For example, in other embodiments, dynamic random access memory (DRAM), magnetoresistive random-access memory (MRAM), and resistive random-access memory (ReRAM or RRAM®) can be used similarly.

[0055]

[0069] Figure 5A shows additional details of an exemplary bit cell 500.

[0056]

[0070] The embodiment in Figure 5A may be an example of the embodiment in Figure 4, or may be related thereto in another way. In the paritucular, bit line 521 is the same as bit line 418a, capacitor 523 is the same as capacitor 423 in Figure 4, read word line 527 is the same as read word line 427a in Figure 4, precharge word line 525 is the same as precharge word line 425a in Figure 4, PMOS transistor 511 is the same as PMOS transistor 411a in Figure 1, and NMOS transistor 513 is the same as NMOS transistor 413 in Figure 1.

[0057]

[0071] Bit cell 500 includes a static random access memory (SRAM) cell 501 (which may represent the SRAM bit cell 405a in Figure 4), as well as transistors 511 (e.g., a PMOS transistor) and 513 (e.g., an NMOS transistor) and a capacitor 523 coupled to ground. Although a PMOS transistor is used for transistor 511, other transistors (e.g., NMOS transistors) can be used instead of the PMOS transistor, along with corresponding adjustments (e.g., inversions) of their respective control signals. The same applies to other transistors described herein. Additional transistors 511 and 513 are included to implement the in-memory computing array according to an aspect of this disclosure. In one aspect, SRAM cell 501 is a conventional six-transistor (6T) SRAM cell.

[0058]

[0072] Weight programming in a bit cell may be performed once for a large number of activations. For example, during operation, SRAM cell 501 receives only one bit of information at nodes 517 and 519 via the write word line (WWL) 516. For example, during writing (when WWL 216 is high), if the write bit line (WBL) 229 is high (e.g., "1"), node 217 is set high and node 219 is set low (e.g., "0"), or if WBL 229 is low, node 217 is set low and node 219 is set high. Conversely, during writing (when WWL 216 is high), if the write bit bar line (WBBL) 231 is high, node 217 is set low and node 219 is set high, or if WBBL 229 is low, node 217 is set high and node 219 is set low.

[0059]

[0073] Following the programming of the weights, activation inputs and multiplication steps may follow to charge the capacitors according to the corresponding product. For example, transistor 511 is activated by an activation signal (PCWL signal) via the precharge word line (PCWL) 525 of the in-memory calculation array to perform the multiplication step. Next, transistor 513 is activated by a signal via another word line (e.g., read word line (RWL) 527) of the in-memory calculation array to perform the accumulation of the multiplied value from bit cell 500 with other bit cells in the array, as described above with respect to Figure 4.

[0060]

[0074] If node 517 is "0" (for example, when the stored weight value is "0"), and the low PCWL indicates a "1" activation at the gate of transistor 511, capacitor 523 is not charged. Therefore, no charge is supplied to bit line 521. However, if node 517 corresponding to the weight value is "1" and PCWL is set low (for example, when the activation input is high), PMOS transistor 511 is turned on, thereby acting as a short circuit and allowing capacitor 523 to be charged. After capacitor 523 is charged, transistor 511 is turned off, and as a result, charge is stored in capacitor 523. To move the charge from capacitor 523 to bit line 521, NMOS transistor 513 is turned on by RWL 527, causing NMOS transistor 513 to act as a short circuit.

[0061]

[0075] Table 1 shows an example of an in-memory computation array operation according to an AND operation setting, which can be implemented by bit cell 500 in Figure 5A.

[0062] [Table 1]

[0063]

[0076] The first column (Activation) in Table 1 contains the possible values ​​of the input activation signal.

[0064]

[0077] The second column (PCWL) of Table 1 contains the PCWL value for activating a transistor designed to implement an in-memory computing function according to an aspect of this disclosure. In this example, since transistor 511 is a PMOS transistor, the PCWL value is the reciprocal of the activation value. For example, an in-memory computing array includes transistor 511 which is activated by an activation signal (PCWL signal) via a precharge word line (PCWL) 525.

[0065]

[0078] The third column (cell nodes) in Table 1 contains weight values ​​stored in the SRAM cell nodes that correspond to weights in a weight tensor, which can be used, for example, in a convolution operation.

[0066]

[0079] The fourth column (Capacitor Nodes) in Table 1 shows the resulting products that are stored as charge on the capacitor. For example, the charge can be stored at the node of capacitor 523 or at one of the nodes of capacitors 423a to 423i. The charge from capacitor 523 is moved to bit line 521 when transistor 513 is activated. For example, referring to transistor 511, when the weight at cell node 517 is "1" (e.g., high voltage) and the input activation is "1" (and therefore PCWL is "0"), capacitor 523 is charged (e.g., the capacitor node is "1"). For all other combinations, the capacitor node has a value of 0.

[0067]

[0080] Figure 5B shows additional details of another exemplary bit cell 550.

[0068]

[0081] Bit cell 550 differs from bit cell 500 in Figure 5A, primarily because it includes an additional precharge word line 552 coupled to an additional transistor 554.

[0069]

[0082] Table 2 shows an example of an in-memory computation array operation similar to Table 1, except that it follows an XNOR operation configuration, which can be implemented by bit cell 550 in Figure 5B.

[0070] [Table 2]

[0071]

[0083] The first column (Activation) in Table 2 contains the possible values ​​of the input activation signal.

[0072]

[0084] The second column (PCWL1) of Table 2 contains the PCWL1 value for activating a transistor designed to implement an in-memory computing function according to an aspect of this disclosure. Here again, transistor 511 is a PMOS transistor, and the PCWL1 value is the reciprocal of the activation value.

[0073]

[0085] The third column (PCWL2) of Table 2 contains PCWL2 values ​​that activate further transistors designed to implement in-memory computing functions, as described in the embodiments of this disclosure.

[0074]

[0086] The fourth column (cell nodes) in Table 2 contains weight values ​​stored in the SRAM cell nodes that correspond to weights in a weight tensor, which can be used, for example, in convolution operations.

[0075]

[0087] The fifth column (capacitor node) in Table 2 shows the resulting products that are accumulated as charge on capacitors such as capacitor 523.

[0076]

[0088] Figure 6 shows an example of the timing of various signals during in-memory computation (CIM) array operations.

[0077]

[0089] In the illustrated embodiment, the first row of the timing diagram 600 shows the precharge word line PCWL (e.g., 425a in Figure 4 or 525 in Figure 5A) which is low. In this embodiment, a low PCWL indicates an activation of "1". The PMOS transistor turns on when PCWL is low, thereby enabling the charging of the capacitor (when the weight is "1"). The second row shows the read word line RWL (e.g., read word line 427a in Figure 4 or 527 in Figure 5A). The third row shows the read bit line RBL (e.g., 418 in Figure 4 or 521 in Figure 5A), the fourth row shows the analog-to-digital converter (ADC) read signal, and the fifth row shows the reset signal.

[0078]

[0090] For example, referring to transistor 511 in Figure 5A, the charge from capacitor 523 is gradually transferred to the read bit line RBL when the read word line RWL is high.

[0079]

[0091] The summed charge / current / voltage (e.g., the charge summed from 403 in Figure 4, or from bit line 521 in Figure 5A) is passed to a comparator or ADC (e.g., ADC 411 in Figure 4), and the summed charge is converted to a digital output (e.g., a digital signal / number). The summing of the charge may be performed in the storage region of timing diagram 600, and the readout from the ADC may be associated with the ADC readout region of timing diagram 600. After the ADC readout is obtained, a reset signal discharges all of the capacitors (e.g., capacitors 423a-423i) in preparation for processing the next set of activation inputs.

[0080]

[0092] The parallel processing techniques of this disclosure can be useful for any type of edge computing involving artificial neural networks. The techniques are applicable at the inference stage or any other stage of neural network processing. The illustrated embodiments are based on binary networks, which can be used when high precision is not required, but the same concepts apply to networks using multi-bit weights.

[0081] Example of a convolution operation in memory

[0093] Figure 7 shows an exemplary convolutional layer architecture 700 implemented by an in-memory computation (CIM) array 708. The convolutional layer architecture 700 may be part of a convolutional neural network (as described above with respect to Figure 1D, for example) and may be designed to process multidimensional data such as tensor data.

[0082]

[0094] In the illustrated embodiment, the input 702 to the convolutional layer architecture 700 has dimensions of 38 (height) × 11 (width) × 1 (depth). The output 704 of the convolutional layer has dimensions of 34 × 10 × 64 and contains 64 output channels corresponding to 64 kernels of a filter tensor 714 applied as part of the convolution process. Furthermore, in this embodiment, each of the 64 kernels of the filter tensor 714 (e.g., exemplary kernel 712) has dimensions of 5 × 2 × 1 (total, the kernels of the filter tensor 714 are equivalent to one 5 × 2 × 64 filter).

[0083]

[0095] During the convolution process, each 5×2×1 kernel is convolved with the input 702 to produce a single 34×10×1 layer of output 704. During the convolution, the 640 weights of the filter tensor 714 (5×2×64) can be stored in an in-memory computation (CIM) array 708 containing columns of each kernel (i.e., 64 columns) in this embodiment. Then, each activation of the 5×2 receptive field (e.g., receptive field input 706) is input to the CIM array 708 using a word line, e.g., 716, and multiplied by the corresponding weight to produce a 1×1×64 output tensor (e.g., output tensor 710). Output tensor 704 represents the accumulation of the 1×1×64 individual output tensors for all of the receptive fields of the input 702 (e.g., receptive field input 706). For simplicity, the CIM array 708 in Figure 7 shows only a few exemplary lines for the inputs and outputs of the CIM array 708.

[0084]

[0096] In the illustrated embodiment, the CIM array 708 includes word lines 716 into which the CIM array 708 receives a receptive field (e.g., a receptive field input 706), as well as bit lines 718 (corresponding to the columns of the CIM array 708). Although not shown, the CIM array 708 may also include precharge word lines (PCWLs) and read word lines RWLs (as described above with respect to Figures 4 and 5).

[0085]

[0097] In this embodiment, word line 716 is used for initial weight definition. However, once the initial weight definition is done, the activation input activates a specially designed line within the CIM bit cell to perform MAC operations. Thus, each intersection of bit line 718 and word line 716 represents a filter weight value, which is multiplied by the input activation on word line 716 to produce a product. The individual products along each bit line 718 are then added to produce the corresponding output value of the output tensor 710. The sum may be charge, current, or voltage. In this embodiment, the dimension of the output tensor 704 after processing the entire input 702 of the convolutional layer is 34 × 10 × 64, but only 64 filter outputs are generated by the CIM array 708. Thus, processing the entire input 702 can be completed in 34 × 10 or 340 cycles.

[0086] CIM architecture for depth separable convolution

[0098] In-memory computation (CIM) based artificial intelligence (AI) hardware accelerators can be used for a variety of tasks, including AI tasks involving image, sensor, and audio processing. CIM can help mitigate the power consumption issues associated with moving data out of memory. In some cases, data movement can consume more power than computation. By using CIM, power savings can be achieved due to the weight-fixing nature of CIM. In other words, weights for neural network computations can be stored in random-access memory (RAM), such as static random-access memory (SRAM) memory cells, allowing computations to be performed in memory and resulting in reduced power consumption.

[0087]

[0099] While vector-matrix multiplication blocks implemented in memory for CIM architectures can generally perform conventional convolutional neural network processing well, they are not efficient for supporting depth-separable convolutional neural networks, which are found in many state-of-the-art machine learning architectures. For example, existing CIM architectures generally cannot perform depth-separable convolutional neural network processing in a single phase because each multidimensional filter uses a different input channel. Therefore, filter weights within the same row may not share the same activation input for different channels. As a result, matrix-matrix multiplication (MxM) architectures are generally required to support depth-separable convolutional processing in a single phase cycle.

[0088]

[0100] Conventional solutions to address this drawback involve adding a separate digital MAC block to handle processing of the depth portion of separable convolutions, whereas a CIM array can handle point-by-point portions of separable convolutions. However, this hybrid approach results in increased data movement, which can offset the memory efficiency advantages of the CIM architecture. Furthermore, hybrid approaches generally involve additional hardware (e.g., digital multiplication and accumulation (DMAC) elements), which increases space and power requirements and increases processing latency. Moreover, the use of DMAC affects the timing of processing operations and can exceed model output timing constraints (or other dependencies). To solve this problem, various compromises may be necessary, such as reducing the frame rate of the input data, increasing the clock rate of processing system elements (including the CIM array), or reducing the input feature size.

[0089]

[0101] The CIM architecture described herein improves the timing performance of processing operations for depth-separable convolution. These improvements are beneficial because they result in shorter cycle times for depth-separable convolution operations and higher total operations per second (TOPS), i.e., TOPS / W, compared to conventional architectures that use more hardware (e.g., DMAC) and / or more data movement.

[0090]

[0102] Figure 8 shows a CIM system 800 including multiple CIM arrays according to several aspects of the present disclosure.

[0091]

[0103] As shown in the figure, the CIM system 800 includes a CIM array 802 configured for depth-direction (DW) convolution and a CIM array 804 configured for point-wise (PW) convolution. In some embodiments, kernels (e.g., 3x3 kernels) can be implemented on different columns of the CIM array 802 in a diagonal manner. For example, kernel 806 can be implemented using CIM cells on columns 810, 812 (e.g., bit lines) and nine rows 814-1, 814-2 to 814-8, and 814-9 (e.g., word lines (WL) collectively referred to as row 814) to implement a 3x3 filter with 2-bit weight parameters. Another kernel 808 can be implemented on columns 816, 818 and nine rows 820-1 to 820-9 (collectively referred to as row 820) to implement another 3x3 filter. Therefore, kernels 806 and 808 are implemented on different rows and columns to facilitate parallel convolution operations for DW. That is, activating the rows and columns of one of kernels 806 or 808 does not affect the rows and columns of the other kernel 806 or 808. Different activation inputs can be provided for each of kernels 806 and 808, allowing them to operate in parallel.

[0092]

[0104] Each kernel's input activation buffer can be filled (for example, stored) with the corresponding output channel patch from the previous layer. For example, a row in kernel 808 (e.g., row 814) may be coupled to activation buffers 830-1, 830-2 to 830-8, and 830-9 (collectively referred to as activation buffer 830), and a row in kernel 806 (e.g., row 820) may be coupled to activation buffers 832-1 to 832-9 (collectively referred to as activation buffer 832).

[0093]

[0105] The outputs of kernel 806 (for example, in columns 810 and 812) may be coupled to analog-to-digital converter (ADC) 840, and the outputs of kernel 808 (for example, in columns 816 and 818) may be coupled to ADC 842. For example, each input of ADC 840 may receive the stored charge in row 814 from each of columns 810 and 812, and each input of ADC 842 may receive the stored charge in row 820 from each of columns 816 and 818, and based on this, ADC 840 and 842 each generate a digital output signal. ADC 840 receives signals from columns 810 and 812 as input and generates a digital representation of the signal, taking into account that the bits stored in column 812 represent lower importance in their respective weights than the bits stored in column 810. Similarly, the ADC842 receives signals from columns 816 and 818 as input and generates a digital representation of the signals, taking into account that the bits stored in column 818 represent lower importance in their respective weights than the bits stored in column 816.

[0094]

[0106] While the ADC840 and 842 are implemented to receive signals from two columns to facilitate analog-to-digital conversion for kernels having two-bit weight parameters, the embodiments described herein can be implemented for ADCs configured to receive signals from any number of columns (e.g., three columns for performing analog-to-digital conversion for kernels having three-bit weight parameters).

[0095]

[0107] The outputs of ADCs 840 and 842 can be coupled to a nonlinear arithmetic circuit 850 (and buffer) to implement nonlinear operations, such as rectified linear units (ReLU) and average pooling (AvePool), to name a few. Nonlinear operations enable the generation of complex mappings between inputs and outputs, and thus enable the learning and modeling of complex data such as images, videos, audio, and datasets that are nonlinear or have a high number of dimensions. The output of the nonlinear arithmetic circuit 850 can be coupled to an input activation buffer 860 for a CIM array 804 configured for PW convolution. As shown in the figure, the output of the CIM array 804 may also be coupled to an ADC 870, and the output of the ADC 870 may be provided to a nonlinear arithmetic circuit 880. Although a single ADC 870 is shown, multiple ADCs may be implemented for different columns of the CIM array 804.

[0096]

[0108] Each kernel 806 and 808 includes two columns that allow 2-bit weights to be stored in each row of the kernel, but kernels 806 and 808 can be implemented using any number of appropriate columns, such as one column for 1-bit binary weights, or two or more columns for multi-bit weights. For example, each kernel 806 and 808 can be implemented using three columns to facilitate the storage of 3-bit weight parameters in each row of the kernel, or using a single column to facilitate the storage of 1-bit weights in each row of the kernel. Furthermore, each kernel 806 and 808 is implemented using nine rows for a 3x3 kernel for ease of understanding, but kernels 806 and 808 can be implemented using any number of rows to implement an appropriate kernel size. In addition, more than two kernels can be implemented using a subset of the cells of the CIM array. For example, the CIM array 802 can contain one or more other kernels, and all kernels of the CIM array 802 are implemented on different rows and columns to facilitate parallel convolution operations. For example, kernel 806 can correspond to kernel 304A as described in Figure 3A, and kernel 808 can correspond to kernel 304B as described in Figure 3A. Another kernel corresponding to kernel 304C (not shown in Figure 8) can also be implemented on different rows and columns than kernels 806 and 808.

[0097]

[0109] Figure 9 shows exemplary operation 900 for signal processing via the CIM system 800 of Figure 8, according to several aspects of the present disclosure. Operation 900 can begin with processing the DW-CNN layer. For example, in block 904, DW convolution weights can be loaded into CIM cells of the CIM array (e.g., for kernels 806, 808) as described herein. For example, in block 904, DW 3×3 kernel weights may be grouped and written to the CIM array 802 of Figure 8. That is, 2-bit kernel weights may be provided in columns 810, 812, and pass gate switches of memory cells (e.g., memory cells b01 and b11 shown in Figure 8) may be closed to store the 2-bit kernel weights in the memory cells. Filter weights can be stored in a similar manner in each row of the CIM cell for each of kernels 806, 808.

[0098]

[0110] Weights that may have been previously stored in memory cells on the same column but in a different row than the active kernel can be set to zero. For example, logical 0 can be stored in memory cells in columns 816, 818 and row 820 (not shown), as well as in memory cells in columns 810, 812 and row 814. In some cases, the CIM array 802 may be initially set to zero before storing the weights of kernels 806 and 808.

[0099]

[0111] In some implementations, the CIM array can be divided into tiles. For example, tiles on the same column as the active kernel can be configured in tristate mode. In tristate mode, the memory cell outputs of the tiles can be configured to have relatively high impedance, effectively eliminating the influence of cells on the output. As described herein, DW convolution kernels in different columns and rows may be stacked. Both DW and PW convolution weights can be updated for each of the subsequent layers.

[0100]

[0112] In block 906, the DW convolution activation inputs (for example, in activation buffers 830 and 832) can be applied to each group of rows in kernels 806 and 808 during the same cycle, allowing both kernels to generate DW convolution outputs in parallel.

[0101]

[0113] In block 908, ADCs 840 and 842 can convert the convolution outputs of kernels 806 and 808 (for example, in columns 810, 812 and columns 816 and 818) from the analog domain to the digital domain. Based on the outputs of ADCs 840 and 842 for DW convolution, nonlinear calculations may be performed via the nonlinear calculation circuit 850.

[0102]

[0114] In block 910, the output from the nonlinear arithmetic circuit 850 can be applied to the PW input of the CIM array 804 (for example, stored in the input activation buffer 860) to perform PW convolution. In block 912, the ADC 870 can convert the PW convolution output from the CIM array 804 from the analog domain to the digital domain. Based on the output of the ADC 870 for PW convolution, a nonlinear arithmetic operation may be performed via the nonlinear arithmetic circuit 880.

[0103]

[0115] By implementing kernels on different rows and columns, convolution operations can be performed in parallel, facilitating faster processing times and lower dynamic power compared to conventional implementations. In other words, performing parallel convolution operations allows for processing multiple filters in a single cycle, as opposed to processing each filter in different cycles, saving processing time and reducing dynamic power. In some embodiments, each kernel can be iterated multiple times to increase row utilization and reduce ADC range compression, as will be described in more detail herein.

[0104] Technology to reduce power consumption and increase CIM array utilization.

[0116] Figure 10 shows a CIM array 1000 divided into tiles (also called subbanks) to conserve power, according to several embodiments of the present disclosure. In one embodiment, the CIM array 1000 may have 1024 rows and 256 columns. Individual tiles (e.g., subbanks) of rows and columns can be enabled or disabled. For example, a tile may contain 128 rows and 23 columns. In one embodiment, a tile array 1002 (including multiple tiles such as tile 1004) may be active for DW-CNN convolution, while the remaining tiles can be disabled. In other words, the remaining tiles may be configured in tristate mode.

[0105]

[0117] In some implementations, row and column filler cells can be implemented within the CIM array 1000. Filler circuits (e.g., buffers or switches) can be used to enable or disable tiles in the CIM array and save power. Column filler cells may be AND gate logic, and row filler cells may be buffers on the write bit line (WBL) and transmission switches on the read bit line (RBL). The size and type of transmission switches can be configured based on linearity specifications.

[0106]

[0118] DW convolution can use relatively small kernel dimensions (3x3, 5x5, ...), and insufficient utilization of the CIM array can affect the signal-to-noise ratio (SNR) due to range compression (e.g., the output of the neural network is dispersed within a small range due to nonlinear activation). Some aspects of this disclosure concern techniques for improving SNR. For example, fine-grained tiling designs can be used to mitigate the impact on SNR, as will be described in more detail herein with respect to Figure 11.

[0107]

[0119] Figure 11 shows a CIM array 802 having diagonally stacked kernels according to several aspects of the present disclosure. Various diagonally stacked kernels can be implemented in the CIM array 802. For example, the CIM array 802 may include CIM cells for kernels 806 and 808, as well as CIM cells for kernels 1108, 1110, 1112, 1114, and 1116, each implemented on different rows and columns of the CIM array 802, as described with respect to Figure 8. As shown, the CIM array 802 can be divided into tiles such as tiles 1104, 1106, etc. Each tile of the CIM array that does not contain at least a portion of a kernel (e.g., tile 1106) can be deactivated to save power.

[0108]

[0120] In some embodiments, a relatively small tile size can be used (e.g., selected during chip design) for a CIM bank performing DW convolution (e.g., 16 rows and 32 columns) to increase CIM array utilization and save power. By using a smaller tile size, the utilization of active CIM cells, which are cells that are not part of the disabled tiles, can be increased.

[0109]

[0121] As one embodiment, three CIM cell groups can be designed to perform the inverse bottleneck of a neural network architecture. The inverse bottleneck operation generally refers to an operation used to extend the input features, followed by DW output dimensionality reduction via DW convolution and PW convolution. The first CIM cell group (CIM1) can be used for the bottleneck operation, the second CIM cell group (CIM2) can be used for the DW convolution operation, and the third CIM cell group (CIM3) can be used for the bottleneck operation. In some embodiments, CIM2 for DW convolution may have a finer tiling configuration (e.g., 16 rows to implement a 3x3 kernel, or 32 rows to implement a 5x5 kernel) to improve CIM array utilization and save power, while CIM1 and CIM3 may have coarser tiling (e.g., 64 or 128 rows) to avoid the impact of filler cells on non-DW convolution operations (e.g., using smaller tiles in the CIM array results in a larger number of filler cells in the CIM array). In this way, the reusability of the CIM array library can be doubled for both DW and non-DW operations.

[0110]

[0122] As an example, the average (e.g., approximate) CIM utilization rate with coarse-grained tiling (e.g., using 64 rows and 32 columns of a CIM array where each tile has 1024 rows) may be 13.08% for a 3x3 kernel and 31.44% for a 5x5 kernel. In other words, only 13.08% of the active memory cells in the CIM array can be used for the 3x3 kernel, and 31.44% can be used for the 5x5 kernel. On the other hand, the average CIM utilization rate with fine-grained tiling using 16 rows and 32 columns per tile, and with a CIM array having 1024 rows, may be 40.46% for a 3x3 kernel and 47.64% for a 5x5 kernel. The average CIM utilization rate using fine-grained tiling with 32 rows and 32 columns per tile for a CIM array with 1024 rows can be 24.18% for a 3x3 kernel and 47.64% for a 5x5 kernel. Therefore, finer tiling improves CIM array utilization for filters with smaller kernel sizes (e.g., for DW convolutions). By improving CIM array utilization, the proportion of active memory cells used increases, reducing power loss caused by unused active memory cells.

[0111]

[0123] In some embodiments, utilization can be improved by selecting a tiling size closer to the kernel size. For example, as shown in Figure 11, a tile size of 16 (e.g., as shown for tile 1104) can be used for a kernel size of 9 (e.g., 9 rows as shown for kernel 806). The tile size may be a power of 2 (logarithmic scale) larger than the kernel size to improve flexibility for handling different neural network models. In some embodiments, the kernel can be iterated to improve row utilization and ADC SNR, as will be explained in more detail with respect to Figure 12.

[0112]

[0124] Figure 12 shows a CIM array implemented using an iterated kernel according to several aspects of this disclosure.

[0113]

[0125] As illustrated, multiple kernels can be iterated to form a kernel group. For example, multiple kernels can be implemented on the same column, such as kernels 806, 1204, or kernels 808, 1208. The same weight parameters can be stored in the iterated kernels of the kernel group on the same column (e.g., kernels 806, 1204), and the same activation inputs can be provided to the iterated kernels. Thus, the iterated kernels can generate the same output signals combined in each column (output), resulting in an increased dynamic range at the output of the iterated kernels. For example, using two iterated kernels can double the dynamic range at the output of the iterated kernels provided to an ADC (e.g., ADC840). Increasing the dynamic range at the kernel output allows for the use of a wider range of the ADC, facilitating higher-precision analog-to-digital conversion. In other words, using the full range of the ADC input allows the ADC's digital output to more accurately identify the ADC's analog input, improving the ADC's SNR.

[0114]

[0126] In some cases, the number of DW convolution channels that can be implemented in a CIM array may be limited by the dimensions of the CIM array. For example, when implementing a 3x3 filter, 113 channels can be implemented in a CIM array with 1024 rows (for example, because 113x9 is less than 1024). In other words, a DW kernel for DW convolution may not fit a single CIM array due to limitations on the number of rows or columns associated with the CIM array. Therefore, input activations and DW convolution weights can be configured by the sequencer so that a partial DW convolution channel sum can be calculated.

[0115]

[0127] In some cases, the maximum number of kernels that can be implemented in a CIM array may be less than the total number of kernels for all channels. The maximum number of kernels that can be implemented in a CIM array is... Then, all corresponding channel inputs can be processed to generate partial channel outputs. Then, the next batch of kernels can be loaded into the array, and the partial outputs can be processed until all kernels have been processed. In another embodiment, the DW convolution input batch size can be determined based on the dimensional information of the subsequent PW layer. Kernel can be loaded multiple times to process the input batch size. Then, the partial DW output can be fed to the next PW convolution layer to generate a partial bottleneck output.

[0116] Exemplary operation for performing neural network processing in a CIM array

[0128] Figure 13 is a flowchart illustrating exemplary operation 1300 for signal processing in a neural network according to several aspects of this disclosure. Operation 1300 may be performed by a controller, such as a CIM controller 1432, as described with respect to Figure 14.

[0117]

[0129] Operation 1300 begins in block 1305 by the controller loading a first set of in-memory computation (CIM) cells having a first set of weight parameters of a first kernel (e.g., kernel 806 in Figure 8) via one or more first columns (e.g., 810, 812 in Figure 8) in order to perform a neural network computation (e.g., DW neural network computation), the first set of CIM cells having one or more first columns and a first set of rows (e.g., row 814 in Figure 8) of a CIM array (e.g., CIM array 802 in Figure 8). In block 1310, the controller loads a second set of CIM cells having a second set of weight parameters of a second kernel (e.g., kernel 808 in Figure 8) via one or more second columns (e.g., columns 816, 818 in Figure 8) in order to perform a neural network computation, wherein the second set of CIM cells has one or more second columns and a second set of rows (e.g., row 820 in Figure 8) of a CIM array. For example, the first set of CIM cells may include a subset of cells in a CIM array, and the second set of CIM cells may include another subset of cells in a CIM array. In some embodiments, one or more first columns may differ from one or more second columns, and the first set of rows may differ from the second set of rows. In block 1315, the controller can perform a neural network computation by applying a first activation input to the first set of rows and a second activation input to the second set of rows.

[0118]

[0130] In some embodiments, operation 1300 may also include loading a third set of weights of the third kernel into another CIM array (e.g., CIM array 804 in Figure 8) to perform point-by-point neural network computations. The controller may also generate an input signal to a second CIM array (e.g., provided via the input activation buffer 860 in Figure 8) based on the output signal from the depth neural network computations.

[0119]

[0131] In some embodiments, operation 1300 may also include generating a first digital signal (e.g., via ADC840 in Figure 8) by converting voltages in one or more first columns from the analog domain to the digital domain, and generating a second digital signal (e.g., ADC842 in Figure 8) by converting voltages in one or more second columns from the analog domain to the digital domain. Operation 1300 may also include performing a nonlinear activation operation (e.g., via a nonlinear activation circuit 850) based on the first and second digital signals.

[0120]

[0132] In some embodiments, the kernel can be iterated over to improve CIM array utilization and increase input range compression for the ADC. For example, the controller may also load a first plurality of weight parameters of a third kernel (e.g., kernel 1204 in Figure 12) into a third CIM cell via one or more first columns to perform neural network computations. The third CIM cell may be located on one or more first columns and a third plurality of rows of the CIM array. The controller can perform neural network computations by applying at least a first activation input (e.g., the same activation input provided to the first kernel) to the third plurality of rows. As described herein, each bit of a weight parameter can be stored via a column of the kernel. For example, quantities in one or more first columns may be associated with quantities of one or more bits of each of the first plurality of weight parameters, and quantities in one or more second columns may be associated with quantities of one or more bits of each of the second plurality of weight parameters.

[0121] An exemplary processing system for performing phase-selective convolution

[0133] Figure 14 shows an exemplary electronic device 1400. The electronic device 1400 can be configured to perform the methods described herein, including the operation 1300 described with respect to Figure 13.

[0122]

[0134] The electronic device 1400 includes a central processing unit (CPU) 1402, which in some embodiments may be a multi-core CPU. Instructions executed in the CPU 1402 may be loaded, for example, from program memory associated with the CPU 1402, or from memory 1424.

[0123]

[0135] The electronic device 1400 also includes additional processing blocks tailored to specific functions, such as a graphics processing unit (GPU) 1404, a digital signal processor (DSP) 1406, a neural processing unit (NPU) 1408, a multimedia processing block 1410, and a wireless connectivity processing block 1412. In one implementation, the NPU 1408 is implemented on one or more of the CPU 1402, GPU 1404, and / or DSP 1406.

[0124]

[0136] In some embodiments, the wireless connection processing block 1412 may include components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE®), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth® connectivity, and wireless data transmission standards. The wireless connection processing block 1412 is further connected to one or more antennas 1414 to facilitate wireless communication.

[0125]

[0137] The electronic device 1400 may also include one or more sensor processors 1416 associated with any type of sensor, one or more image signal processors (ISPs) 1418 associated with any type of image sensor, and / or a navigation processor 1420 which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0126]

[0138] The electronic device 1400 may also include one or more input and / or output devices 1422, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone. In some embodiments, one or more of the processors of the electronic device 1400 may be based on the ARM instruction set.

[0127]

[0139] The electronic device 1400 also includes a memory 1424, which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this embodiment, the memory 1424 includes a computer executable component, which may be executed by one or more of the aforementioned processors of the electronic device 1400 or by a CIM controller 1432. For example, the electronic device 1400 may include a CIM circuit 1426, which includes one or more CIM arrays, such as CIM array 802 and CIM array 804, as described herein. The CIM circuit 1426 can be controlled via a CIM controller 1432. For example, in some embodiments, the memory 1424 may include a loading code 1424A (e.g., loading weight parameters into CIM cells) and a computation code 1424B (e.g., performing neural network computations by applying activation inputs). As shown in the figure, the CIM controller 1432 may include a circuit 1428A for loading (e.g., loading weight parameters into CIM cells) and a circuit 1428B for computation (e.g., performing neural network computations by applying activation inputs). The illustrated components and other components not shown may be configured to perform various aspects of the methods described herein.

[0128]

[0140] In some embodiments, such as when the electronic device 1400 is a server device, various components, such as one or more of the multimedia components 1410, wireless connection components 1412, antenna 1414, sensor 1416, ISP 1418, or navigation 1420, can be omitted from the embodiments shown in Figure 14.

[0129] Examples of Implementation

[0141] Embodiment 1. An apparatus for signal processing in a neural network, comprising: a first in-memory computation (CIM) cell configured as a first kernel for neural network computation, wherein a first set of CIM cells includes one or more first columns and a first plurality of rows of a CIM array; and a second set of CIM cells configured as a second kernel for neural network computation, wherein a second set of CIM cells includes one or more second columns and a second plurality of rows of a CIM array, wherein one or more first columns are different from one or more second columns, and the first plurality of rows are different from the second plurality of rows.

[0130]

[0142] Embodiment 2. The apparatus according to Embodiment 1, wherein a first set of CIM cells comprises a subset of cells in a CIM array, and a second set of CIM cells comprises another subset of cells in a CIM array.

[0131]

[0143] Embodiment 3. The apparatus according to Embodiment 1 or 2, wherein the neural network computation includes depth-direction (DW) neural network computation.

[0132]

[0144] Embodiment 4. The apparatus according to Embodiment 3, further comprising another CIM array configured as a third kernel for point-by-point (PW) neural network computation, wherein an input signal to the other CIM array is generated based on an output signal from the CIM array.

[0133]

[0145] Embodiment 5. The apparatus according to any one of Embodiments 1 to 4, further comprising a first analog-to-digital converter (ADC) coupled in one or more first columns and a second ADC coupled in one or more second columns.

[0134]

[0146] Embodiment 6. The apparatus according to Embodiment 5, further comprising a nonlinear activation circuit coupled to the outputs of the first ADC and the second ADC.

[0135]

[0147] Embodiment 7. The apparatus according to any one of embodiments 1 to 6, further comprising a third CIM cell configured as a third kernel for neural network computation, wherein the third CIM cell lies on one or more first columns and a third or more rows of a CIM array.

[0136]

[0148] Embodiment 8. The apparatus according to Embodiment 7, wherein the same weight parameters are configured to be stored in a first set of CIM cells and a third CIM cell.

[0137]

[0149] Embodiment 9. The apparatus according to any one of Embodiments 1 to 8, wherein one or more of a first set of CIM cells on each row of a first plurality of rows are configured to store a first weight parameter, and one or more of a second set of CIM cells on each row of a second plurality of rows are configured to store a second weight parameter.

[0138]

[0150] Embodiment 10. The apparatus according to Embodiment 9, wherein the quantities in one or more first columns are associated with the quantities of one or more bits of a first weight parameter, and the quantities in one or more second columns are associated with the quantities of one or more bits of a second weight parameter.

[0139]

[0151] Embodiment 11. A method for signal processing in a neural network, comprising: loading first plurality of weight parameters of a first kernel into first in-memory computation (CIM) cells via one or more first columns, wherein the first set of CIM cells comprises one or more first columns and first plurality of rows of a CIM array; loading second plurality of weight parameters of a second kernel into a second set of CIM cells via one or more second columns, wherein the second set of CIM cells comprises one or more second columns and second plurality of rows of a CIM array, wherein one or more first columns are different from one or more second columns and first plurality of rows are different from second plurality of rows; and loading first activation inputs into first plurality of rows and second activation inputs into second plurality of rows to perform a neural network computation.

[0140]

[0152] Embodiment 12. The method according to Embodiment 11, wherein a first set of CIM cells comprises a subset of cells in a CIM array, and a second set of CIM cells comprises another subset of cells in a CIM array.

[0141]

[0153] Embodiment 13. The method according to Embodiment 11 or 12, wherein the neural network computation includes depth (DW) neural network computation.

[0142]

[0154] Embodiment 14. The method according to Embodiment 13, further comprising loading a third plurality of weights of a third kernel into another CIM array in order to perform a point-by-point (PW) neural network computation, and generating an input signal to another CIM array based on the output signal from the DW neural network computation.

[0143]

[0155] Embodiment 15. The method according to any one of Embodiments 11 to 14, further comprising generating a first digital signal by converting the voltages in one or more first columns from the analog domain to the digital domain, and generating a second digital signal by converting the voltages in one or more second columns from the analog domain to the digital domain.

[0144]

[0156] Embodiment 16. The method according to Embodiment 15, further comprising performing a nonlinear activation operation based on a first digital signal and a second digital signal.

[0145]

[0157] Embodiment 17. The method according to any one of embodiments 11 to 16, further comprising loading a first plurality of weight parameters of a third kernel into a third CIM cell via one or more first columns, wherein the third CIM cell lies on one or more first columns and a third plurality of rows of memory, and performing the neural network computation further comprises applying a first activation input to the third plurality of rows.

[0146]

[0158] Embodiment 18. The method according to any one of Embodiments 11 to 17, wherein the quantities in one or more first columns are associated with the quantities of one or more bits of each of a first plurality of weight parameters, and the quantities in one or more second columns are associated with the quantities of one or more bits of each of a second plurality of weight parameters.

[0147]

[0159] Embodiment 19. A non-temporary computer-readable medium comprising instructions, when executed by one or more processors of a processing system, causing the processing system to perform a method of signal processing in a neural network, wherein the method involves loading a first plurality of weight parameters of a first kernel into a first in-memory computation (CIM) cell via one or more first columns, the first set of CIM cells comprising one or more first columns and a first plurality of rows of a CIM array, and loading and performing a neural network computation A non-temporary computer-readable medium comprising loading second plurality of weight parameters of a second kernel into a second set of CIM cells via one or more second columns, wherein the second set of CIM cells comprises one or more second columns and second plurality of rows of a CIM array, where one or more first columns are different from one or more second columns, and the first plurality of rows are different from the second plurality of rows, and performing a neural network computation by applying first activation inputs to the first plurality of rows and second activation inputs to the second plurality of rows.

[0148] Additional considerations

[0160] The preceding description is provided to enable any person skilled in the art to practice the various embodiments described herein. The embodiments described herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications of these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made to the function and configuration of the elements described without departing from the scope of this disclosure. Various embodiments may omit, replace, or add various procedures or components as needed. For example, the methods described may be performed in a different order than described, and various steps may be added, omitted, or combined. Also, features described in relation to some embodiments may be combined in relation to some other embodiments. For example, an apparatus may be implemented or a method may be practiced using any number of embodiments described herein. In addition, the scope of this disclosure is intended to encompass, in addition to or in addition to the various embodiments of this disclosure described herein, such apparatuses or methods practiced using other structures, functions, or structures and functions. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0149]

[0161] As used herein, the term “exemplary” means “to serve as an example, case, or illustration.” No embodiment described herein as “exemplary” should be construed as necessarily preferable or advantageous to any other embodiment.

[0150]

[0162] When used herein, the phrase “at least one of” an enumeration of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” includes a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0151]

[0163] As used herein, the term “determining” encompasses a wide range of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or other data structure), and verifying. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, electing, and establishing.

[0152]

[0164] The methods disclosed herein include one or more steps or actions to achieve the method. The method steps and / or actions may be interchanged with one another without departing from the claims. In other words, unless a specific order of steps or actions is specified, the specific order and / or use of the steps and / or actions may be modified without departing from the claims. Furthermore, the various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include, but are not limited to, various hardware and / or software components and / or modules, including circuits, application-specific integrated circuits (ASICs), or processors. Generally, where there are operations shown in the figures, those operations may have corresponding relative means-plus-function components with similar numbering.

[0153]

[0165] The following claims are not limited to the embodiments shown herein, but should be given the full scope consistent with the language of the claims. In the claims, a singular reference to an element means "one or more" unless it is explicitly stated as "one or more". Unless otherwise explicitly stated, the term "several" means one or more. The elements of the claims should not be construed under Section 112(f) of the U.S. Patent Act unless the element is expressly described using the phrase "means of" or, in the case of a method claim, the element is described using the phrase "steps of". All structural and functional equivalents of the elements of various embodiments described throughout this disclosure, which are known to those skilled in the art or will be known thereafter, are expressly incorporated by reference herein and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, whether such disclosure is expressly enumerated in the claims or not. The invention described in the original claims of this application is listed below. [C1] A device, A first set of in-memory computation (CIM) cells configured as a first kernel for neural network computation, comprising a first set of CIM cells having one or more first columns and first multiple rows of a CIM array, A second set of CIM cells configured as a second kernel for the neural network computation, the second set of CIM cells comprising one or more second columns and a second plurality of rows of the CIM array, Equipped with, The one or more first columns are different from the one or more second columns, The aforementioned first set of rows is different from the aforementioned second set of rows. Device. [C2] The apparatus according to C1, wherein the first set of CIM cells comprises a subset of the cells of the CIM array, and the second set of CIM cells comprises another subset of the cells of the CIM array. [C3] The apparatus according to C1, wherein the neural network computation comprises depth-direction (DW) neural network computation. [C4] Further comprising another CIM array configured as a third kernel for point-by-point (PW) neural network computation, The input signal to the other CIM array is generated based on the output signal from the CIM array. The device described in C3. [C5] A first analog-to-digital converter (ADC) coupled to one or more first columns, and a second ADC coupled to one or more second columns, The apparatus described in C1, further comprising the above. [C6] The apparatus according to C5, further comprising a nonlinear activation circuit coupled to the outputs of the first ADC and the second ADC. [C7] The apparatus according to C1, further comprising a third set of CIM cells configured as a third kernel for neural network computation, wherein the third set of CIM cells lies on one or more first columns and a third or more rows of the CIM array. [C8] The apparatus according to C7, configured such that the same weight parameters are stored in the first set of CIM cells and the third set of CIM cells. [C9] One or more of the first set of CIM cells on each row of the first plurality of rows are configured to store a first weight parameter, One or more of the second set of CIM cells on each row of the second set of rows are configured to store the second weight parameter. The apparatus described in C1. [C10] The quantities of one or more first columns are associated with the quantities of one or more bits of the first weight parameter, The quantities of one or more second columns are associated with the quantities of one or more bits of the second weight parameter, The device described in C9. [C11] Loading a first plurality of weight parameters of a first kernel into a first set of in-memory computation (CIM) cells via one or more first columns in order to perform a neural network computation, wherein the first set of CIM cells comprises the one or more first columns and the first plurality of rows of a CIM array. Loading a second plurality of weight parameters of a second kernel into a second set of CIM cells via one or more second columns, wherein the second set of CIM cells comprises one or more second columns and a second plurality of rows of the CIM array, the one or more first columns being different from the one or more second columns, and the first plurality of rows being different from the second plurality of rows, and so on. The neural network computation is performed by applying a first activation input to the first plurality of rows and a second activation input to the second plurality of rows. A method that includes [a certain feature]. [C12] The method according to C11, wherein the first set of CIM cells comprises a subset of cells of the CIM array, and the second set of CIM cells comprises another subset of cells of the CIM array. [C13] The method according to C11, wherein the neural network computation comprises depth (DW) neural network computation. [C14] To perform point-by-point (PW) neural network computation, load the third set of weights of the third kernel into another CIM array, Based on the output signal from the DW neural network calculation, an input signal to the other CIM array is generated. A method of C13 that further includes the following. [C15] A first digital signal is generated by converting the voltages in one or more first columns from the analog domain to the digital domain, A second digital signal is generated by converting the voltage in one or more second columns from the analog domain to the digital domain. A method of C11 that further includes the following. [C16] The method of C15, further comprising performing a nonlinear activation operation based on a first digital signal and the second digital signal. [C17] To perform the neural network computation, further comprising loading the first plurality of weight parameters of the third kernel into a third set of CIM cells via the one or more first columns, wherein the third set of CIM cells lies on the one or more first columns and the third plurality of rows of the memory, and performing the neural network computation further comprises applying the first activation input to the third plurality of rows. Method described in C11. [C18] The quantities of one or more first columns are associated with the quantities of one or more bits of each of the first plurality of weight parameters, The quantities of the one or more second columns are associated with the quantities of one or more bits of each of the second plurality of weight parameters. Method described in C11. [C19] A non-temporary computer-readable medium having instructions that cause the processing system to perform a method, when executed by one or more processors of the processing system, the method is Loading a first plurality of weight parameters of a first kernel into a first set of in-memory computation (CIM) cells via one or more first columns, wherein the first set of CIM cells comprises one or more first columns and first plurality of rows of a CIM array, Loading a second plurality of weight parameters of a second kernel into a second set of CIM cells via one or more second columns, wherein the second set of CIM cells comprises one or more second columns and a second plurality of rows of the CIM array, the one or more first columns being different from the one or more second columns, and the first plurality of rows being different from the second plurality of rows, and so on. The neural network computation is performed by applying a first activation input to the first plurality of rows and a second activation input to the second plurality of rows. Non-temporary computer-readable media.

Claims

1. It is a device, A first set of in-memory computation (CIM) cells configured as a first kernel for depth-separable convolution operations of a convolutional neural network, comprising a first set of CIM cells comprising one or more first columns and first multiple rows of a first CIM array, A second set of CIM cells configured as a second kernel for the DW convolution operation, comprising a second set of CIM cells included in one or more second columns and a second plurality of rows of the first CIM array, Here, The one or more first columns are different from the one or more second columns, The row in the first plurality of rows is different from the row in the second plurality of rows. A second CIM array configured as a third kernel for point-by-point (PW) convolution operations, Here, the input signal to the second CIM array is generated based on the output signal from the first CIM array. A device equipped with the following features.

2. A first analog-to-digital converter (ADC) coupled to one or more first columns, A second ADC coupled to one or more of the second columns, The apparatus according to claim 1, further comprising the following:

3. The apparatus according to claim 2, further comprising a nonlinear activation circuit coupled to the outputs of the first ADC and the second ADC.

4. The apparatus according to claim 1, further comprising a third set of CIM cells configured as a fourth kernel for the DW convolution operation of the convolutional neural network, wherein the third set of CIM cells is located in one or more first columns and a third or more rows of the first CIM array.

5. The apparatus according to claim 4, wherein the apparatus is configured to store the same weight parameters in the first set of CIM cells and the third set of CIM cells.

6. One or more cells from the first set of CIM cells within each row of the first plurality of rows are configured to store a first weight parameter. One or more cells from the second set of CIM cells within each row of the second plurality of rows are configured to store the second weight parameter. The apparatus according to claim 1.

7. The quantities of one or more first columns are associated with the quantities of one or more bits of the first weight parameter, The quantities of one or more second columns are associated with the quantities of one or more bits of the second weight parameter, The apparatus according to claim 6.

8. A method for operating the apparatus described in claim 1, Loading a first set of first weight parameters of the first kernel into a first set of in-memory computation (CIM) cells via one or more first columns in order to perform the DW convolution operation of the convolutional neural network, wherein the first set of CIM cells is comprised of one or more first columns and a first set of rows of the first CIM array, Loading a second set of second weight parameters of a second kernel into a second set of CIM cells via one or more second columns in order to perform the DW convolution operation of the convolutional neural network, wherein the second set of CIM cells comprises one or more second columns and a second set of rows of the first CIM array, the one or more first columns being different from the one or more second columns, and the rows of the first set of rows being different from the rows of the second set of rows, The DW convolution operation of the convolutional neural network is performed by applying a first activation input to the first plurality of rows and a second activation input to the second plurality of rows. To perform the PW convolution operation, the third plurality of weights of the third kernel are loaded into the second CIM array, The input signal to the second CIM array is generated based on the output signal from the DW convolution operation of the convolutional neural network, A method that includes [a certain feature].

9. A first digital signal is generated by converting the voltages in one or more first columns from the analog domain to the digital domain. A second digital signal is generated by converting the voltage in one or more second columns from the analog domain to the digital domain. The method according to claim 8, further comprising:

10. The method according to claim 9, further comprising performing a nonlinear activation operation based on a first digital signal and the second digital signal.

11. The DW convolution operation of the convolutional neural network further comprises loading the first plurality of weight parameters of the fourth kernel into a third set of CIM cells via the one or more first columns, wherein the third set of CIM cells lies in the one or more first columns and third plurality of rows of the first CIM array, and the DW convolution operation of the convolutional neural network further comprises applying the first activation input to the third plurality of rows. The method according to claim 8.

12. The quantities of one or more first columns are associated with the quantities of one or more bits of each of the first plurality of weight parameters, The quantities of the one or more second columns are associated with the quantities of one or more bits of each of the second plurality of weight parameters. The method according to claim 8.

13. A non-temporary computer-readable medium comprising an instruction, when executed by one or more processors of a processing system, that causes the processing system to perform the method according to any one of claims 8 to 12.

Images