Systems and methods for modifying a neural network for binary processing applications

Abstract

In one embodiment, a method of simulating the operation of an artificial neural network on a binary neural network processor includes: receiving a binary input vector for a layer including a probabilistic binary weight matrix; and performing a vector matrix multiplication of the input vector with the probabilistic binary weight matrix to generate a binary output vector, wherein the multiplication result is modified by simulated binary neural processing hardware noise, and wherein the simulation is performed in a forward pass of a training algorithm for a neural network model for the binary neural processing hardware.

Description

[0001] This patent application claims priority to U.S. Non - provisional application No. 17 / 016,130, filed on September 9, 2020, entitled "SYSTEMS AND METHODS FOR MODIFYING NEURAL NETWORKS FOR BINARY PROCESSING APPLICATIONS", and claims priority to U.S. Provisional Patent Application No. 62 / 897,934, filed on September 9, 2019, entitled "SYSTEMS AND METHODS FOR MODIFIED NEURAL NETWORKS FOR COMPUTE - IN - MEMORY APPLICATIONS", which are hereby incorporated by reference in their entirety.

[0002] Background

[0003] This disclosure generally relates to artificial neural networks for in - memory computing (CIM) arrays.

[0004] The use of artificial neural networks (ANNs) continues to grow in number and use. A particularly common type of ANN is the convolutional neural network (CNN), which is an ANN that includes one or more convolutional layers. A convolutional layer processes an input tensor or array of input tensors using one or more kernels (also called filters), which are arrays of weights, and generates an output tensor. Performing inference using a CNN typically requires numerous computations to perform many necessary tensor operations. Specifically, using a CNN requires performing a vast number of matrix multiplications that involve multiplying an array of weights by a corresponding input array (also called an activation). However, the computational and power resources of any device are limited. The resources of some electronic devices, such as, for example, mobile devices and Internet of Things (IoT) devices, are particularly limited compared to desktop or server devices. Additionally, using a conventional von - Neumann computing architecture with separate memory and processor modules requires moving large amounts of data between memory and processor registers, which often forms a bottleneck in today's conventional hardware.

[0005] Overview

[0006] Certain aspects of the present disclosure relate to a method of simulating the operation of an artificial neural network on a binary neural network processor, including: receiving a binary input vector for a layer including a probabilistic binary weight matrix; and performing a vector - matrix multiplication of the input vector and the probabilistic binary weight matrix to generate a binary output vector, wherein the multiplication result is modified by noise of the simulated binary neural processing hardware.

[0007] Certain aspects of the present disclosure relate to an apparatus including a processor and a memory, the apparatus being configured to simulate the operation of an artificial neural network on a binary neural network processor. The simulation includes: receiving a binary input vector for a layer including a probabilistic binary weight matrix; and performing a vector matrix multiplication of the input vector and the probabilistic binary weight matrix to generate a binary output vector, wherein the multiplication result is modified by simulated binary neural processing hardware noise.

[0008] Certain aspects of the present disclosure relate to a non-transitory computer-readable medium storing computer-executable code for the operation of a neural network, including code for simulating the operation of an artificial neural network on a binary neural network processor. The simulation includes: receiving a binary input vector for a layer including a probabilistic binary weight matrix; and performing a vector matrix multiplication of the input vector and the probabilistic binary weight matrix to generate a binary output vector, wherein the multiplication result is modified by simulated binary neural processing hardware noise.

[0009] Additional aspects, advantages, and features of the present disclosure will become apparent upon reading the entire application. Brief Description of the Drawings

[0011] The disclosed embodiments will be described below in conjunction with the accompanying drawings, which are provided for illustration and not limitation of the disclosed embodiments, where like reference numerals denote like elements, and wherein:

[0012] Figure 1 is a simplified schematic diagram of an exemplary electronic computing device.

[0013] Figure 2 Illustrates an exemplary binary CIM system that uses a CIM array (such as, for example, Figure 1 the CIM array).

[0014] Figure 3 is a simplified schematic diagram of an exemplary CIM module (which may correspond to, for example, Figure 1 the CIM module).

[0015] Figure 4 Shows an exemplary algorithm for simulating a process for performing a convolution operation on a CIM chip in a non-shared setting.

[0016] Figure 5 Shows an exemplary algorithm for describing a CIM training procedure from the perspective of an implementation in a deep learning framework.

[0017] Figure 6 is a flowchart of an exemplary process according to an embodiment of the present disclosure.

[0018] Detailed Description

[0019] With reference to the accompanying drawings, several exemplary aspects of the present disclosure are described. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as superior to or better than other aspects.

[0020] A newly emerging architecture that allows for a significant reduction in power consumption is the in-memory computing (CIM) architecture. Some implementations of CIM devices use modified static random access memory (SRAM) cells. Other implementations may use other types of memory cells (e.g., magnetoresistive RAM (MRAM) or resistive RAM (RRAM)). Exemplary electronic computing devices may include single or multiple CIM arrays. In some embodiments, a CIM array may include an array of modified SRAM cells that are programmable to store, for example, weights corresponding to a CNN, where these cells are also configured to perform calculations using received input values. Note that CIM arrays may also be used by neural networks other than CNNs, and in general, CIM arrays are useful for performing any calculations involving matrix multiplication. While conventional electronic computing devices include various digital-domain computing components, CIM electronic computing devices combine digital-domain components, analog-domain components, and converter components for converting between the two. These conversions may introduce noise or other inaccuracies into the calculations of the electronic computing device.

[0021] Binary neural networks (BNNs) using binary activations and weights achieve sufficient accuracy for many artificial intelligence (AI) tasks. As an implementation including analog, CIM arrays are one of the best ways to implement BNNs. Using BNNs allows replacing MAC operations with bitwise operations such as XNOR and analog population count. The XNOR operation is used because performing an XNOR operation on 0 and 1 corresponds to performing a multiplication operation on -1 and +1, respectively. Implementing BNNs in CIM allows for power and time savings because each CIM storage cell is also a computing unit.

[0022] One challenge in using CIM devices for neural network acceleration is caused by matrix multiplication occurring in the analog domain (which is affected by noise). The noise may manifest as a result of process, voltage, and temperature (PVT) variations that take different forms depending on the chip design. Other sources of noise during operations may include drift or jitter of periodic signals. In some implementations, processing a layer may require using an analog-to-digital converter (ADC) to split a matrix-vector multiplication and rejoin its segments, which introduces additional quantization noise. ADCs and digital-to-analog converters (DACs) have a resolution with a limited number of bits, and since the power consumption and chip area required for these converters scale with the resolution they provide, it is preferred to use low-bit converters. However, the lower the resolution, the greater the quantization noise added.

[0023] As mentioned above, the CIM chip contains one or more CIM arrays. Each array is designed to perform binary matrix-vector multiplication - in other words, multiply a binary input vector by a binary weight matrix. The nominal maximum size of the input vector depends on the number of rows in the CIM array, while the nominal maximum size of the output vector depends on the number of columns in the CMI array. This configuration - where the CIM array is large enough to perform the entire matrix-vector multiplication within one computational cycle - can be referred to as a non-shared infinite configuration. However, it is worth noting that if additional measures are taken to split and recombine operations, a neural network layer can implement matrix-vector multiplication for an array larger than the nominally available array. If the number of rows in the CIM array is insufficient, these measures can include horizontally splitting the matrix-vector multiplication into parts, using an ADC to digitize the outputs of these multiplications, and then adding them together. In the case where the number of columns in the CIM array is insufficient, the matrix-vector multiplication can be vertically split (without digitizing the intermediate results with an ADC). A combination of these two drawbacks is also possible. In either of these cases, multiple CIM arrays are used together (in a configuration that can be referred to as a non-shared finite configuration), or a single CIM array can be used multiple times (in a configuration that can be referred to as a shared configuration) to perform the full matrix-vector multiplication.

[0024] For a general-purpose CIM chip, it cannot be assumed that it always has a suitable number of individual CIM arrays available for mapping any neural network architecture. An exemplary approach is to have only one CIM array on the chip and iteratively load and unload neural network weights into and from this array as needed to process parts of the matrix-vector multiplication. Although this approach increases the number of data transfers, this number is still significantly less than the equivalent operations on a fully digital non-CIM device.

[0025] Similar considerations apply to convolutional neural network layers. For a convolutional layer, each entry in the output feature map is calculated in a separate execution of a matrix-vector product along the lines described above. A significant drawback of using a conventionally trained neural network on a binary computing device (such as a CIM device) is that its prediction performance will be significantly degraded because the binary computations of the CIM device are different from the digital domain in which the model has been or was trained. The PVT effects, ADC quantization, and noise correlations due to the reuse of CIM arrays result in an execution environment that is different from that of a digital device, different from CIM device to CIM device, and different between subsequent executions on the same CIM device.

[0026] Accordingly, systems and methods for improving the accuracy of convolutional neural networks on binary systems, such as CIM devices, would be desirable. As further described herein, such methods include accounting for noise effects during the training of neural networks designed for use on binary systems, such that the resulting neural network model is more suitable for operations using a CIM array. In other words, the training process includes simulating the noise effects of the CIM device.

[0027] Figure 1 is a simplified schematic diagram of an exemplary electronic computing device 100. The device 100 includes one or more CIM modules 101, such as the exemplary CIM modules 101(1) and 101(2). The device 100 may also include a processor 102, a memory module 103, and an I / O module 104. These and other components (not shown) of the device 100 may be interconnected by a bus, fiber optic, or other interconnection (not shown). Each CIM module 101 includes a CIM array 105, an input module 106, and an analog-to-digital converter (ADC) module 107, such as the illustrated exemplary CIM array 105(1), input module 106(1), and ADC module 107(1).

[0028] The input module 106 routes inputs from digital components of the device 100, such as, by way of example, the processor 102 or the memory 103, for use by the cells of the CIM array 105. In some embodiments, the input module 106 includes one or more digital-to-analog converters (DACs) to convert digital input signals to analog signals for use by the cells of the CIM array 105. The ADC module 107 converts the analog output of the CIM array 105 to a digital output for providing to digital components of the device 100, such as the processor 102 or the memory 103. The device 100 may be in the form of, for example, a system-on-chip (SoC). The device 100 may include additional components and / or chips (not shown) and itself be a mobile computing device or an IoT device. The CIM array 105 may be used by the device 100 to perform convolutional operations, such as, by way of example, those convolutional operations for neural network inference.

[0029] Figure 2 Illustrates an exemplary binary CIM system 200 using a CIM array 201, such as, by way of example, Figure 1 the CIM array 105(1). An exemplary input 202 to the layer of the convolutional layer system 200 has dimensions of 1 by 38 by 11 (1 (depth) by 38 (height) by 11 (width)). The layer has sixty-four (64) output channels and correspondingly has 64 filters (represented by the exemplary 64-filter tensor 214).

[0030] Each of the 64 filters of the tensor 214 (e.g., the exemplary filter 212) has a size of 1 by 5 by 2 (1x5x2) (collectively, the 64 filters of the tensor 214 form a 64x5x2 tensor). Each 1x5x2 filter (e.g., filter 212) is convolved with the input 202 to generate a 1x34x10 segment of the output 204 (collectively, the output 204 is a tensor of size 64x34x10). Each 1x34x10 segment of the output 204 represents three thousand four hundred (3400) XNOR operations on 340 5x2 receptive fields (such as the exemplary receptive field 206) of the input 202 and the corresponding filters (such as the exemplary filter 212) using a stride of one (1x1).

[0031] Six hundred and forty (640) weights of the filter tensor 214 (64x5x2) can be stored in 640 corresponding cells of the CIM array 201 (which has at least 10 rows and 64 columns). Each column of the CIM array 201 can store ten values of the corresponding filter 212, which essentially transforms the two-dimensional (5x2) filter 212 into a one-dimensional (10x1) vector using any suitable mapping. It should be noted that for simplicity, the CIM array 201 is illustrated in Figure 2 with only a few lines for the input and output respectively representing the rows and columns of the CIM array 201.

[0032] Over three hundred and forty (340) operation cycles, ten (10) values of each of the 340 5x2 receptive fields (e.g., receptive field 206) of the input 202 are input into the CIM array 201 using the word lines 216. It should be noted that each two-dimensional (5x2) receptive field is essentially transformed into a one-dimensional (10x1) vector using the same mapping used for the filters. The values of the receptive fields are XNOR'd with the corresponding weights of all 64 columns. The results of each column are summed and compared with a threshold to determine the binary output of that column, where the results of all columns are aggregated to form a 1x1x64 output tensor (e.g., output tensor 210), which forms part of the output tensor 204 of this layer. The output tensor 204 corresponds to the aggregation of 340 1x1x64 output tensors for 340 corresponding receptive fields (e.g., receptive field 206) of the input 202. It should be noted that in an alternative implementation, the size of the output tensor 204 can be reduced by, for example, using a larger stride, or dilated convolution.

[0033] Figure 3 is the exemplary CIM module 300 (which may correspond to, for example Figure 1Simplified schematic diagram of the CIM module 101). The CIM module 300 includes an input module 302, an ADC module 304, and a CIM array 306. The CIM array 306 includes c columns and r rows of CIM cells 314(1)(1)-314(c)(r), where each CIM cell (i)(j) is configured to store a corresponding weight value and multiply it by the received input value. It should be noted that, as mentioned elsewhere in this article, the multiplication of binary weights and input values can be performed using the XNOR operation. The CIM array 306 can correspond to, for example Figure 1 the CIM array 105 or Figure 2 the CIM array 201. Each input value is received by the rows i of the CIM array 306 and correspondingly by the c CIM cells 314(1)(i)-314(c)(i) of row i from the input module 302 via the corresponding input lines 310(i). Each CIM cell 314 includes a capacitor that is charged when the result of the XNOR operation is 1 and discharged when the result is 0.

[0034] The result of the XNOR operation performed by the CIM cells 314(j)(1)–314(j)(r) of each column j is output to the corresponding read bit line 312(j) (such as, for example, the read bit line 312(1) for column 1). The outputs (in the form of corresponding capacitances) of the CIM cells 314(j)(1)–314(j)(r) of column j are summed by the corresponding bit line 312(j) and provided as an input indicating the overall count to the corresponding ADC 308(j) in the ADC module 304. It is worth noting that due to phenomena such as capacitor mismatch and thermal effects, the relationship between the total voltage and the overall count is affected by noise. Additionally, since the maximum voltage increase across the bit lines is independent of the number of word lines within the array, as the array size increases, a difference of 1 in the overall count corresponds to a smaller voltage difference measured from that bit line. Each ADC 308 also receives a reference voltage VREF for use as a threshold for determining the corresponding output value. It should be noted that an ADC 308 with 1-bit resolution is equivalent to a comparator. However, it should be noted that alternatively, the ADC 308 can be a multi-bit ADC or a variable-resolution ADC. The outputs of the respective ADCs 308 are provided to the digital processing module 313 for further processing, where this further processing can include operations for multi-bit implementation, operations for splitting array processing, or operations corresponding to batch normalization, pooling, or applying non-linearity.

[0035] As mentioned above, analog operations and conversions in CIM devices introduce various noises, such as capacitor variations, thermal noise, and offset noise. Capacitor variations can be fixed per CIM array, but may depend on the specific overall count for the operation. Thermal noise varies for each computation. Offset noise can be fixed per CIM array and is added at each activation. Reusing CIM arrays as in a shared configuration may introduce correlated noises of capacitor variations and offsets. These various noise effects should be accounted for during training in order to generate an accurate model.

[0036] To train a neural network model robust to CIM-specific effects, low-level circuit simulations (e.g., SPICE) of the CIM array are generated to have a low-level noise model, which is abstracted into a high-level differentiable CIM array simulator. The CIM array simulation is then integrated into a CIM chip simulator.

[0037] The low-level circuit simulation includes a CIM array with N word lines and a single bit line. The cell weights are randomly initialized to 0 or 1. Subsequently, all rows are sequentially activated by switching the corresponding word lines such that the XNOR evaluates to 1. For each word line activated in this way, the bit line voltage corresponding to the overall count from 0 to N is read out. Typically, after characterizing the bit line voltage against the overall count, Monte Carlo simulation is used to generate the bit line voltage variations at each individual overall count in [0, N] according to the hardware noise model.

[0038] Based on these low-level simulations, the CIM array noise acting on the noiseless overall count calculation a can be characterized for a specific column as three zero-mean normal distribution sources α, β, and γ, such that

[0039]

[0040] where is the overall count with noise. These noise sources initially affect the computations in the voltage domain. Therefore, the standard deviation σ′ in the voltage domain is translated to the overall count domain by Equation (2) below, where Δv corresponds to the maximum voltage difference across the bit line and N corresponds to the number of rows in the CIM array. ∈

[0041]

[0042] These noise sources are distinguished depending on their origin and how they affect a. First, the distribution of CIM calculations across all generated CIM arrays is random. The randomness is added in the form of process variations during manufacturing. For a given CIM array instantiation, samples from these noise sources can be drawn and may affect subsequent calculations of the array in a deterministic manner. Specifically, α encompasses charge capacitance variations and acts as multiplicative noise depending on the data affecting the overall count calculation. β subsumes additive noise such as offset variations. Second, γ subsumes noise sources such as thermal fluctuations applied to each readout of the columns of the CIM array. The magnitude of these noise sources is determined by the hardware design and the overall assumptions being modeled. It should be noted that in addition to these types of noise sources, replacement implementations may also include noise sources whose characteristics may lie in between, such as, for example, 1 / f noise and non-linearity.

[0043] Figure 4 An exemplary algorithm 400 showing the simulation process for performing a convolution operation on a CIM chip in a non-shared setting — in other words, assuming access to any number of fixed-size arrays. The simulation controls how the input activations and elements of the weight matrix of each layer are routed to the one or more CIM arrays. As described elsewhere in this document, algorithm 400 shows splitting the input (if needed), performing XNOR operations, injecting various types of noise, converting between the voltage domain and the overall count domain, comparing with a threshold (digital or analog, depending on whether the input needs to be split), and outputting the feature map result. It should be noted that the execution of the fully connected layer is similar, but without the additional step of flattening the kernel and each input patch.

[0044] As previously described, depending on the size of the convolutional layer or fully connected layer relative to the CIM array size, the CIM chip operates differently. When the kernel capacity does not exceed the height of the array, the ADC operates as a comparator to directly produce a binary input to the next layer. Alternatively, the matrix-vector operation is split across several CIM arrays, and the partial overall count is digitized using the ADC for summation in the digital domain. Training simulations can be used to trade off between reducing ADC usage and reducing resolution and signal-to-noise ratio (SNR) by attempting different array heights to adjust the design of the CIM chip.

[0045] If the number of output channels exceeds the number of columns in the CIM array, the output channels are mapped to different arrays and the corresponding input tiles are presented to each of these arrays. Since it is assumed that CIM noise is independent across columns, explicit modeling of this horizontal splitting of the kernels is not required in the non-shared scenario. However, in the shared CIM array scenario, Algorithm 400 can be slightly modified as follows. First, after executing one layer, its kernel is unloaded to free the array for execution of subsequent layers. Second, horizontally splitting the kernel requires unloading and loading these different kernel parts and computing all the output feature maps in multiple steps. In the non-shared case, the process variation noises α and β only induce systematic errors in the entries across the feature maps. However, in the shared case, the same column is reused multiple times between layers, between different feature maps (horizontal splitting), and across partial convolutions (when vertically splitting).

[0046] As previously mentioned, an important principle for training neural networks for CIM applications is to expose the neural network to the environment it will be exposed to at test time during training. If the network is trained to perform well across all or a wide range of possible chip distributions during training, it will perform well on test chips drawn from the same distribution as the large number of CIM chips simulated during training, provided that these simulations approximate reality well enough. This also applies to low-bit quantization.

[0047] Some implementations use Probabilistic Binary Neural Networks (PBNN) to train binary layers. Training and using PBNN is described in Peters, J.W. and Welling, M., Probabilistic binary neural networks, arXiv:1809.03368 (2018), which is incorporated herein by reference in its entirety. Essentially, during training, the weights of the PBNN are managed as probability distributions (notably, the probabilities that the weights are 0 or 1). Thus, the network learns the probabilities that the weights have those values, rather than directly learning the specific values of the weights. Embodiments that perform multi-bit quantization on the first layer of the network can use relaxed quantization, as described in Louizos, C., Reisser, M., Blankevoort, T., Gavves, E. and Welling, M., Relaxed quantization for discretized neural networks, arXiv:1810.01875 (2018), which is incorporated herein by reference in its entirety.

[0048] Both methods use the idea of modeling the weights of the network as random variables, which is supported by the possible values that the quantized weights can take at test time. The probabilistic treatment does not directly quantize the weights during training, but allows avoiding the use of a biased straight-through estimator for gradient calculation. Instead, relaxed quantization uses a specific distribution to sample the weights while slowly annealing the variance of the distribution during training. PBNN avoids sampling by directly approximating the result of the linear operation of the layer with probabilistic weights having a Gaussian distribution (central limit theorem). Some embodiments integrating CIM-aware training and quantization-aware training operate only on binary layers. Other embodiments can also operate on multi-bit layers. To train a high-performance binary model for CIM devices, the PBNN model is extended to be CIM-aware.

[0049] Figure 5 An exemplary algorithm 500 is shown that describes the CIM training procedure from the perspective of an implementation in a deep learning framework. During training using PBNN, it is assumed that the weights and activations are in {-1, +1}. Thus, it is necessary to map the Gaussian pre-activation which is the result of the central limit theorem approximation, to the overall count of the XNOR operation performed on the weights w and activations x in the {0, 1} domain. At test time, the transformation between the domains is described by Equation (3) below for kernel j (which corresponds to column j in the CIM array).

[0050]

[0051] At training time, an equivalent Gaussian pre-activation overall count a is obtained by applying the same transformation to the Gaussian pre-activation

[0052]

[0053] After mapping the PBNN calculations to equivalent CIM array calculations, the operation proceeds to integrating the simulated noise model described by Equation (1). During forward propagation, the noise sources α, β, and γ are sampled and applied to the feature map of the Gaussian pre-activation a.

[0054] In one embodiment of this simulation, the scenario where the length of the flattened kernel exceeds the number of rows N in the CIM array can be explicitly modeled. In this case, by quantizing a to 2 between 0 and N bA number of evenly spaced values are used to simulate the ADC. During training, a is a Gaussian random variable and thus, the reparameterization trick can be used to sample from a before rounding, or a probabilistic replacement scheme such as relaxed quantization can be used. It should be noted that combining sampling with the straight-through estimator may be beneficial as it avoids the computationally intensive probabilistic relaxation of relaxed quantization. After all the quantized partial pre-activations are added together, the transformation of Equation (3) is undone and the probability of the stochastic binary activation is binned as the difference from a threshold O. In the case where the CIM array is large enough, the PBNN binarization procedure for the probability mass above the threshold O can be continued.

[0055] Many applications require the input and the weights of the first layer to be multi-bit precision to avoid unacceptable loss of prediction performance. For example, in one implementation, while the hidden layers may have binary inputs and weights, the first layer receives 4-bit inputs and uses 2-bit weights. Depending on various considerations (such as input size, required bit-width, and power distribution of the hardware), the first layer can be performed within the CIM module using multi-bit adaptation for the binary CIM array, or alternatively, by an upstream digital processor (such as, by way of example, Figure 1 the CPU 102) in the digital domain. It should be noted that in some embodiments, these simulations only add noise to the binary layers and thus do not add noise to, for example, the multi-bit first layer. Convolution with the input and kernels quantized to a uniform grid can be implemented as integer multiplication and addition, with appropriate handling of the grid scale and zero offsets. For integer convolution on binary CIM hardware, the integer convolution can be implemented as a weighted sum of binary convolutions, as shown in Equation (5).

[0056]

[0057] On CIM hardware, each binary operation can be performed on the CIM array, digitized using an ADC, and digitally scaled and summed to compute the final result.

[0058] Figure 6 is a flowchart of an exemplary process 600 in accordance with an embodiment of the present disclosure. Process 600 begins with receiving a binary input vector to a layer including a probabilistic binary weight matrix (step 601). The process continues with performing a vector-matrix multiplication of the input vector with the probabilistic binary weight matrix to generate a binary output vector, where the multiplication result is modified by simulated binary neural processing hardware noise (step 602). The process can then be repeated by returning to step 601. Process 600 can be performed by, for example, an electronic computer including a processor and a connected memory.

[0059] As mentioned above, the insights from these simulations can be used to adjust the design of the neural network model for CIM devices. Since an important bottleneck is the number of ADC evaluations required, the number of those required ADC evaluations is reduced by reducing the kernel capacity to be less than or equal to the number of rows Ⅳ in the CIM array (such as Figure 3 the CIM array 306). Operating the ADC as a binary comparator requires less energy and only one DAC evaluation per input slice is needed for the threshold. By normalizing the shape of the convolutional filters for all layers except the first layer, the kernel capacity can be kept no greater than Ⅳ, thus keeping the total number of computations per filter less than Ⅳ. For the first layer, since the input depth is usually 3 or less, it is relatively easy to keep the kernel capacity below Ⅳ, even in the case of large kernels. In some embodiments, an enlarged stride length and / or dilated convolution are used for the first layer to reduce the size of the output. Large kernel sizes and strides can be used early in the network to additionally keep the memory requirements for storing binary feature maps low.

[0060] As described above, by using a simulator that reproduces the relevant characteristics of the hardware, the problem of test-time mismatch during training can be solved. The simulator is differentiable, stochastic, and implemented using a deep learning framework (such as, for example, Pytorch, an open-source machine learning framework developed by Facebook, Inc. of Menlo Park, California), which allows it to be integrated into an accelerated training pipeline. Together with the simulation of the ADC, the correlations due to CIM array reuse are also explicitly considered, as described above. In addition, neural network training procedures can be used to drive the hardware design. It is worth noting that due to the special considerations regarding the input and the first layer, there are multiple ways to accommodate the replacement process of the first layer by the CIM device.

[0061] The above detailed description set forth in conjunction with the accompanying drawings describes examples and does not represent the only examples that can be implemented or that fall within the scope of the claims. The term "example" or "exemplary" as used in this specification means "serving as an example, instance, or illustration" and does not mean "superior to" or "better than other examples". This detailed description includes specific details to provide an understanding of the described technologies. However, these technologies may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described examples.

[0062] It is also noted that the operating steps described in any exemplary aspect herein are described for the purpose of providing examples. The described operations can be performed in many different orders other than the illustrated order. In addition, the operations described in a single operating step can actually be performed in multiple different steps. Additionally, one or more operating steps discussed in the exemplary aspects can be combined. It should be understood that, as will be obvious to those skilled in the art, the operating steps illustrated in the flowcharts can be subject to many different modifications. Those skilled in the art will also understand that any of a variety of different technologies and techniques can be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0063] The various illustrative circuits described in connection with the aspects described herein can be implemented in or with an integrated circuit (IC), such as a processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device. The processor can be a microprocessor, but in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0064] As used herein, the phrase reciting "at least one of" a list of items refers to any combination of those items, including a single member. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, a - b, a - c, b - c, and a - b - c, as well as any combination with multiple identical elements (e.g., a - a, a - a - a, a - a - b, a - a - c, a - b - b, a - c - c, b - b, b - b - b, b - b - c, C - C, and C - C - C, or any other ordering of a, b, and c).

[0065] The present disclosure is provided to enable any person skilled in the art to make or use the aspects of the present disclosure. Various modifications to the present disclosure will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the spirit or scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training an artificial neural network based on the simulated characteristics of a binary neural network processor, comprising: Receiving a binary input vector for a layer of the artificial neural network that includes a probabilistic binary weight matrix; Generating simulated binary neural processing hardware noise based on the characteristic arrangement of the binary neural network processor; Performing a vector matrix multiplication of the input vector and the probabilistic binary weight matrix; Modifying the multiplication result based on the simulated binary neural processing hardware noise to generate a binary output vector; And Forward propagating the binary output vector to one or more other layers of the artificial neural network.

2. The method according to claim 1, wherein, The binary neural network processor is an in-memory computing CIM device.

3. The method according to claim 1, wherein The probabilistic binary weight matrix includes probabilities of weights being 0 or 1.

4. The method according to claim 1, wherein The simulated binary neural processing hardware noise simulates process, voltage, and temperature PVT variations of the simulated binary neural processing hardware.

5. The method according to claim 4, wherein, Thermal noise varies for each multiplication.

6. The method according to claim 4, wherein: The PVT variation noise includes capacitor variation noise; Generating the binary output vector includes calculating an overall count; and The capacitor variation noise depends on the overall count.

7. The method according to claim 1, wherein: Generating the binary output vector includes calculating the overall count with noise ; a is the noise-free overall count; α represents capacitor variation noise; β represents offset variation noise; and γ represents thermal noise.

8. The method according to claim 1, wherein: The layer is larger than the processing array of the binary neural network processor; The binary neural network processor includes an analog-to-digital converter ADC; Performing the vector matrix multiplication includes splitting the layer and generating an intermediate result digitized by the ADC; and The simulated binary neural processing hardware noise includes quantization noise resulting from the ADC digitizing the intermediate result.

9. The method according to claim 1, wherein: Performing the vector matrix multiplication includes using the binary neural network process in a shared configuration; and The simulated binary neural processing hardware noise includes correlated noise for the shared configuration.

10. The method according to claim 1, wherein, The simulated binary neural processing hardware noise is generated from a low-level circuit simulation of the binary neural network processor.

11. The method according to claim 1, wherein: The artificial neural network includes a first layer and a plurality of additional layers; The probabilistic binary weight matrix can correspond to any one of the plurality of additional layers; and The probabilistic binary weight matrix cannot correspond to the first layer.

12. An apparatus, comprising a processor and a memory, the apparatus being configured to train an artificial neural network based on the simulated characteristics of a binary neural network processor, the training including: Receiving a binary input vector for a layer of the artificial neural network that includes a probabilistic binary weight matrix; Generating simulated binary neural processing hardware noise based on the characteristic arrangement of the binary neural network processor; Performing a vector matrix multiplication of the input vector and the probabilistic binary weight matrix; Modifying the multiplication result based on the simulated binary neural processing hardware noise to generate a binary output vector; And Forward propagate the binary output vector to one or more other layers of the artificial neural network.

13. The device according to claim 12, wherein, The binary neural network processor is a in-memory computing (CIM) device.

14. The apparatus according to claim 12, wherein, The probabilistic binary weight matrix includes probabilities of weights being 0 or 1.

15. The device according to claim 12, wherein, The simulated binary neural processing hardware noise simulates process, voltage, and temperature (PVT) variations of the simulated binary neural processing hardware.

16. The device according to claim 15, wherein, The thermal noise varies for each multiplication.

17. The apparatus according to claim 15, wherein: The PVT variation noise includes capacitor variation noise; Generating the binary output vector includes calculating a total count; and The capacitor variation noise depends on the total count.

18. The apparatus according to claim 12, wherein: Generating the binary output vector includes calculating the overall count with noise ; a is the noiseless total count; α represents the capacitor variation noise; β represents the offset variation noise; and γ represents the thermal noise.

19. The apparatus according to claim 12, wherein: The layer is larger than the processing array of the binary neural network processor; The binary neural network processor includes an analog-to-digital converter (ADC); Performing the vector-matrix multiplication includes splitting the layer and generating intermediate results digitized by the ADC; and The simulated binary neural processing hardware noise includes quantization noise resulting from the ADC digitizing the intermediate results.

20. The apparatus according to claim 12, wherein: Performing the vector-matrix multiplication includes using the binary neural network process in a shared configuration; and The simulated binary neural processing hardware noise includes correlated noise for the shared configuration.

21. The device according to claim 12, wherein, The simulated binary neural processing hardware noise is generated from a simulation of the low-level circuitry of the binary neural network processor.

22. The apparatus according to claim 12, wherein: The artificial neural network includes a first layer and a plurality of additional layers; The probabilistic binary weight matrix can correspond to any one of the plurality of additional layers; and The probabilistic binary weight matrix cannot correspond to the first layer.

23. A non-transitory computer-readable medium storing computer-executable code for operations of a neural network, including code for training an artificial neural network based on simulated characteristics of a binary neural network processor, the training including: Receiving a binary input vector for the artificial neural network that includes a probabilistic binary weight matrix layer; Generating simulated binary neural processing hardware noise based on the characteristic arrangement of the binary neural network processor; Performing a vector-matrix multiplication of the input vector and the probabilistic binary weight matrix; Modifying the multiplication result based on the simulated binary neural processing hardware noise to generate a binary output vector; And Forward propagating the binary output vector to one or more other layers of the artificial neural network.

Images