Multi-layer analog in-memory computing device

Abstract

An analog in-memory computing (AIMC) system includes a plurality of tiles. Each tile has a plurality of vertically stacked layers. Each layer includes a crossbar array of resistive memory devices configured to encode a weight matrix. A digital-to-analog converter (DAC) is shared by the plurality of tiles. The DAC is configured to encode an input vector as voltage pulses applied to the crossbar array. An analog-to-digital converter (ADC) is shared by the plurality of tiles and includes a counter register. The ADC is configured to measure induced current on each column of the crossbar array and digitize the induced current to a digital value. A programmable logic controller is configured to: control the ADC to preserve integration values between integrations performed for each layer. Accumulation of partial integration results is performed in place within the tile.

Description

Technical Field

[0001] This disclosure generally relates to computing hardware, and more specifically to multi-layered analog in-memory computing devices. Background Technology

[0002] Analog In-Memory Computing (AIMC) has been identified as a viable alternative to the traditional von Neumann computing paradigm. By performing computation in-place (in memory), it reduces the time and energy costs associated with shuffling data between processing elements and memory, thus enabling more efficient systems.

[0003] The basic component of an AIMC system is a tile. An AIMC tile typically consists of a crossbar array of resistive memory devices, which encodes the matrix elements of the operation. Additionally, a series of digital-to-analog converters (DACs) encode the input vector as voltage pulses applied to the crossbar array. A series of analog-to-digital converters (ADCs) measure the induced current and digitize it.

[0004] AIMC is particularly well-suited for data-intensive workloads such as deep neural network (DNN) inference and other optimization problems. These workloads are primarily dominated by matrix-vector multiplication (MVM) operations. AIMC, due to its weight-stable nature, can be used to perform MVM operations with a time complexity of 𝒪(1) and extremely high power efficiency. By encoding the matrix parameters in the conductance of the stored elements and applying voltage pulses that encode the vectors, we can compute the dot product by measuring the generated current using Ohm's law and Kirchhoff's laws. Summary of the Invention

[0005] According to embodiments of this disclosure, an Analog In-Memory Computing (AIMC) system is disclosed. The AIMC includes a first block. The first block includes two or more stacked layers. A cross-switch array of resistive switching memory devices is included on each layer. The cross-switch array is configured to encode a weight matrix. A digital-to-analog converter (DAC) is coupled to the periphery of the first block. The DAC is configured to encode an input vector as a voltage pulse applied to the cross-switch array. An analog-to-digital converter (ADC) is coupled to the periphery of the first block. The ADC includes a register of counters. The ADC is configured to measure the induced current on each column of the cross-switch array and digitize the induced current into a digital value. A programmable logic controller (PLC) is coupled to the first block, the DAC, and the ADC. The PLC is configured to perform a first matrix-vector multiplication (MVM) integration on a first layer of the first block. A first result is obtained from the first MVM integration performed on the first layer. A second MVM integration is performed on a second layer of the first block. A second result is obtained from the second MVM integration performed on the second layer. The first and second results are accumulated into the accumulated digital value of the first block, which is represented as the counter value in the ADC's register.

[0006] According to embodiments of this disclosure, an Analog In-Memory Computing (AIMC) system is disclosed. The AIMC includes multiple blocks. Multiple vertically stacked layers exist on each block. Each layer includes a cross-switching array of resistive switching memory devices, comprising multiple columns. A digital-to-analog converter (DAC) is shared by the multiple blocks. The DAC is configured to encode an input vector into voltage pulses applied to the cross-switching array. An analog-to-digital converter (ADC) is shared by the multiple blocks and includes a register of counters. The ADC is configured to measure the induced current on each column of the cross-switching array and digitize the induced current into a digital value. A programmable logic controller (PLC) is coupled to the multiple blocks, the DAC, and the ADC. The PLC is configured to: control the ADC to maintain the integral value between integrations performed for each layer; and perform partial accumulation of integration results in situ within the blocks.

[0007] According to embodiments of this disclosure, a programmable logic controller (PLC) is disclosed in an analog in-memory computing (AIMC) system. The PLC includes instructions configured to control analog-to-digital converters (ADCs) coupled to multi-layer blocks to maintain integral values ​​between integrals performed for each layer in the multi-layer blocks. The PLC performs partial integration results in-situ within the blocks.

[0008] The techniques described in this article can be implemented in a variety of ways. Example implementations are provided below with reference to the accompanying figures. Attached Figure Description

[0009] The accompanying drawings are exemplary embodiments. Not all embodiments are shown in the drawings. Other embodiments may be used alternatively or as an alternative. Details that may be obvious or unnecessary may be omitted to save space or for more efficient illustration. Some embodiments may be implemented with additional components or steps and / or without all components or steps shown. When the same numerals appear in different drawings, they refer to the same or similar components or steps.

[0010] Figure 1 This is a diagram of multi-layer blocks in a simulated in-memory computing system according to an exemplary embodiment.

[0011] Figure 2 This is a diagram illustrating a matrix-vector multiplication operation according to an exemplary embodiment.

[0012] Figure 3 This is a block diagram of a simulated in-memory computing system according to an exemplary embodiment.

[0013] Figure 4 This is a schematic diagram of an analog input conversion circuit for an analog in-memory computing system according to an exemplary embodiment.

[0014] Figure 5 This is an illustration of the input activation mode of an analog in-memory computing system according to an exemplary embodiment.

[0015] Figure 6 This is a block diagram of an analog-to-digital conversion system according to an exemplary embodiment.

[0016] Figure 7 This is a block diagram of a post-processing procedure for analog-to-digital conversion bits according to an exemplary embodiment.

[0017] Figure 8 This is a block diagram of the post-processing procedure for analog-to-digital conversion bits using a right-shift method according to an exemplary embodiment.

[0018] Figure 9 This is a block diagram illustrating weight mapping in a matrix-vector multiplication operation according to an exemplary embodiment.

[0019] Figure 10 This is a block diagram illustrating a method for processing weights from cross-switch array elements (from matrices whose input vectors exceed the capacity of the cross-switch array) in a block, according to an exemplary embodiment.

[0020] Figure 11 This is a block diagram illustrating a method for processing weights from cross-switch array elements (from matrices whose output dimensions exceed the capacity of the cross-switch array) in a multi-layer block according to an exemplary embodiment.

[0021] Figure 12This is a block diagram illustrating a method for processing weights from cross-switch array elements (from matrices whose input vectors and output dimensions exceed the capacity of the cross-switch array) in a multi-layer block, according to an exemplary embodiment.

[0022] Figure 13 This is a block diagram of a multi-layer block structure in an analog in-memory computing system according to an exemplary embodiment.

[0023] Figure 14A This is a block diagram illustrating a method for processing matrix-vector multiplication integrals in a multi-layered block within an analog in-memory computing system according to an exemplary embodiment.

[0024] Figure 14B This is a block diagram illustrating a method for processing matrix-vector multiplication integrals in a multi-layered block within an analog in-memory computing system, according to an exemplary embodiment, when some input vectors arrive approximately simultaneously for each block.

[0025] Figure 14C This is a block diagram illustrating a method for processing matrix-vector multiplication integrals in a multi-layered block within an analog in-memory computing system, according to an exemplary embodiment, when partial input vectors arrive at different times for each block.

[0026] Figure 15 This is a block diagram of a multi-layer analog in-memory computing device for utilizing matrix-vector multiplication operations and multi-model operations of large matrices, according to an exemplary embodiment.

[0027] Figure 16 This is a block diagram of a multi-layer analog in-memory computing device incorporating a downsampling stage, according to an exemplary embodiment.

[0028] Figure 17 This is a block diagram of a multi-layer analog in-memory computing device incorporating sliding window processing in an analog-to-digital converter counter module, according to an exemplary embodiment.

[0029] Figure 18 This is a block diagram of a multi-layer analog in-memory computing device incorporating right-shift processing in an analog-to-digital converter counter module, according to an exemplary embodiment.

[0030] Figure 19 This is a block diagram illustrating a convolutional neural network mapped to a multilayer analog in-memory computing device according to an exemplary embodiment. Detailed Implementation

[0031] In the following detailed description, numerous specific details are illustrated by way of example to provide a thorough understanding of the relevant teachings. However, it should be understood that the teachings of the invention can be practiced without these details. In other instances, well-known methods, procedures, components, and / or circuits are described at a relatively high level without detail in order to avoid unnecessarily obscuring aspects of the teachings.

[0032] definition

[0033] As used herein, Analog In-Memory Computing (AIMC) refers to a computing paradigm in which storage devices used in an analog manner are used to encode data and perform some or all of the computations associated with a workload (e.g., a neural network).

[0034] As used herein, an AIMC system refers to a software-operable system that includes analog and possibly digital circuits and performs calculations according to the AIMC paradigm.

[0035] As used herein, a block refers to a non-volatile memory cell in a two- or three-dimensional array, including transistors or other circuitry that control reading from and writing to the non-volatile memory cell. In some embodiments, the transistors / circuitry perform matrix-vector multiplication operations.

[0036] As used in this article, a layer refers to a two-dimensional slice of a block.

[0037] As used herein, a two-dimensional slice refers to a selected level of a three-dimensional memory array. For example, a block-based memory array may have a size of 512×512 and 64 such levels. A two-dimensional slice is one of these levels corresponding to a single 512×512 array.

[0038] As used in this article, a neural network refers to a computational learning system that uses a network of functions to understand a form of data input and transform it into a desired output.

[0039] Overview

[0040] This disclosure generally relates to multi-layer AIMC systems (sometimes referred to as 3D AIMC systems). AIMC systems mitigate the costs, energy, and time associated with shuffling data between processing elements and memory. In-memory analog computing is particularly useful when large amounts of data are present, as classical computers, for example, may require significant overhead and communication costs in memory.

[0041] According to embodiments of this disclosure, an Analog In-Memory Computing (AIMC) system is disclosed. The AIMC includes a first block comprising two or more stacked layers. A cross-switch array of resistive switching memory devices is located on each layer. The cross-switch array is configured to encode a weight matrix. A digital-to-analog converter (DAC) is coupled to the periphery of the block. The DAC is configured to encode an input vector as a voltage pulse applied to the cross-switch array. An analog-to-digital converter (ADC) is coupled to the periphery of the block. The ADC includes a counter. The ADC is configured to measure the induced current on each column of the cross-switch array and digitize the induced current into a digital value contained in the counter. A programmable logic controller (PLC) is coupled to the block, the DAC, and the ADC. The PLC is configured to perform a first MVM integration on a first layer of the first block. A first partial vector result is obtained from the first MVM integration performed on the first layer and is stored in the ADC's counter. A second MVM integration is performed on a second layer of the first block. When the second result is digitized by the ADC, the second result is accumulated with the first result. At the end of the second MVM integration, the counter contains the accumulated result of the two integrations. As will be understood, multiple layers and programmable logic can provide MVM operations with large matrices in-situ within the system. The need to access other computer memory or hardware becomes zero. Therefore, computation time and the use of other computing resources are greatly reduced.

[0042] According to one embodiment that can be combined with one or more prior embodiments, the programmable logic controller is configured to determine whether the size of the weight matrix in the input dimension is greater than the capacity of the crossbar switch array in the input dimension, and to configure the ADC to aggregate the first result with the second result based on the fact that the size of the weight matrix in the input dimension is greater than the capacity of the crossbar switch array in the input dimension. This feature also allows large matrices to be processed locally in a block without having to send portions of the matrix to different hardware components.

[0043] According to one embodiment that can be combined with one or more prior embodiments, the AIMC system includes a second block comprising a third and a fourth layer. The programmable logic controller is further configured to determine whether the size of the weight matrix in the input and output dimensions is greater than the capacity of the crossbar array in the input and output dimensions. Upon determining that the size of the weight matrix in the input and output dimensions is greater than the capacity of the crossbar array in the input and output dimensions, the programmable logic controller determines whether to aggregate or concatenate the MVM integration results from the first and second blocks. This feature considers processing a portion of the results within a block when the matrix is ​​larger than the results that the crossbar array in a layer can process.

[0044] According to one embodiment that can be combined with one or more prior embodiments, after determining that the input vector arrives faster than the time required to perform a single integration, the programmable logic controller (PLC) configures the ADC to aggregate a first result with a second result. The aggregated result is represented as a counter value in the ADC's register. The PLC forwards the counter value to a second block. The PLC resets the counter value in the ADC's register. The PLC performs a third MVM integration at a third layer. The PLC obtains a third result from the third MVM integration performed at the third layer. The PLC performs a fourth MVM integration at a fourth layer. The PLC obtains a fourth result from the fourth MVM integration performed at the fourth layer. The PLC configures the ADC to aggregate the third result with the fourth result. The aggregated result of the third and fourth results is represented as a new counter value in the ADC's register. This feature accelerates the processing of partial results by retaining the counter within a block at the cost of increasing hardware space in the register region and increasing the complexity of the code for the local controller.

[0045] According to one embodiment that can be combined with one or more prior embodiments, the programmable logic controller (PLC) is further configured to load a first input dimension input vector into a DAC register when it is determined that the input vector arrives more slowly than the time required to perform a single integration. The PLC performs a first MVM integration on a first layer of a first block. The PLC obtains a first result from the first MVM integration performed on the first layer. The PLC stores the first result as a first stored counter value in a register of the ADC. The PLC performs a third MVM integration on a third layer of a second block. The PLC obtains a third result. The PLC stores the third result as a second stored counter value in a register of the ADC. The PLC loads a second input dimension input vector into a DAC register. The PLC loads the stored first counter value. The PLC performs a second MVM integration on a second layer of the first block. The PLC stores the second result as a third stored counter value in a register of the ADC. The PLC loads the second stored counter value. The PLC performs a fourth MVM integration on a fourth layer. The programmable logic controller obtains a fourth result from the fourth MVM integral performed at the fourth level. This feature helps handle large matrices that exceed the capacity of the crossbar switch array in both the input and output dimensions by computing a partial result that travels along the input dimension while simultaneously waiting for the vector in the output dimension.

[0046] According to one embodiment that can be combined with one or more prior embodiments, a first final result of a first block is concatenated with a second final result of a second block. It is understood that when processing partial integrations of weight matrices whose vectors in their output dimension exceed the capacity of the cross-switch array, it is not simply a matter of adding the partial results from different layers together. In this case, the device is programmed to concatenate the partial results together, which will represent the accurate result.

[0047] According to one embodiment that can be combined with one or more prior embodiments, the AIMC system also includes a configurable switch coupled to a programmable logic controller. The configurable switch is programmed by the programmable logic controller to select a counter value from a register of a counter used in current MVM integration operation. By changing the design to have multiple counters (a row of counters) and configurable switching to select the counter to be boosted in current integration, the temporary memory is brought closer to the ADC itself, thus mitigating the overhead that context interleaving may cause.

[0048] According to one embodiment that can be combined with one or more prior embodiments, the AIMC system also includes a downsampling module coupled to a programmable logic controller. The downsampling module is configured to reduce the number of voltage pulses at discrete frequencies. In the current context, downsampling can be useful for reducing the number of bits stored in a counter (and thus reducing the space required by the register) at the expense of the accuracy of the result.

[0049] According to one embodiment that can be combined with one or more prior embodiments, the AIMC system also includes a configurable switch coupled to a programmable logic controller. The configurable switch is programmed by the programmable logic controller to select a counter value from a register of a counter used in the current MVM integration operation. A downsampling module is arranged to provide the configurable switch with a number of voltage pulses reduced from the programmed number of input pulses. This feature provides flexibility in the computation mechanism by allowing the controller to select which counter value to use next, thereby mitigating some downtime while waiting for the vector input to arrive. Simultaneously, it reduces the number of bits stored in the counter at the cost of the accuracy of the result.

[0050] According to one embodiment that can be combined with one or more prior embodiments, the programmable logic controller is configured to use bit-serial input encoding with a sliding window procedure in the register of the counter in the ADC. Bit-serial input encoding accelerates the integration operation and, in some cases, increases accuracy. The sliding window method provides an easy way to perform partial result accumulation on each input bit while successfully performing partial result accumulation across layers.

[0051] According to one embodiment that can be combined with one or more prior embodiments, the programmable logic controller is configured to use bit-serialized input encoding with a bit-right shift procedure in the register of the counter in the ADC. Because one bit is discarded for each increment, the right shift method saves register space.

[0052] According to one embodiment that can be combined with one or more prior embodiments, the programmable logic controller is configured to map convolutional layers across multiple blocks. Traditionally, convolutional layers are processed by traveling back and forth between blocks and periphery elements, which consumes significant time and resources. By mapping convolutional layers across multiple layers, processing of the convolutional neural network can be performed in situ, thereby minimizing the back-and-forth forwarding of data between blocks and periphery elements.

[0053] According to embodiments of this disclosure, an Analog In-Memory Computing (AIMC) system is disclosed. The AIMC includes multiple blocks. Multiple vertically stacked layers exist on each block. Each layer includes a cross-switching array of resistive switching memory devices on each layer, the cross-switching array including multiple columns, wherein the cross-switching array is configured to encode a weight matrix. A digital-to-analog converter (DAC) is shared by the multiple blocks. The DAC is configured to encode an input vector as a voltage pulse applied to the cross-switching array. An analog-to-digital converter (ADC) is shared by the multiple blocks and includes a register of a counter. The ADC is configured to measure the induced current on each column of the cross-switching array and digitize the induced current into a digital value. A programmable logic controller (PLC) is coupled to the multiple blocks, the DAC, and the ADC. The PLC is configured to: control the ADC to maintain the integral value between integrations performed for each layer; and perform partial integration result accumulation in situ within the blocks. As will be understood, the multiple layers and the programmable logic enable in-situ MVM operation of the system. The need to access other computer memory or hardware becomes zero. Therefore, computation time and the use of other computing resources are significantly reduced.

[0054] According to one embodiment that can be combined with one or more prior embodiments, the programmable logic controller is configured to aggregate partial integration results of a layer onto the same block. This feature facilitates in-situ processing of large matrices within a block, without having to send portions of the matrix to different hardware components.

[0055] According to one embodiment that can be combined with one or more prior embodiments, the programmable logic controller is configured to connect partial results from the first layer with partial results from the second layer after determining that the output dimension of the weight matrix exceeds the capacity of the crossbar array of the resistive switching memory device. This feature provides accurate results while allowing the integration of vectors that cannot traditionally be easily aggregated together.

[0056] According to one embodiment that can be combined with one or more prior embodiments, the AIMC system also includes a configurable switch coupled to a programmable logic controller. The configurable switch is programmed by the programmable logic controller to select a counter value from a register of a counter used in the current MVM integration operation. A downsampling module is arranged to provide the configurable switch with a number of voltage pulses reduced from the programmed number of input pulses. This feature provides flexibility in the computation scheme by allowing the controller to select which counter value to use next, thereby mitigating some downtime while waiting for vector inputs to arrive. Simultaneously, the number of bits stored in the counter is reduced at the cost of the accuracy of the result.

[0057] According to one embodiment that can be combined with one or more of the foregoing embodiments, the programmable logic controller is configured to use bit-serialized input encoding with a sliding window procedure in the register of the counter in the ADC. The sliding window method provides an easy way to perform partial result accumulation on each input bit while successfully performing partial result accumulation across layers.

[0058] According to one embodiment that can be combined with one or more of the foregoing embodiments, the programmable logic controller is configured to use bit-serialized input encoding with a bit-right shift procedure in the register of the counter in the ADC. Because one bit is discarded for each increment, the right shift method saves register space.

[0059] According to one embodiment that can be combined with one or more prior embodiments, the programmable logic controller is configured to map convolutional layers across multiple blocks. Traditionally, convolutional layers are processed by traveling back and forth between blocks and periphery elements, which consumes significant time and resources. By mapping convolutional layers across multiple layers, processing of the convolutional neural network can be performed in situ, thereby minimizing the back-and-forth forwarding of data between blocks and periphery elements.

[0060] According to embodiments of this disclosure, a programmable logic controller (PLC) is disclosed in an analog in-memory computing (AIMC) system. The PLC includes instructions configured to control analog-to-digital converters (ADCs) coupled to multi-layer blocks to maintain integral values ​​between integrals performed for each layer in the multi-layer block. The PLC performs partial summation of integral results in situ within the block. By retaining integral values ​​between integrals, MVM operations can be performed on matrices exceeding the cross-capacity of blocks / layers. Since the cross-switch array elements share the same peripheral resources, retaining integral values ​​allows the same weighted matrix array to be processed using different layers or blocks without requiring computational hardware outside the AIMC system.

[0061] Example Architecture

[0062] According to embodiments of this disclosure, a programmable logic controller (PLC) is disclosed in an analog in-memory computing (AIMC) system. The PLC includes instructions configured to control analog-to-digital converters (ADCs) coupled to multi-layer blocks to maintain integral values ​​between integrals performed for each layer in the multi-layer blocks. The data flow implemented by the PLC allows partial integration results to be accumulated in situ within the blocks.

[0063] Figure 1 An example representation of a block array 100 in a 3D AIMC device (sometimes referred to as a "multilayer AIMC device") is shown. In the 3D AIMC system, the 3D AIMC block 110 comprises multiple layers 120 of vertically stacked cross-switch arrays that share the same peripheral structures (DAC, ADC, etc.) (their relationship can be seen in...). Figure 3 (As seen in [the previous section]), but typically operates in a mutually exclusive manner. To reiterate, each block 110 has multiple layers 120. At this point, each layer 120 has its own cross-switch array. Therefore, instead of having a single array device, a multi-layer block 110 can have multiple cross-switch arrays stacked on top of each other. However, since each layer 120 shares the same peripheral resources, layers 120 generally cannot operate in parallel. Therefore, even if the ability to operate multiple layers 120 were possible, a multi-layer AIMC device could only operate one layer 120 at a time, depending on the layer 120 currently accessing the peripheral structure. Thus, there is a challenge in terms of parallelism: while there is more physical capacity, there is not more computational parallelism.

[0064] AIMC performs better when handling large amounts of data and can be used, for example, with deep neural networks. An example of an operation accelerated using AIMC is matrix-vector multiplication (MVM). Figure 2 Example matrix-vector multiplication operation 200 is shown. Figure 3 A sample schematic diagram of an AIMC system 300 for processing simulated MVM operations is shown. MVM operations are crucial because they are used in many machine learning operations today. MVMs are often the largest workloads for artificial intelligence and other optimization problems. AIMC can be power-efficient because it does not require data movement. For example, matrix parameters can be used to evaluate data. By encoding matrix parameters in the conductance of stored elements and applying a voltage pulse to the encoded vector, the dot product can be calculated by measuring the resulting current using Ohm's law and Kirchhoff's laws. Resistors are in a cross-switch array 310. We use the voltage pulse as input 315. Using a circuit load, the dot product in the current can be calculated. Applying voltage to some resistors produces an accumulated current. More dot products are obtained, which can be quantized as an MVM as a quantity. The columns in the matrix represent the current as a result of the MVM. The preceding description is a simulation of how MVM operations are generated.

[0065] AIMC system 300 includes one or more blocks 110, which typically comprises three basic elements. One element of block 110 is a cross-switch array 310 of resistive variable memory devices that encodes the matrix elements of the operation. Only a single “layer” of block 110 is shown in the AIMC circuit 300, but it will be understood that multiple layers of cross-switch arrays 310 are vertically stacked on the illustrated cross-switch array 310, each sharing peripheral elements in the following description. Another element in block 110 is a series of digital-to-analog converters (DACs) 320 that encode an input vector 325 into voltage pulses (inputs 315) which are applied to the cross-switch array 310. The third element in block 110 is a series of analog-to-digital converters (ADCs) 330 that measure an induced current 335 and digitize the current into a digital output 340, which can be stored as part of the result value in the ADC counter register 350 in the subject matter. The cross-switch array 330 is typically located in the middle of the block 110 architecture. A series of digital-to-analog converters 320 are provided to receive digital inputs 325 (e.g., numbers as value inputs) to generate voltage pulses (inputs 315). The circuit performs analog conversion. Figure 4 An example of a process according to an exemplary embodiment is shown. Figure 3 The AIMC system 300 shown is an enlarged diagram of an example input mechanism and conversion path for an AIMC system. The analog input is then applied to the crossbar switch array. Figure 5 Two examples of input configurations that can be used in the AIMC system 300 are shown. Input (IN) activation can be represented in two ways (i.e., here, V...). IN The amplitude of * is V DD (or GND). In this subject matter, the AIMC system 300 includes a programmable logic controller 360 configured to control the computation of input 325 according to a weight value matrix via a multi-layered crossbar switch array 310. Embodiments disclose multiple processes for calculating the weight value matrix performed in situ within the AIMC system 300. These embodiments avoid transmitting computational actions outside the AIMC system 300, thereby avoiding back-and-forth computations between the AIMC system 300 and external hardware elements, which increases processing time and consumes external hardware resources of the AIMC system 300.

[0066] In one approach, a bit-parallel configuration is used, employing pulse width modulation (PWM). The memory cell enable duration is proportional to the IN amplitude, and the unit delay depends on the IN bit. In a bit-serial configuration, multi-cycle reads exist, each with a unit delay duration, where the maximum number of pulse cycles is determined by the IN bit. Each cycle has V with data bits 1 and 0 respectively. DD VIN The current generated on the cross-switch array is directed to the analog-to-digital converter (ADC). The ADC takes the current and generates a digital value from it. In a sense, the entire block (or module) is a digital-to-digital process. When a digital value is input, the block produces a digital output; however, simultaneously, two transformations occur, including converting the input from a digital value to an analog value. There exists a matrix of elements that serve as analog values. Multiplication occurs in the analog domain. When a result is produced in the analog domain, the result is converted back to a digital value.

[0067] A major challenge in AIMC systems is often the requirement for peripherals to occupy a similar or even larger area than a crossbar switch array. The peripheral structures within a block (e.g., some digital processing units, DACs, and ADCs) perform numerous conversion processes. These peripheral structures can utilize most of the block area (footprint) and consume the majority of the block's energy to perform these conversions. Typically, the peripheral structures are the primary energy consumer in the block. Therefore, many AIMC architectures attempt to alleviate the stringent area and efficiency constraints by multiplexing the outputs from their peripheral structures. In some approaches, this results in sacrificing the fully parallel operation of the elements themselves. For example, an architecture might use a 256x256 array in a crossbar switch array, thus using 256 ADCs for processing. However, some approaches use fewer ADC converter elements but perform the converters two or three times. Therefore, instead of performing one operation simultaneously, the multiplexing method performs two or three steps, thus sacrificing parallelism for higher efficiency outside the same area used.

[0068] Another key challenge of AIMC is the weight capacity parameter (for deep neural networks). As used in this paper, "weight" refers to the value applied to the input. A "weight" can be a number encoded into, for example, a resistor in a block. When a voltage is applied to a cross-switch array, the output is affected by the weight of that pulse. In AIMC blocks, there is a weight-fixed architecture where all weight values ​​of the network must be in the system (i.e., encoded) before operation. Weights cannot be reloaded as they can be done, for example, on a GPU or CPU. Reloading weights can have high computational overhead. In existing techniques for deep neural networks, tens of millions to billions of weight parameters can be used. Therefore, as can be understood, fitting so many parameters in the same physical space without shifting data is a challenge for weight-fixed systems. The subject matter of this paper addresses this challenge by using 3D (or multi-layer) memory technology in AIMC systems.

[0069] Figure 6An example of an analog-to-digital conversion process 600 for an AIMC system 300, consistent with an exemplary embodiment, is shown. The ADC 330 block is typically the structure governing the efficiency and accuracy of in-memory computation operations. Since the signals within the crossbar switch array 310 are analog, analog conductances exist within the crossbar switch array 310, and weights are encoded in these conductances. In practice, analog voltage pulses are applied to the AIMC system 300, generating analog currents. The ADC 330 block converts the analog currents back to the digital domain, allowing data to be sent to different units and digital computations to be performed on the data. The ADC 330 block typically comprises three stages, and the purpose of the ADC 330 is to provide a linear response to the input. The first stage is typically a sensing stage 610. The sensing stage 610 ensures that a linear value arrives at the ADC with the ideal equivalent analog MVM value of the analog input (IBL) ∝ 612. In the sensing stage 610, the current from the crossbar switch array is received as input analog data 602. In a current-based ADC, the sensing stage 610 includes a current mirror, an OTA (614), etc., and provides intermediate analog converted values ​​to the next stage. The conversion stage 620 converts the intermediate analog values ​​into discrete quantities, which are then fed to the final stage (i.e., the decision stage 630). The magnitude of the current is converted into the frequency of the dataset. The conversion stage 620 also includes a compensation block to address any nonlinearities in the block. In a current-based ADC, the conversion stage includes a current-controlled oscillator (CCO) 624 that receives the mirrored current from the sensing stage 610 and generates spikes or pulses ∝IBL. Discretization occurs in the conversion stage 620. Analog values ​​are used to create a discrete data stream. For example, a current value is taken to create an intermediate representation. The current value is then converted into a pulse train 640. The magnitude of the current is encoded according to the frequency of these pulses. As an example, and not a limitation, one pulse is generated every 10 nanoseconds. Now, if twice the number of pulses are generated, the signal should be generated at twice the frequency (i.e., a new pulse every five nanoseconds). Decision stage 630 counts the number of pulses generated by CCO 624 as a digital output. Decision stage 630 measures how many pulses are received at a given time. To measure how many pulses are received by the previous stage, the decision stage may include an edge detector to understand when pulses transition from high to low and from low to high, and a counter to count how many transitions of interest (e.g., how many low-to-high transitions) were detected using the edge detector.

[0070] Figure 7 Figure 700 illustrates post-processing of ADC bits, consistent with an exemplary embodiment. In bit-serial mode, in each cycle, for example, from LSB (least significant bit) to MSB (most significant bit) to V... INOne bit is provided. In each of these cycles, an 8-bit ADC output is generated and collected in an incrementing counter (INC). For the current illustration, it can be assumed that the size of the incrementing counter is 16 bits and the bits are A15-A0. The significance of the IN bits is typically handled by a set of 8 bits selected for incrementing through shifting. With each step from LSB to MSB, the selected bits used for incrementing are shifted by one (meaning scaling by a factor of x2). For example, for IN (LSB): counter bits A7-A0 are incremented; for IN (LSB + 1): A8-A1 are incremented; and for IN (MSB): A14-A7 are incremented. In the event of overflow, A15 is typically held. As shown, the increment will shift the updated bits from right to left. The number of bits required for further processing is typically much smaller than 16 bits (e.g., in the range of 8 bits for AI applications). In the case of AI, the most significant 8 bits of the counter (A15-A8) are further propagated, and (A7-A0) are discarded.

[0071] Figure 8 Another post-processing example 800 for ADC bits, consistent with the exemplary embodiment, is shown. Figure 8 The post-processing method for the ADC bits shown follows Figure 7 The mechanism described above, but Figure 8 The differences shown focus on reducing the counter size. The IN bit weights are handled by right-shifting and truncating the LSB in each cycle. Using... Figure 8 The process involves reducing the size of the counter from (m+n) bits to essentially (m+1) bits.

[0072] Now for reference Figure 9 Example 900, consistent with an exemplary embodiment, illustrates a cross-connect array mapping weights to blocks. When mapping weights to an AIMC cross-connect array, it is common for the array size to not precisely fit a block. Once the array size capacity is determined for the cross-connect array, the size remains unchanged. If the weight array is smaller than the cross-connect array size, block utilization will decrease, leading to reduced efficiency. However, this situation does not alter the system's data flow. If the weight array is larger than the cross-connect array size, the array can be divided into multiple cross-connect arrays. This splitting introduces processing steps for combining partial results. These processing steps include accumulation and concatenation. Accumulation of partial results is an explicit operation added to the network graph after the network is mapped to blocks. Concatenation can be implicit and is therefore generally less problematic. Therefore, for applications processing MVM, when the weight array exceeds the array size, this subject matter applies the following conditions to process the input:

[0073] If the weight array is larger than the size of the cross-switch array in the input (row) dimension, the layer is split into multiple blocks and some results can be accumulated. Figure 10 This diagram illustrates a scenario where the weight array 1040 (which can be interchangeably referred to as "weight matrix 1040") is larger than the size of the crossbar array in the input (row) dimension. Figure 10 In Example 1000, the crossbar array capacity is 512 elements. Input vector 1020 comprises 1024 elements. Output dimension 1030 is 512 elements (which is the capacity of the crossbar array). Weight matrix 1040 is 1024 by 512 elements. To process input vector 1020, it is split into two (or more) partial input vector portions 1020A and 1020B (which can be split evenly) and sent to different blocks. Partial input vector portions 1020A and 1020B are no larger than the crossbar array capacity in the input dimension. In the example shown, input vector 1020 is split into two 512-element portions, where 512 elements is the capacity (size) of the crossbar array. Because input vector 1020 is larger in the row dimension, weight matrix 1040 is split into upper weight matrix portion 1040A and lower weight matrix portion 1040B. The MVM operation can now be performed using each part split from the original input vector 1020 (partial input vector parts 1020A and 1020B) and the upper weight matrix part 1040A and lower weight matrix part 1040B split from the weight matrix 1040. Each partial result 1050A and 1050B represents the result of the multiplication performed on its corresponding part split from the input vector 1020 and the weight matrix 1040. To obtain the final result, the two partial results 1050A and 1050B are added together to produce an aggregated result 1060 with 512 elements. As can be understood, the aggregated partial results 1050A and 1050B can be considered as adding additional steps to the overall data processing. When processing large amounts of data and / or processing many parts split from the original input vector 1020, the additional steps can lead to significant overhead. For each split performed (e.g., 2, 3, 4, 5, ... n), many accumulations need to be performed, such as... Figure 9 As shown on the left.

[0074] Figure 11 Figure 1100 illustrates a configuration consistent with an exemplary embodiment, where the weight array 1140 is larger than the size of the crossbar switch array in the output (column) dimension 1130. If the array of weight matrices 1140 is larger than the size of the crossbar switch array in the output (column) dimension 1130, the layer is split across multiple blocks, and some results can be concatenated. Figure 11In the example, the crossbar array capacity is 512 elements. The input vector 1120 comprises 512 elements, which can be processed without consideration. However, the output dimension 1130 has 1024 elements (which exceeds the capacity of the crossbar array in the output dimension 1130). The weight matrix 1140 has a size of 512 × 1024 elements. To process the data, the weight matrix 1140 is divided into two (or more) parts 1140A and 1140B (which can be split evenly). The split parts 1140A and 1140B of the weight matrix 1140 are no larger than the capacity of the crossbar array in the output dimension 1130. In the example shown, the weight matrix 1140 is divided into two parts of 512 elements, where 512 elements is the capacity (size) of the crossbar array. MVM operations can be performed using the original input vector 1120 and each part 1140A and 1140B of the weight matrix 1140. The embodiment can first process the first half (part 1140A) of the weight matrix 1140, producing the left half result 1150A. Then, the second half (part 1140B) can be processed, producing the right half result 1150B. The final result 1160 can be generated by concatenating (sponging) the two partial results 1150A and 1150B together. This process is... Figure 9 The right side indicates.

[0075] Figure 12Figure 1200 shows a weight array 1240 that is larger than the size of the crossbar switch array in both column and row dimensions. If the weight array 1240 is larger than the size of the crossbar switch array in both dimensions (input vector 1220 / output 1230 or row / column), the layer can be split across multiple blocks, some partial results can be accumulated, and some other results can be chained. In the example shown, the input vector 1220 is split into two parts, namely partial input vector parts 1220A and 1220B, each with 512 elements. The weight matrix 1240 is split into four partial weight matrix parts 1240A, 1240B, 1240C, and 1240D, each with 512 elements. Although the example shows a combination of four submatrices with input vector parts, it should be understood that the number of submatrices used for any given computation will depend on the size of the weight matrix 1240 ideally split evenly. In the example shown, the partial result 1250A obtained from the partial input vector portion 1220A (the upper half of input vector 1220) and the partial weight matrix portion 1240A (the upper left part of weight matrix 1240 (sub-matrix A)) can be accumulated with the partial result 1250C obtained from the partial input vector portion 1220B (the lower half of input vector 1220) and the partial weight matrix portion 1240C (the lower left part of weight matrix 1240 (sub-matrix C)). The partial result 1250B obtained from the partial input vector portion 1220A (the upper half of input vector 1220) and the partial weight matrix portion 1240B (the upper right part of weight matrix 1240 (sub-matrix B)) can be accumulated with the partial result 1250D obtained from the partial input vector portion 1220B (the lower half of input vector 1220) and the partial weight matrix portion 1240D (the lower right part of weight matrix 1240 (sub-matrix D)). The summation result 1260A associated with submatrices A and C can be concatenated with the summation result 1260B associated with submatrices B and D to generate the final result 1270.

[0076] As can be inferred from the above description of the process, additional communication resources are consumed at any time when computational steps are used across multiple blocks. This reduces efficiency in the AIMC system. Furthermore, latency in the AIMC system increases.

[0077] In a multi-layered system, if layers are mapped to the same block, the resulting partial results should be considered for storage in volatile memory (SRAM) until the last MVM operation in the sequence is executed. If the cell performing the accumulation is not in the immediate vicinity of the block, data can be transmitted to that cell via a communication channel. If the subsequent weight array is also in the same block, data is returned to the vicinity of the block for the next operation to execute.

[0078] This subject matter discloses a multi-layer AIMC block that enables faster and more resource-efficient AIMC systems. When the input data exceeds the capacity of the crossbar switch array, the subject block processes partial results more efficiently. When the block receives input that will be processed by splitting the data into portions used to generate partial results, the block can process the partial results in-place.

[0079] Example block

[0080] Now for reference Figure 13 The diagram illustrates a block architecture 1300 for an AIMC system according to an embodiment. Block 1310 includes a three-dimensional crossbar switch array 1315 (represented by the space within the frame of block 1310) comprising multiple (N) layers 1350 having a crossbar switch array of size KxM. Digital-to-analog converter (DAC) peripheral circuitry 1320 coupled to the crossbar switch array 1315 supports bit-parallel or bit-serial input encoding, or both. Analog-to-digital converter (ADC) circuitry 1330 may be peripheral to the crossbar switch array 1315. The ADC circuitry 1330 may include configurable behavior regarding the converted data stream. Block 1310 also includes a programmable local controller circuitry 1340 (described in more detail below) that enables new data streams. The programmable local controller circuitry 1340 may be an active component; for example, a reduced instruction set computer processor or a passive finite state machine (FSM) with configuration registers. The programmable local controller circuitry 1340 controls how ADC conversions will occur. In addition, the programmable local controller circuit 1340 determines when the process requests parsing the input and calculating a partial result by accumulating (sometimes called aggregation), by concatenating (sometimes called splicing), or a combination of both.

[0081] The illustrated structure 1300 handles a weight matrix 1360 exceeding the capacity of the crossbar array. Consider the proposed block 1310 comprising N layers 1350 with a crossbar array size of C×C. The operation to be performed is an MVM between a vector of size K and an array of size K×M. In this example, the cases of K>C and M≤C are considered, which means that the summation of the computational partial results is used to obtain the final result. To handle the weight matrix 1360, it can be divided into multiple submatrices 1365 prior to execution time, where the size of each submatrix 1365 is within the capacity of the crossbar array. The splitting can be determined a priori before the weights are mapped in the system. The splitting of the input can be handled by the programmable local controller circuitry 1340 as if the splitting occurred during runtime. For this example, the weight matrix 1360 is four times larger than the capacity of the crossbar array in the input dimension.

[0082] In block 1310, four layers 1350 are shown; however, it will be understood that embodiments typically include two or more layers 1350. For in-situ computation of the accumulated results, a corresponding layer 1350 can be mapped to one of the submatrices 1365. For example, for a crossbar switch array with a capacity of 512 elements, the first 512 rows in the bottom submatrix 1365 can be mapped to the bottom layer 1350 (“t=1”), the next 512 rows in the next submatrix 1365 can be mapped to the second layer 1350 (“t=2”), and so on, until the entire array is mapped to layers within the same block 1310. The mapped data may fit the entire block or may be smaller than the capacity of the entire block. Once mapped, MVM operations can be performed on each layer 1350 that produces partial results.

[0083] In block 1310, the following data flow characteristics are enabled by a programmable local controller circuit 1340 with configurable integration behavior. The array is again mapped in the L = ceil(K / C) layer. The programmable local controller circuit 1340 controls L partial result accumulations to occur to obtain the final result. L MVM operations occur sequentially, but the programmable local controller circuit 1340 does not reset the ADC counter between integrations. After L integrations, the final result is in the ADC counter, and the programmable local controller circuit 1340 can move the data for further processing in the next block / block. The counter value is reset for the next set of MVM operations.

[0084] As an illustrative example, and still for reference Figure 13 Input processing can be performed by sequentially calculating matrix multiplications. For example, taking the first 512 rows in the bottom layer 1350, the ADC 1330 can convert the current into a digital value. The digital value can be forwarded to the next block or sent to memory. And the digital value is reset. However, recalling earlier, because the ADC 1330 processing provides a counter value; therefore, there is a usable data stream that can count the number of observed pulses. In one embodiment, the counter function can be modified to retain this value instead of resetting it each time the MVM is performed. For example, after calculating the first MVM, the result is retained. (and...) Figure 6In contrast, where the counter is reset after decision level 630 and before processing restarts at the input of sensing level 610. When performing the second MVM calculation, ADC 1330 starts with the retained value (not zero) calculated for the first MVM calculation and continues to add to that value during the second MVM calculation. Programmable local controller circuitry 1340 can prevent the counter from being reset, such that the counter holds the result Y1 + Y2 when the complete second integration ends. The counter value continues to accumulate for each layer's MVM calculation without resetting until all layers from matrix 1360 have been processed by their respective layers 1350 in block 1310. When performing the final MVM calculation, the counter value represents the final result, which is the accumulation of the MVM calculation for each layer without resetting the counter. The accumulation occurs in situ within block 1310, without moving data around to other blocks and without the need for separate digital processing units. Once the final result is obtained, it can be forwarded to other parts of the computing system. The programmable local controller circuit 1340 can reset the process, reset the counter, and restart a new MVM calculation for a new matrix.

[0085] Example Method

[0086] Figure 14A , 14B Figure 14C illustrates a method for processing a data stream according to an embodiment consistent with the architecture 1300 described above. Unless otherwise indicated, actions can be performed by the programmable local controller circuitry 1340. Generally, the figures illustrate how the programmable local controller circuitry 1340 can be configured to handle different input scenarios. Typically, when MVM operations use partial result accumulation, the operation can be accelerated using block 1310, as described below. Figure 14A As shown, consider a weight array where the row dimension is three times the layer dimension. Figure 14B and 14C In this context, consider a weighted array where the row and column dimensions are three times the layer dimension. According to... Figure 10 and Figure 12 The conclusion can be drawn by summing partial results from the subarrays shown in the same column. The number on each square subarray represents the layer in the block to which the subarray is mapped. In this example, all subarrays are mapped to the same block.

[0087] exist Figure 14A In this array, the matrix (size K×M) is larger than the crossbar array size (C) only in the row (input) dimension. R ×C C ), (K>C R And M≤C CThe data flow and control signals from the programmable local controller circuit 1340 can be as follows. Therefore, only rows are split; not columns. This matrix is ​​mapped to L=ceil(K / C R In the L layers, integration can be performed sequentially on all L layers based on the sequence of partial inputs arriving from the previous block. Partial inputs can appear in any order because accumulation is unaffected. The programmable local controller circuit 1340 retains the value of the ADC counter between L MVM operations to perform in-situ partial result accumulation. After L integrations are completed, the programmable local controller circuit 1340 moves the accumulated result to the next block / region and resets the counter value for the next set of operations. For example, as... Figure 14A The flow diagram illustrates that the programmable local controller circuit 1340 performs integration in layer 1. A portion of the vector result is retained. The programmable local controller circuit 1340 performs integration in layer 2, accumulating the value retained from layer 1. The accumulated result from layer 2 is retained. The programmable local controller circuit 1340 performs integration in layer 3, accumulating the value retained from layer 2. The accumulated result from layer 3 can be added to the count value of the ADC 1330 (which includes the accumulated result vector to the next processing block). The programmable local controller circuit 1340 can reset the ADC counter register.

[0088] exist Figure 14B and Figure 14C In this matrix (size K×M), the matrix is ​​larger than the crossbar array size (C) in both row and column dimensions. R ×C C ), (K>C R And M > C C The data stream and control signals from the programmable local controller circuit 1340 can map the matrix onto L=L row *L column = ceil(K / C R )* ceil(K / C C In the layers, the input data will be split along the row dimension because layers within the same row use the same input data. In the example, there are only three distinct input vectors: one consumed by layers 1, 2, and 3; one consumed by layers 4, 5, and 6; and one consumed by layers 7, 8, and 9. Depending on how some of the input vectors reach the block, the programmable local controller circuit 1340 can operate in one of two ways.

[0089] Figure 14BThis illustrates that if two consecutive partial input vectors arrive faster than the time required to perform a single integration, the programmable local controller circuit 1340 can first perform MVM along the row dimension (Case 2a) without manipulating the ADC counter (following the flow presented in Case 1). For example, if the vector values ​​of layer elements 1, 4, and 7 in the first block arrive substantially simultaneously, the process can typically perform MVM operations for each layer element in the same block and aggregate the results. For example, MVM is performed for layer element 1, then for layer element 4, then for layer element 7, and the results of each MVM are accumulated into the final result of the block. Since the input to layer 2 is the same as the input to layer 1, the first vertical flow can be prioritized if the three distinct inputs arrive quickly enough, as the first vertical flow is easier to process from the controller side (no context switching is required). However, if two consecutive input vectors do not arrive quickly enough, the process can continue to layer 2 instead of layer 4 because layers 1 and 2 use the same input. In operation, and with Figure 12 Consistent with the description, the programmable local controller circuit 1340 can perform the operation of accumulating results along columns and can concatenate (merge) the results of each column to the calculation of the next column. In L row After integration, the programmable local controller circuit 1340 moves the accumulated partial result to the next block / region and resets the counter value, allowing the next column region to be calculated. For example, the accumulated results on layers 1, 4, and 7 of the MVM can be concatenated with the results on layers 2, 5, and 8, and can be concatenated with the results on layers 3, 6, and 9. This process can be performed in total L steps before starting the next MVM operation. column Second-rate.

[0090] Figure 14CThis illustrates what happens if two consecutive input vectors arrive more slowly than it would take to perform a single integration (Case 2b), for example, when input vector data arrives at layer element 1 but not at layer element 4 in time. An alternative data stream can be followed to avoid idle time while waiting for the next input vector. In this case, the programmable local controller circuit 1340 can begin MVM in the column dimension first, since different column regions receive the same input vector. For example, MVM is performed for layer element 1, and before moving to layer element 4, the process performs MVM for layer element 2, and then for layer element 3. In this case, the value of each performed MVM can be stored, and then reloaded when the process returns to the next layer element in the row dimension (e.g., when performing MVM for layer element 4, the result for layer element 1 is retrieved). To facilitate this, while still utilizing the in-situ partial result accumulation capability of the ADC, the local controller (programmable local controller circuit 1340) can store the current value of the counter register in temporary memory and load the value corresponding to the next column region before moving to the next layer (context interleaving). This means that the programmable local controller circuit 1340 can access a certain amount of scratchpad memory, and the ADC counter can be read and written by a portion of the controller. In (L row After -1)* L_column integrations, the next L_column integration will produce a fully accumulated result for each column region, and the local controller can move them to the next block / region of the system.

[0091] Still referencing Figure 14C An example method (performed by the programmable local controller circuit 1340) may include loading the first row of region input vectors into the DAC register. MVM integration can be performed in layer element 1, the ADC counter value can be stored, and the counter can be reset. MVM integration is performed in layer element 2 (no input load required). The ADC counter value can be stored, and the counter can be reset. MVM integration is performed in layer element 3 (no input load required). The ADC counter value can be stored, and the counter can be reset. The programmable local controller circuit 1340 may load the second row of input vectors into the DAC register. The ADC counter can be loaded with the value saved from the MVM integration of layer element 1. MVM integration can be performed in layer element 4. A new ADC counter value can be stored, and the ADC counter can be loaded with the value saved from the MVM integration of layer element 2, and so on, until each layer element in the row is processed, and before processing continues to the next layer element, the value previously stored from the same column dimension element is retrieved and processed as the new value.

[0092] Figure 15A multi-layer AIMC device for MVM operations with large matrices and for multi-model operations, consistent with the illustrative embodiments, is shown. For example, block 1310 may be more suitable for large matrices (K > C). R and M>C C Furthermore, it is better suited for multi-model network mapping within the same unit. Multi-model operation is another way to leverage the weighting capabilities provided by multi-layer AIMC blocks. In this operation mode, multiple models coexist in the same system, and input tokens can be routed to one or more of these models. This operation mode may increase the need for context interleaving (see...). Figure 14C Case 2b) occurs because the input vectors of multiple models may arrive in an interleaved manner, halting part of the result accumulation process and requiring store and load operations. Furthermore, multi-model operations complicate the instructions that the local controller must execute to implement this data flow. Embodiments mitigating the overhead of context interleaving may include changing the architecture to have multiple counters (e.g., a row of counters) and configurable switches to select the counter that will be boosted in current integration, thus bringing the temporary memory closer to the ADC. In this case, the programmable local controller circuit 1340 can change the configurable switch instead of always performing a store-load operation. If the number of counters is less than the necessary context for the interleaving steps, a combination of store-load and counter switching can be used. Therefore, the programmable local controller circuit 1340 can be programmed to control which layer element is processed next and which counter is retrieved to perform the next MVM operation. For example, and returning to the reference... Figure 14C Instead of always processing layer element 2 and storing the ADC counter value after processing layer element 1, the programmable local controller circuit 1340 can be pre-programmed to process any other layer element on the block after processing layer element 1. This is understandable. Figure 15 The implementation of this method trades additional hardware elements for faster processing.

[0093] Figure 16 A multi-layer AIMC device for MVM operation is shown, similar to Figure 15 In this embodiment, the device is configured to reduce the precision accumulation of some results to save the bit size of the counter. As the counter value increases, the processing overhead in the register space also increases. In this embodiment, a downsampling stage 1620 is added before the configurable switch, which operates at a discrete frequency N. down The number of pulses is reduced. This stage might be implemented using a separate counter to count N before generating a pulse for a downstream component. down -1 pulse. The size of this counter may depend on the required maximum downsampling rate (S=log2max(N)). downThe downsampling stage allows for a reduction of S bits per counter, resulting in a total saving of (S-1)*N in register space. The programmable local controller circuit 1340 may have the option to configure the downsampling frequency for each counter. This can be calculated a priori based on the expected amount of accumulation, as follows: Considering the ADC conversion stage, which produces at most K bits per integration, and L partial result accumulations for the counters. The number of bits required to maintain the total result without loss of precision is B = K + (L-1) bits. If the number of bits is greater than the size of the counter (B>n), then N down Defined as 2 (B-n) If the dynamic range of each integral is less than K bits, a smaller number of shifts can be chosen to minimize the loss. The optimal N can be found a priori through sampling MVM. down Frequency. It's understandable that by downsampling the number of counted pulses, the number of values ​​in the counter group is reduced, thus reducing the space overhead in that row. The trade-off is to reduce the precision of the result (e.g., larger quantization error), thereby reducing the space required for the counter values.

[0094] Input Record Example

[0095] Figure 17 A three-dimensional AIMC device combining a sliding window method according to an embodiment is shown, which performs MVM operations using a large matrix with bit-serial input encoding. When performing MVM integration in an analog manner, voltage pulses are applied to the cross-switch array and generate current. Two types of pulses can exist: a single pulse or multiple pulses, which can be used to encode the duration by their values. As an example and not a limitation, a 100-nanosecond voltage pulse can be applied if the digital 100 is desired to be applied to the input cross-switch array. The value is encoded in the time dimension. The current value is determined based on the time for which a voltage pulse is applied. The current value is used in the sensing and conversion stages.

[0096] Bit-to-serial input encoding (sometimes called "bit slicing") is a way of encoding values ​​by their frequency (instead of using a single pulse). Multiple pulses can be used depending on the binary representation of the number. For example, when the value is 100, that value has a binary representation. Instead of applying a single 100-nanosecond pulse, bit slicing would use one bit value to represent the pulse being on and another bit value to represent the pulse being off. What happens is that when the next bit of the input occurs, it means that the value is twice the value of the previous bit. Hence the term binary representation. Therefore, if the least significant bit is 2 to the power of zero, the next bit is 2 to the power of one. The next bit is 2 to the power of two, and so on. Now, different bit weights are typically considered in ADCs. If there are 8-bit inputs, several multiplications are required. The results are then added together. However, the device must take into account the fact that each result has a different bit weight because the next input is twice the size of the value of the previous input.

[0097] Bit-serial input encoding is an efficient method for accelerating MVM integration operations and, in some cases, improving the accuracy of integration in the embodiments of this subject matter. When using bit-serial input encoding, the DAC system encodes the input value as a series of pulses, one pulse for each bit of the input value. In systems employing bit-serial input encoding, the ADC typically accumulates a portion of the result generated for each cycle. These results have different bit weights (differences of a factor of two each time, as described above), because each input pulse also has a different bit weight. Embodiments may incorporate a sliding window method into the ADC. The size of the selector counter register is N = k + n, where k is the output size per integration and n is the number of input bits. For each bit cycle, only a sub-region of the ADC counter is updated. The sub-region is selected based on the number of bits of the current integration, as shown in the figure.

[0098] This embodiment demonstrates that even when using bit-serial input encoding, it is possible to use... Figures 14A-14C The apparatus described herein. The sliding window method provides an easy way to successfully perform partial result accumulation across layers while the ADC also performs partial result accumulation for each input bit. The Mth bit period of each layer is added to the same region, thus ensuring equal bit weight accumulation across layers.

[0099] ADCS has already performed the accumulation within the block, but using bit-serialized input encoding, the accumulation of MVM results can be directly added together. For example, MVM is performed on a layer element. To perform MVM integration on another layer element in the line, several other MVM integrations need to be performed, and partial results obtained. Typically, something similar to a nested accumulation loop is used to find partial results. In embodiments using a sliding window, the counter is modified to shift the value of the MVM integration counter from the position where it counts the previous value in a set of bits. For example, as... Figure 17 As shown, instead of counting and placing the rightmost bit value (as shown in the top row of the counter), the next MVM integration counter value is counted from one bit to the left (and the rightmost bit position, which is currently empty, is ignored for now). The counter is now shifted to the left, meaning it effectively doubles the counted value. As the results are accumulated, each consecutive partial result moves the counter value by an additional position until the MVM integration for the layer element is complete. When the MVM integration is complete, the process restarts, and the next MVM integration counter value is executed starting from the rightmost edge of the sliding window.

[0100] Figure 18 A three-dimensional AIMC device is shown that performs MVM operations on a large matrix using bit-serial input encoding combined with a right-shift method according to an embodiment. In this embodiment, bit-serial input encoding is used. (See also: Regarding...) Figure 17 The ADCs in the systems discussed, which support bit-serial input encoding, typically perform partial result accumulation between the results of each cycle. These results have different bit weights. Using the sliding window mechanism described above is effective, but it requires a large register to accumulate all the bit cycles. Figure 18 The counter in the AIMC device accumulates a result with reduced precision by shifting right after integration in each cycle (e.g., as...). Figure 8 (As discussed in the previous section). For example, the rightmost bit is discarded. The counter value occupies the same bit position as the new bit is added to the left of the bit. Shifting right by one bit can work with a register of size N=k+1, where k is the output size for each integral. It should be noted that right shift may not conform to the accumulation of results across layers because the bit weights of the digits in the counter change in each cycle. However, right shift saves space in the counter region and is therefore economical from the perspective of reducing the counter region at the cost of reduced precision.

[0101] To integrate this mechanism into the proposed device and perform cross-layer accumulation, the order of integrations performed, configured by the programmable local controller circuit 1340, needs to be changed. For k-layer cross-layer accumulation and m-bit input, the method includes performing integration on all k layers with their respective 0th input bit. Between integrations, the programmable local controller circuit 1340... Figures 14A-14C Operate as described in 15 and 16 (aggregate and join partial results based on conditions). Figures 14A-14C 15 and 16 shift the partially accumulated counter to the right. The programmable local controller circuit 1340 performs integration on all k layers using their respective first input bits. Similarly, during these operations, the programmable local controller circuit 1340... Figures 14A-14C The process is as described in 15 and 16. This process repeats the integration and summation of partial results until all m bits are completed.

[0102] Figure 19 A method for mapping a convolutional neural network using a 3D AIMC device according to an embodiment is illustrated. For example, the 3D AIMC device can use row-by-row processing to perform layers of a convolutional neural network (CNN). Partial computation can be performed for each integration cycle, producing partial results. These partial results are accumulated to produce the final result. Accumulation is performed using an integrator in the analog domain. In this subject device, convolutional layers can be mapped onto layers such that the accumulation of partial results required for row-by-row processing occurs in the ADC counter, as in the case of an MVM with a large matrix. When mapping a CNN layer (represented by feature map 1910), each row of the filter can be transposed and mapped to a different layer to facilitate in-situ ADC accumulation of partial results. When the input reaches block 1920, the first row of the feature map is multiplied by the first layer, thereby creating a first set of partial results on the ADC. The next row is then processed for the second layer, and so on. Every k integrations, an output feature map is created and can be forwarded to the next layer.

[0103] in conclusion

[0104] Various embodiments of this teaching have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0105] While the content and / or other instances considered to be in the best possible state have been described above, it is understood that various modifications may be made therein, and the subject matter disclosed herein can be implemented in various forms and examples, and the teachings can be applied to many applications, of which only a few have been described herein. The appended claims are intended to claim protection for any and all applications, modifications, and variations that fall within the true scope of this teaching.

[0106] The components, steps, features, purposes, benefits, and advantages discussed herein are merely illustrative. None of them, or the discussion relating to them, is intended to limit the scope of protection. While various advantages have been discussed herein, it will be understood that not all embodiments are necessarily intended to include all advantages. Unless otherwise stated, all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification, including in the appended claims, are approximate and not precise. They are intended to have a reasonable scope consistent with the functionality associated with them and with the conventions of the art to which they pertain.

[0107] Many other embodiments have also been considered. These embodiments include those with fewer, additional, and / or different components, steps, features, purposes, benefits, and advantages. These also include embodiments in which components and / or steps are arranged and / or ordered in different ways.

[0108] This document describes aspects of the present disclosure with reference to calling flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each step of the flowcharts and / or block diagrams, and combinations of boxes in the calling flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions may be provided to a processor of a computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a call flow and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other means to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a call flow and / or block diagram.

[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, implement the functions / actions specified in one or more blocks of a call flow procedure and / or block diagram.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the calling flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a non-linear order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or calling flowcharts, and combinations of blocks in the block diagrams and / or calling flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0112] While the foregoing has been described in conjunction with exemplary embodiments, it should be understood that the term "exemplary" means only as an example, and not the best or optimal. In addition to what has just been stated above, whether or not it is stated in the claims, what has been stated or described is not intended or should not be construed as causing any component, step, feature, object, benefit, advantage, or equivalent to be made public.

[0113] It should be understood that, unless otherwise specified herein, the terms and expressions used herein have the general meaning consistent with those in the respective fields of investigation and research to which they pertain. Relational terms such as "first" and "second" may be used merely to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between these entities or actions. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such a process, method, article, or apparatus. Without further constraints, an element preceded by "a" or "an" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0114] This summary of the disclosure is provided to allow the reader to quickly determine the nature of the technical disclosure. It is to be understood that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the foregoing detailed description, it can be seen that various features are grouped together in various embodiments to make the disclosure flow smoothly. This method of disclosure should not be construed as reflecting an intention that the claimed embodiments have more features than those expressly recited in each claim. Rather, as reflected in the appended claims, the inventive subject matter lies in fewer than all features of a single disclosed embodiment. Therefore, the following claims are thus incorporated into the detailed description, wherein each claim is independently claimed as a separate subject matter.

Claims

1. An analog in-memory computing (AIMC) system, comprising: The first block has two or more stacked layers; Each layer of resistive switching memory devices has a cross-switch array comprising multiple columns, wherein the cross-switch array is configured to encode a weight matrix; A digital-to-analog converter (DAC) is coupled to the periphery of the first block, wherein the DAC is configured to encode an input vector into voltage pulses applied to the crossbar switch array; An analog-to-digital converter (ADC), coupled to the periphery of the first block and including a counter register, is configured to measure the induced current on each column of the cross-switch array and digitize the induced current into a digital value; and A programmable logic controller (PLC) coupled to the first block, the DAC, and the ADC, wherein the PLC is configured to: Perform the first matrix-vector multiplication (MVM) integral on the first layer of the first block; Obtain the first result of the first MVM integral performed on the first layer; Perform a second MVM integral on the second layer of the first block; Obtain the second result of the second MVM integral performed on the second layer; and The first result and the second result are summed to form the accumulated digital value of the first block, and the accumulated digital value is represented as the counter value in the register of the ADC.

2. The AIMC system as described in claim 1, wherein, The programmable logic controller is configured to: Determine whether the size of the weight matrix in the input dimension is greater than the capacity of the cross switch array in the input dimension; as well as Since the size of the weight matrix in the input dimension is greater than the capacity of the cross switch array in the input dimension, the first result and the second result are accumulated.

3. The AIMC system as described in claim 1 further includes a second block, which comprises a third layer and a fourth layer, wherein, The programmable logic controller is further configured to: Determine whether the dimensions of the weight matrix in the input and output dimensions are greater than the capacity of the crossbar switch array in the input and output dimensions; as well as Based on the determination that the size of the weight matrix in the input dimension and the output dimension is greater than the capacity of the cross switch array in the input dimension and the output dimension, it is determined to accumulate or concatenate the MVM integration result of the first block with the MVM integration result of the second block.

4. The AIMC system as described in claim 3, wherein, The programmable logic controller is further configured to: When determining the arrival of the input vector is faster than the time required to perform a single integration: Forward the counter value to the second block; Reset the counter value in the register of the ADC; Perform a third MVM integral on the third layer; Obtain the third result of the third MVM integral performed on the third layer; Perform the fourth MVM integration on the fourth layer; Obtain the fourth result of the fourth MVM integral performed on the fourth layer; as well as The third result is added to the fourth result, and the result of the addition of the third result and the fourth result is represented as the new counter value in the register of the ADC.

5. The AIMC system as described in claim 3, wherein, The programmable logic controller is further configured to: When determining the arrival time of the input vector is slower than the time required to perform a single integration: Load the first input dimension input vector into the DAC register; Perform the first MVM integration on the first layer of the first block; Obtain the first result of the first MVM integral performed on the first layer; The first result is stored as the first storage counter value in the register of the ADC; Perform a third MVM integral on the third layer of the second block; Obtain a third result; The third result is stored as a second storage counter value in the register of the ADC; The second input dimension input vector is loaded into the DAC register; Load the stored first counter value; Perform the second MVM integration on the second layer of the first block; The second result is stored as a third storage counter value in the register of the ADC; Load the stored second counter value; Perform the fourth MVM integration on the fourth layer; as well as Obtain the fourth result of the fourth MVM integral performed on the fourth layer.

6. The AIMC system as described in claim 5, wherein, The first final result of the first block is connected to the second final result of the second block.

7. The AIMC system of claim 1, further comprising a configurable switch coupled to the programmable logic controller, wherein, The configurable switch is programmed by the programmable logic controller to select the counter value used in the current MVM integration operation from the counter register.

8. The AIMC system of claim 1 further includes a downsampling module coupled to the programmable logic controller, wherein, The downsampling module is configured to reduce the number of voltage pulses at discrete frequencies.

9. The AIMC system of claim 8, further comprising a configurable switch coupled to the programmable logic controller, wherein: The configurable switch is programmed by the programmable logic controller to select the counter value used in the current MVM integration operation from the counter register; as well as The downsampling module is configured to provide the configurable switch with a number of voltage pulses that are reduced from the programmed number of voltage pulse inputs.

10. The AIMC system as described in claim 1, wherein, The programmable logic controller is configured to use bit-serial input encoding with sliding window processing in the counter register of the ADC.

11. The AIMC system as described in claim 1, wherein, The programmable logic controller is configured to use bit-serial input encoding with bit-right shift processing in the counter register of the ADC.

12. The AIMC system as described in claim 1, wherein, The programmable logic controller is configured to map convolutional layers onto multiple layers of multiple blocks.

13. An analog in-memory computing (AIMC) system, comprising: Multiple blocks; For each block, there are multiple vertically stacked layers, wherein each layer includes a resistive switching array of resistive switching devices on each layer, which includes multiple columns, wherein the cross-switching array is configured to encode a weight matrix. A digital-to-analog converter (DAC) shared by the plurality of blocks, wherein the DAC is configured to encode an input vector into a voltage pulse applied to the crossbar switch array; An analog-to-digital converter (ADC) shared by the plurality of blocks, and including a counter register, wherein the ADC is configured to measure the induced current on each column of the cross-switch array and digitize the induced current into a digital value; and A programmable logic controller (PLC) coupled to the plurality of blocks, the DAC, and the ADC, wherein the PLC is configured to: Control the ADC to retain the integral values ​​between the integrals performed at each layer; and The in-situ integration results are accumulated within the block.

14. The AIMC system as described in claim 13, wherein, The programmable logic controller is configured to accumulate partial integral results of layers on the same block.

15. The AIMC system as described in claim 13, wherein, When the programmable logic controller determines that the output dimension of the weight matrix exceeds the capacity of the crossbar switch array of the resistive variable memory device, it is configured to connect a portion of the results from the first layer with a portion of the results from the second layer.

16. The AIMC system of claim 13, further comprising a configurable switch coupled to the programmable logic controller, wherein: The configurable switch is programmed by the programmable logic controller to select the counter value used in the current matrix-vector multiplication (MVM) integration operation from the counter register; as well as The downsampling module is configured to provide the configurable switch with a number of voltage pulses that are reduced from the programmed number of voltage pulse inputs.

17. The AIMC system as described in claim 13, wherein, The programmable logic controller is configured to use bit-serial input encoding with sliding window processing in the counter register of the ADC.

18. The AIMC system as described in claim 13, wherein, The programmable logic controller is configured to use bit-serial input encoding with bit-right shift processing in the counter register of the ADC.

19. The AIMC system as described in claim 13, wherein, The programmable logic controller is configured to map convolutional layers onto multiple layers of multiple blocks.

20. A programmable logic controller in an analog in-memory computing (AIMC) system, wherein, The programmable logic controller includes instructions configured as follows: The analog-to-digital converter (ADC) coupled to the multi-layer block is controlled to retain the integral value between the integrals performed for each layer in the multi-layer block; as well as The in-situ integration results are accumulated within the block.

Images