In-memory processing based on multiple weight sets

Abstract

The invention is particularly directed to an in-memory processing method, the purpose of which is to perform matrix-vector calculations. The method relies on a device having a crossbar array structure (15). The latter comprises N input lines (152) and M output lines (153) interconnected at cross points defining N×M cells (155), where N≧2 and M≧2. The cells comprise respective memory systems, each of which stores K weights W i、j、k where K≧2. Thus, the crossbar array structure includes an N×M memory system capable of storing K N×M weight sets. To perform a multiply-accumulate (MAC) operation, the method first enables the N×M active weights for the N×M cells by selecting a weight from the K weights for each of the memory systems and setting the selected weight as the active weight. Then, a signal encoding a vector of N components is applied to the N input lines of the crossbar array structure. The latter then performs a MAC operation based on the vector and the N×M active weights. Finally, the resulting output signals at the outputs of the M output lines are read to obtain corresponding values. This allows different weight sets to be enabled locally in the crossbar array, which allows the rotation of weights to be performed locally and accordingly reduces the frequency of data exchange by the storage device. As a result, the idle time of the crossbar array structure is reduced. Thus, the proposed approach can substantially reduce the frequency of data transfer, which leads to faster computation. Furthermore, weights may possibly be read ahead while performing a MAC operation according to currently active weights. Among particular advantages, the look ahead step may be at least partially hidden by the pipelining. The present invention is further directed to related devices, systems, and computer program products.

Description

[Background technology]

[0001] The present invention relates generally to in-memory processing techniques (i.e., methods, devices, and systems) and related acceleration techniques. In particular, the present invention relates to an in-memory processing device including a crossbar array structure for performing matrix-vector multiplication with consecutive partial product accumulation and coefficient look-ahead in memory.

[0002] Matrix-vector multiplication is often required in many applications, including technical computing and cognitive tasks such as the training of and inference performed by cognitive models such as neural networks for computer vision and natural language processing, as well as other machine learning models used in weather forecasting and financial predictions.

[0003] Such operations pose multiple challenges due to their recursion, generality, and size and memory requirements: on the one hand, there is a need to accelerate these operations, especially in high performance computing applications, and on the other hand, there is a need to realize an energy-efficient way of performing them.

[0004] Traditional computer architectures are based on the von Neumann concept of computing, where processing functions and data storage are separated into separate physical devices. Such architectures suffer from congestion and high power consumption due to the need to continuously transfer data from memory devices to control and computing devices through physically constrained and expensive interfaces.

[0005] One possibility for accelerating matrix-vector multiplication is to use dedicated hardware acceleration devices, such as a dedicated circuit with a crossbar array configuration. The circuit includes input and output lines, which are interconnected at cross points that define a cell. The cell includes respective memory devices designed to store respective matrix coefficients. The vectors are encoded as signals applied to the input lines of the crossbar array, causing the latter input lines to perform a multiply-accumulate (MAC) operation. There are several possible implementations. For example, the coefficients of the matrix ("weights") can be stored in columns of cells. Adjacent to each column of cells is a column of arithmetic units that can multiply the weights by the input vector values ​​(resulting in partial products), and finally accumulate all partial products to produce a full inner product result. Such an architecture can simplify and efficiently map matrix-vector multiplication. The weights can be updated by reprogramming memory elements to perform matrix-vector multiplication as needed. Such a solution would break down the "memory wall" and make in-memory or near-memory processing much more efficient by fusing computation and storage into a single in-memory computation (IMC) device.

[0006] Devices with such crossbar array structures are in routine use, and the inventor has now set himself the task of improving such devices to make them more energy efficient and more computationally fast. Summary of the Invention

[0007] According to a first aspect, the invention is embodied as an in-memory processing method aimed at performing a matrix-vector calculation. The method relies on a device having a crossbar array structure. The latter comprises N input lines and M output lines, which are interconnected at cross points defining N×M cells, where N≧2 and M≧2. The cells comprise respective memory systems designed to store K weights, where K≧2. Thus, the crossbar array structure comprises N×M memory systems capable of storing K N×M weight sets. To perform a multiply-accumulate (MAC) operation, the method first enables N×M active weights for the N×M cells by selecting a weight from the K weights for each of the memory systems and setting the selected weight as the active weight. Then, a signal encoding a vector of N components is applied to the N input lines of the crossbar array structure. The latter thereby performs a MAC operation based on this vector and the N×M active weights. Finally, the output signals obtained at the outputs of the M output lines are read to obtain corresponding values.

[0008] The above scheme allows different weight sets to be locally enabled in the crossbar array, which allows for local switching between active weights and correspondingly reduces the frequency of data exchange with the storage device. This results in reduced idle time of the core computing device, i.e., the crossbar array structure. That is, some intermediate weight updates can be avoided, since up to K consecutive computation cycles can be performed without the need to transfer new weight sets. Rather, the corresponding weight sets can be locally enabled as active weights in each computation cycle. Furthermore, partial results can be locally accumulated to avoid the transfer of intermediate results. Thus, the proposed approach can substantially reduce the frequency of data transfer, which leads to faster computations and reduced power consumption required to perform matrix-vector computations.

[0009] In a particularly advantageous embodiment, the method further comprises looking ahead weights while performing the MAC operation according to the N×M weights that are currently enabled as active weights. That is, if 1≦q≦K−1, the N×M weight sets q (i.e., the weights to be used next) are looked ahead and stored in the N×M memory system instead of the previously active N×M weight sets q. In other words, the weights can be proactively loaded (i.e., looked ahead during certain computation cycles) as needed to further reduce the idle time of the crossbar structure. Among the particular advantages is that the looking ahead step is at least partially hidden by the pipelining.

[0010] Preferably, the N×M active weights are enabled by concomitantly selecting a k-th weight of the K weights of each memory system of at least a subset of the N×M memory systems, where 1≦k≦K, and setting the selected respective weight accordingly as the currently active weight, such that the array structure can change context almost instantaneously.

[0011] In a typical embodiment, several matrix-vector calculation cycles are performed in succession, each cycle including the operations as described above: first, for each of the memory systems, a new N×M active weight set is enabled for the N×M cells by selecting a weight from its K weights and setting the selected weight as the active weight. Then, signals encoding the N-component vector are applied to the N input lines of the crossbar array structure, causing the latter to perform a MAC operation based on the current vector and the new N×M active weight set. Finally, the resulting output signals at the outputs of the M output lines are read to obtain the corresponding values. Up to K such cycles can be performed without the need to transfer new weights to the memory systems of the array.

[0012] Preferably, each cycle further includes successively performing the accumulations by accumulating partial product results corresponding to the read output signals. Upon completing several matrix-vector calculation cycles, the method may, for example, return the results obtained based on the successive accumulations to an external (i.e., external to the array) storage device. Thus, there is no need to transfer intermediate results.

[0013] If necessary, new weights may be looked ahead before completing K of several matrix-vector calculation cycles. That is, the method may look ahead q N×M sets of weights, for 1≦q≦K−1, and store the latter in the N×M memory system instead of the q N×M sets previously enabled as active weights. Interestingly, new weight values ​​may possibly be looked ahead intermediately, so that further matrix-vector calculation cycles (beyond K cycles) can be performed uninterruptedly while continuing to accumulate partial results. Also, the look ahead step is hidden by the pipelining. The final result can be returned at the very end of the entire matrix-vector calculation without suffering from idle times due to intermediate data transfers.

[0014] Large operands can be addressed, for example, by decomposing the required operations into K×T matrix-vector calculations. That is, the method may perform K×T matrix-vector calculation cycles, where T corresponds to the number of input vectors. Each input vector is decomposed into K sub-vectors of N components and associated with K respective block matrices, the latter corresponding to K N×M weight sets. Note that the sub-vectors actually correspond to the vectors introduced above, each sub-vector being a portion of the input vector. A K×T matrix-vector calculation cycle is then performed as follows: First, the K N×M weight sets are loaded. The K N×M weight sets correspond to the K block matrices. The memory system is accordingly programmed to store the K N×M weight sets. Then, several operations are performed, one for each of the K sub-vectors of each of the T input vectors. First, one of the K respective block matrices, i.e., the N×M weight corresponding to the block matrix associated with the current sub-vector, is enabled (as the currently active weight). Second, signals encoding vectors corresponding to each sub-vector are applied to the N input lines, such that the crossbar array structure performs a MAC operation based on each sub-vector and the currently active weights. Third, the method reads the output signals available at the outputs of the M output lines to obtain the corresponding partial values.

[0015] The retrieval preferably includes accumulating the partial values ​​obtained for each sub-vector with previously obtained partial values ​​for previous ones of the K sub-vectors, if any, to obtain an updated result. Finally, the method returns the obtained result based on the last obtained updated result.

[0016] In an embodiment, an external (i.e., external to the array) processing unit is used to map a given problem to a certain number of sub-vectors and a set of K N×M weight sets before programming an N×M memory system with K N×M weight sets and encoding the sub-vectors into input signals and then applying such input signals to the N input lines to perform several matrix-vector calculation cycles. It is noted that the external processing unit may optionally be integrated with the crossbar array structure. In a variant, the unit forms part of a separate device or machine.

[0017] The N×M memory system can be either a digital memory system or an analog memory system. In either case, the MAC operations can be performed in parallel or as bit-serial operations.

[0018] In an embodiment where the N×M memory systems are digital memory systems, each of the N×M cells further includes a computing device connected to a corresponding one of the N×M memory systems.

[0019] For example, a MAC operation may be performed bit-serial in P cycles, P≧2, where P corresponds to the bit-width of each of the N components of each of the vectors (or sub-vectors) used at the input. Partial product values ​​are then obtained, which are accumulated locally (in the crossbar arrangement) upon completion of each of the P cycles. However, this accumulation should be distinguished from the accumulation performed upon completion of vector-level operations, i.e. operations related to a vector (or sub-vector), when processing several vectors (or sub-vectors) in succession.

[0020] According to another aspect, the invention is embodied as a computer program for in-memory processing. The computer program product comprises a computer readable storage medium having embodied thereon program instructions executable by a processing means of an in-memory processing hardware device to cause the latter to perform any of the steps of the method described above.

[0021] According to a further aspect, the invention is embodied as an in-memory processing hardware device. Conformable to the method, the device comprises a crossbar array structure including N input lines and M output lines interconnected at cross points defining N×M cells, where N≧2 and M≧2. The cells include respective memory systems each designed to store K weights, where K≧2. That is, the crossbar array structure includes N×M memory systems, which are collectively adapted to store K N×M weight sets for performing a MAC operation. The device further includes a selection circuit connected to the N×M memory systems. The selection circuit is configured to select a weight from the respective K weights of the memory systems and set the selected weight as an active weight to enable the N×M active weights for the N×M cells. In addition, the device includes an input device configured to apply a signal encoding a vector of N components to the N input lines of the crossbar array structure to cause the latter to perform a MAC operation based on this vector and the N×M active weights to be enabled by the selection circuit during the operation. The device further includes a readout arrangement configured to read out the output signals obtained on the M output lines.

[0022] In an embodiment, each of the N×M memory systems is designed such that the K weights are independently programmable. The device may further include a program circuit connected to each memory system. The program circuit is configured to program the K weights of the N×M memory system. The program circuit may advantageously be configured to look ahead q sets of N×M weights that are not currently set as active weights if 1≦q≦K−1, and accordingly program the N×M memory system such that the latter stores the look ahead weights in place of the q sets of N×M weights.

[0023] In a preferred embodiment, each of the N×M memory systems includes K memory elements, each adapted to store a corresponding one of the K weights, and the selection circuitry includes N×M multiplexers, each connected to each of the K memory elements of a corresponding one of the N×M memory systems, and a selection control line connected to each of the multiplexers such that any one of the K weights of each of the memory systems can be selected and set as the active weight during an operation.

[0024] Preferably, the selection circuitry is further configured to select a subset of n×m weights from one of the K N×M weight sets by concomitantly selecting a kth weight of the K weights of each memory system of the subset of n×m memory systems of the N×M memory system, where 2≦n≦N, 2≦m≦M, and 1≦k≦K.

[0025] In an embodiment, the in-memory processing hardware device further comprises a sequencer circuit and an accumulator circuit. The sequencer circuit is connected to the input device and the selection circuit and organizes the operation of the input device and the selection circuit to perform several cycles of matrix-vector calculations in succession based on one or more vector sets. During operation, each cycle of matrix-vector calculations includes one or more cycles of MAC calculations. Different N×M weight sets are selected from the K N×M weight sets and set as N×M active weights in each cycle of matrix-vector calculations. The accumulator circuit is configured to accumulate partial product values ​​obtained upon completing each MAC calculation cycle. Preferably, the accumulator circuit is arranged at the output of the output line.

[0026] In a preferred embodiment, each of the N×M memory systems includes K memory elements, each adapted to store a corresponding one of the K weights. Each of the K memory elements of each of the N×M memory systems may be, for example, a digital memory element. In that case, each of the N×M cells further includes an arithmetic unit, which is connected to each of the K memory elements of a corresponding one of the N×M memory systems via a corresponding portion of the selection circuit.

[0027] In an embodiment, each of the K memory elements of each of the N×M memory systems is designed to store a P-bit weight. The input device is configured to bit-serially provide a vector of N components to the input lines in P cycles by applying the above signals, where each of the N components corresponds to a P-bit input word, and P≧2. The N×M cells are configured to bit-serially perform the MAC operation in the P cycles. In addition, the hardware device further includes an accumulator circuit configured to accumulate values ​​corresponding to partial bit-serial product values ​​obtained in each of the P cycles. And the selection circuit is configured to maintain the same N×M weight set as the active weights during each of the P cycles.

[0028] Preferably, the in-memory processing hardware device further comprises a configuration and control logic unit connected to each of the input devices and the selection circuit, a pre-data processing unit connected to the configuration and control logic unit, and a post-data processing unit connected at the output of the output line.

[0029] According to another aspect, the invention is embodied as a computing system comprising one or more in-memory processing hardware devices as described above. Preferably, the computing system further comprises a storage device and a general purpose processing device connected to the storage device for reading and writing data to the storage device. Each of the in-memory processing hardware devices is configured to read and write data to the storage device. The general purpose processing device is configured to map a given computational task to vectors and weights to the memory system of the one or more in-memory processing hardware devices.

[0030] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which should be read in connection with the accompanying drawings, the illustrations being provided for clarity so that those skilled in the art may readily understand the present invention in conjunction with the detailed description. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a schematic representation of a computerized system in which a user interacts with a server via a personal computer to offload matrix-vector calculations to a dedicated hardware accelerator, according to an embodiment of the present invention. [Diagram 2] 2A and 2B are schematic diagrams illustrating selected components of a hardware accelerator optimized for performing in-memory computing (IMC) matrix-vector multiplication as included in an embodiment, where FIG. 2A illustrates a crossbar array structure of the hardware accelerator and FIG. 2B illustrates additional components of the hardware accelerator. [Figure 3A] FIG. 1 illustrates an IMC array including a column of arithmetic units (multipliers and adder trees) connected to each column of memory elements, in accordance with an embodiment, where each memory cell includes a memory system of several memory elements, and overall, the crossbar array structure includes an N×M memory system capable of storing K N×M weight sets. [Figure 3B] FIG. 1 shows a schematic diagram of a given row of memory cells, as well as a portion of the program circuitry and select circuitry, as included in an embodiment, where the shown portion of the program circuitry and select circuitry is connected to a single memory cell and other portions are not shown for purposes of illustration, although in reality the program circuitry and select circuitry would be connected to each memory cell. [Figure 3C] 4 is a simplified circuit diagram of components of a selection circuit coupled to corresponding memory cells as included in an embodiment. FIG. [Figure 4A] FIG. 4A shows two examples of IMC chip configurations according to embodiments, in which the IMC array is connected to an accumulator in each case, with FIG. 4A assuming bit-serial injection of vector components for matrix-vector multiplication. [Figure 4B]4A and 4B show two examples of IMC chip configurations according to embodiments, in which the IMC array is connected to an accumulator in each case, with FIG. 4A assuming parallel operation. [Figure 5A] FIG. 13 illustrates how, as in an embodiment, matrix-matrix multiplication involving large operands (matrices) can be accommodated by a smaller size IMC array, whereby the input matrices are decomposed into input vectors, which themselves are partitioned into sub-vectors where different block matrices are assigned to perform successive matrix-vector multiplications by locally validating each matrix coefficient array and accumulating partial results before returning the final result. [Figure 5B] FIG. 13 illustrates the corresponding rotation of a set of K coefficients (weights) after initially loading the set of K weights, as in an embodiment. [Figure 6] FIG. 13 illustrates how (next to be used) matrix coefficients can be looked ahead to speed up calculations, as in an embodiment. [Figure 7] 5B is a flowchart illustrating high level steps of a method for performing a matrix-matrix multiplication similar to the multiplication shown in FIG. 5A, in an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The accompanying drawings show simplified representations of devices or portions thereof as included in the embodiments. Like or functionally similar elements in the drawings are assigned the same reference numerals unless otherwise indicated.

[0033] Computerized devices, systems, methods and computer programs embodying the invention will now be described by way of non-limiting examples.

[0034] The following description is organized as follows: general embodiments and high-level variants are described in Section 1, and particularly preferred embodiments are addressed in Section 2. Final thoughts are summarized in Section 3. Please note that the method and its variants are collectively referred to as "the method". All reference signs Sn refer to the method steps of the flow chart of FIG. 7, and the reference numbers relate to devices, components, and concepts included in the embodiments of the present invention.

[0035] 1. General embodiment and high level variants A first aspect of the present invention will now be described with reference to Figures 2A to 4B and Figure 7. This aspect relates to a method of in-memory processing, the purpose of which is to accelerate multiply-accumulate or MAC operations.

[0036] The method relies on a device 10, 10a having a crossbar array structure 15, 15a. The crossbar array is shown in FIG. 2A. The structure 15, 15a includes N input lines 152 and M output lines 153, where N≧2 and M≧2. The input lines 152 and output lines 153 are interconnected at cross points. The cross points accordingly define N×M cells 155. The cells 155 include respective memory systems 157. Notably, each memory system 157 is designed to store K weights, where K≧2. In practice, K may typically be equal to 4, 8, 16, or 32 (as assumed in FIG. 3A, FIG. 3B, and FIG. 5A, FIG. 5B). Overall, the crossbar array structure 15 includes N×M memory systems capable of storing K N×M weight sets, i.e., K×N×M weights in total.

[0037] In other words, the crossbar array structure 15 includes N×M cells 155 in a crossbar configuration, where each crosspoint of the crossbar configuration corresponds to a cell, and each cell includes a memory system 157 capable of storing K weights. Such weights are denoted as W in FIG. 2A. i、j、kwhere i ranges from 1 to N, j ranges from 1 to M, and k ranges from 1 to K. In practice, the number of input lines 152 and output lines 153 will typically be on the order of hundreds to thousands of lines. For example, 256×256, 512×512 (as in FIGS. 4A and 4B), or 1024×1024 arrays are possible, although N does not necessarily have to be equal to M. The concept of input lines and output lines is further explained below.

[0038] The proposed method basically revolves around enabling a certain weight before performing a MAC operation based on a given vector and matrix coefficients corresponding to the enabled weight. That is, N×M weights are enabled for N×M cells 155 in step S70 (see the flowchart in FIG. 7). This is achieved by selecting a weight from the K possible weights for each memory system and then setting the selected weight as the active weight. It should be noted that the selection and setting of the weight can in fact be performed in a single operation, especially when using a selection circuit 159 that relies on multiplexers connected to each memory system, as in the embodiment discussed below with reference to FIG. 3C.

[0039] Once the weight set is enabled, the vector components are injected into the crossbar array structure 15 (step S82). More precisely, signals encoding a vector of N components (hereafter referred to as an N-vector) are applied S82 to the N input lines 152 of the crossbar array structure 15. This causes the crossbar array structure 15 to perform a MAC operation S84 based on the N-vector and the N×M active weights that are currently enabled. The MAC operation results in the values ​​encoded by the signals provided to the N input lines being each multiplied by a currently active weight value that is enabled from the K weight sets stored in the memory system 157.

[0040] For the crossbar configuration, M MAC operations are performed in parallel during each computation cycle. Note that the operations performed per cell correspond to two scalar operations, i.e., one multiplication and one addition. Thus, M MAC operations imply N×M multiplications and N×M additions, which means a total of 2×N×M scalar operations.

[0041] The output signals available on the M output lines 153 are then read in step S90 to obtain the corresponding values. In practice, several calculation cycles often need to be performed in succession, whereby the weights are locally enabled (i.e. selected and set as active weights) every cycle before providing the components of the N vectors to perform the MAC operation and read the output values. Such output values ​​may advantageously correspond to partial values ​​that may be locally accumulated in the device 10, 10a. In that respect, the read operation performed should be understood in a broad sense in short. The read operation may not only aim to extract the output values, but also to accumulate them (if necessary) with previous output values ​​and / or to store such values.

[0042] Notably, the proposed scheme allows different weight sets to be locally enabled in the crossbar array 15, which allows weight rotation to be performed locally and accordingly reduces the frequency of data exchange by storage devices external or embedded in the device 10, 10a. This also reduces the idle time of the device 10, 10a. That is, some intermediate weight updates are avoided, since up to K consecutive computation cycles can be performed without the need to transfer new weight sets. Rather, the corresponding weight sets are locally enabled as active weights in each computation cycle. Furthermore, weights can possibly be proactively loaded (i.e., read ahead during computation cycles) as needed to further reduce the idle time of the crossbar structure 15. Thus, the proposed approach allows a substantial reduction in the frequency of weight data transfer, which results in faster computation. Also, since partial results can be locally accumulated, such results do not need to be transferred, which reduces the power consumption of the device 10, 10a.

[0043] A few observations in order: Each memory system 157 preferably contains K distinct memory elements for simplicity. Such elements are connectable to be independently programmable. This allows for look-ahead of (next to be used) weights, as in the preferred embodiment discussed below. The memory elements are programmable to store binary or multi-bit data, for example, as synaptic weights in a synaptic crossbar array structure.

[0044] The weights are associated with numerical values ​​and represent matrix coefficients. Such weights overcome (a part of) the problem to be solved and need to be programmed into the memory system 157 accordingly. In that respect, the hardware device 10 may advantageously include a program circuit 158 ​​(FIG. 3B) configured to program the N×M memory system 157 so that the latter stores each of the K weight sets. The program circuit may for example be controlled by the logic unit 12 in the device 10, 10a. In a variant, the program circuit may be external, in which case the device may include pads and traces dedicated to programming the memory elements.

[0045] Programming memory elements of an IMC device is known per se. However, in the present context, what is needed is to properly and timely program some memory elements (or some memory values ​​of a memory system) for the respective cells. What is further needed is to appropriately select the active weights. To that end, a selection circuit 159 (FIGS. 3B, 3C) can be used to perform the selection of the required weights and to enable the selected weights as active weights.

[0046] The vectors used at the input (also referred to above as N vectors) each have N components according to the number N of input lines. Such vectors may in fact correspond to portions of a larger input vector. That is, the problem to be solved (e.g. matrix-matrix multiplication) may typically involve large operands. Thus, in the first problem, it may have to be decomposed into smaller matrix-vector operations involving portions of the input vector and matrix partitioned according to the size of the array 15. For example, an input matrix may be decomposed into an input vector, which itself is decomposed into sub-vectors (i.e. N vectors), each of which is assigned a block matrix, in order to perform multiple operations, the outputs of which can finally be reconstructed to form the final result.

[0047] So, in practice, the basic calculation principle is to feed N vectors to array 15 and perform a matrix-vector calculation on the one hand on the vector components fed in and on the other hand on the currently active weights, the latter being appropriately enabled according to the current N vectors.

[0048] To perform a MAC operation, input signals are applied to N input lines, by which components of N vectors are encoded. That is, each input signal encodes a different vector component and is applied to a corresponding input line. The input signals correspond to so-called data channels in a synaptic crossbar structure. Each vector component and each matrix coefficient can be encoded, for example, as a P-bit value. That is, the MAC operation can be performed bit-serially (as assumed in FIG. 4A or FIG. 7) or in parallel (as in FIG. 4B). In a bit-serial implementation, each multiplication operation is performed in P bit-serial cycles (for example, P=8 or 16), as will be discussed in detail later with reference to the preferred embodiment. In a variant, each P-bit word (vector component) is injected in parallel into M cells of the corresponding input line.

[0049] The approach corresponds to analog memory elements and analog operations. In an analog electrical implementation, the digital input is converted to an analog representation by a digital-to-analog converter (DAC) or a pulse-width modulator (PWM) and then applied to the input lines. Each cell operation typically corresponds to a single analog operation, whereby the input signal is multiplied by a weight value conveyed by a memory component, which then electrically interacts with the component and branches to a column output, effectively resulting in an analog summing operation. A similar principle can be used with optical input signals. A digital implementation relies on a digital memory system. That is, the N×M memory systems 157 are digital memory systems (e.g., each includes K digital memory elements). In that case, each of the N×M cells 155 includes an arithmetic unit 156 (including a multiplier and an adder tree) connected to the corresponding memory system 157, as assumed in Figures 2A, 3A, and 3B.

[0050] For completeness, it should be noted that the input lines 152 refer to channels over which data is communicated by signals to the M cells. However, such input lines do not necessarily correspond to the number of physical conductors or logical channels required to actually reach the cells. In a bit-serial implementation, each input line may comprise a single physical line, which is sufficient to provide an input signal carrying N vector component data. However, in a parallel data collection approach, each input line may comprise up to P parallel conductors, each connected to the M cells of the corresponding input line. In such a case, the P bits are injected in parallel via the parallel conductors into each of the M corresponding cells. Still, various intermediate configurations including both parallel and bit-serial data feeds may be envisaged.

[0051] The hardware device 10, 10a is preferably fabricated as an integrated structure, e.g., a microchip, incorporating all components necessary to perform the core computation steps. Such components may notably include an input device 151, 151a (FIGS. 4A, 4B) for applying input signals to N input lines 152, a program circuit 158 ​​(FIG. 3B), a selection circuit 159 (FIGS. 3B, 3C), and a readout device 154, 154a (FIGS. 4A, 4B) that may include an accumulator. The selection circuit 159 and the input device 151 may, for example, in an embodiment, form part of the same configuration and control logic and be controlled by the same logic device 12 (FIG. 2B). The device 10, 10a itself relates to another aspect of the invention, which will be described in more detail later.

[0052] A particular embodiment of the method is now discussed. First, weights can be proactively loaded (i.e., read ahead during a computation cycle) if necessary to further reduce the idle time of the crossbar structure. The read ahead mechanism is shown in Figure 6. A huge advantage is that the look ahead step can be hidden (at least partially) by the pipelining method.

[0053] In particular, new weights can be read ahead S115 and stored in the N×M memory system 157 during the current computation cycle, i.e., while performing a MAC operation according to the N×M weights currently enabled as active weights. If 1≦q≦K−1, then up to q sets of N×M weights (i.e., next to be used weights) can be read ahead S115 and stored in the N×M memory system 157, instead of the q sets of N×M weights that were previously active.

[0054] Before starting a cycle of matrix-vector calculation, K N×M weight sets may be initially loaded in an array, and subsequent look-ahead steps are then typically performed iteratively. Looking ahead weight sets allows for a proactive approach, which allows for further speedup of calculations, as shown in FIG.

[0055] For example, assume that one or more matrix-matrix multiplications must be performed. In the example in FIG. 6, four consecutive different matrix-matrix multiplications are performed, each using only a single weight set and requiring T computation cycles. In this case, it is possible to iteratively load weight sets from, for example, an external memory to look ahead S115 weights for the next matrix-matrix. As another example, the first matrix multiplication uses two weight sets, while the third and fourth matrix multiplications require only one weight set. In that case, the weight sets for the third and fourth matrix multiplications can be look ahead while the two weight sets for the first matrix multiplication are being used for computation. Such a look ahead strategy can lead to significant speedups, since the time to load weights can then be partially hidden by the pipelined approach (as shown in FIG. 6), in contrast to previous crossbar array structures that utilize only a single weight set.

[0056] In that respect, note that the cycle shown at the top of Figure 6 corresponds to a cycle performed in a typical crossbar array, where each cell stores a single weight value. The loading and processing steps (to load the weights into this array) must be interleaved, which makes it impossible to hide the load time by pipelining. In contrast, in an array that stores multiple weight sets, the matrix coefficients can possibly be preloaded into unused memory elements while other weight sets are currently active. This can result in a further significant speedup in processing time compared to systems that rely on a single weight set.

[0057] Referring to FIGS. 3B, 3C, and 7, it is preferable that the required weights can advantageously enable all of them to be effective almost instantaneously at once. For example, for an N×M weight, when 1≦k≦K, the k-th weight of the K weights of each memory system 157 can be selected along with it, and each of the correspondingly selected weights can be set as the currently active weight, thereby making it effective S70. Depending on the situation, the entire N×M memory system 157, or only a subset thereof, for example, the selection of weights for a subarray may be performed. For example, when multiplying a vector smaller than the L-component standard (L<N) by an L×L matrix, the corresponding weight array can be selected and set without the need to enable the remaining weights, because the remaining components of the vector smaller than the standard can be set to zero (zero-padding). In all cases, some weights can optionally be selected and set along with it. As a result, the change in context is almost instantaneous. That is, the switch from one weight set to another does not involve any substantial downtime. This can be realized by a selection circuit including a multiplexer 159 as shown in FIGS. 3B and 3C, for example.

[0058] In practice, as assumed in Figures 5A, 5B and 7, several calculation cycles may have to be performed in succession. Up to K cycles may be performed in succession by switching weights locally. That is, the method may result in performing several matrix-vector calculation cycles S60-S100, where each cycle includes (i) enabling S70 a new N x M active weight set, (ii) performing S84 a MAC operation based on the enabled weights and the associated N vectors, and (iii) reading S90 the resulting output signals at the output of the M output lines 153 to obtain corresponding values. Each time, a new N x M weight set is enabled S70 for the N x M cells 155 by selecting a weight from its K weights for each memory system 157 and setting the selected weight as the active weight. To perform a MAC operation S84, signals encoding the N vectors are applied S82 to the N input lines 152 of the crossbar array structure 15, whereby the latter performs a MAC operation S84 based on the N vectors and the new N×M set of active weights.

[0059] That is, the input signal is repeatedly applied to the N input lines 152 to provide N vectors in succession and cause the MAC operations to be performed accordingly. As mentioned above, each N vector may in fact correspond to a portion of a larger input vector (e.g., from a given input matrix), which portion is assigned a corresponding block matrix, as shown in FIG. 5A. In that case, the N vectors provided to the crossbar array in each matrix-vector calculation cycle are different. In other cases, the same N vector may possibly be applied several times in succession, depending on the operation decomposition scheme determined upstream of step S30 (FIG. 7).

[0060] In all cases, the new active weight set can be locally enabled for each matrix-vector calculation cycle that is executed, without undergoing an intermediate programming step to change the weights. As mentioned above, this can of course involve a possible look-ahead operation, which is nevertheless hidden by the pipeline method. That is, q N×M weight sets can be looked-ahead S115 and stored (instead of the previous q weight sets) possibly before completing K matrix-vector calculation cycles. For example, a single N×M weight set can be looked-ahead in each iteration (q=1). In a variant, two N×M weight sets can be looked-ahead after completing each second iteration. Various other look-ahead schemes can be considered. It should be noted that such look-ahead schemes can possibly be dynamically applied depending on the workload.

[0061] Also, for operations involving large operands, the partial results obtained at the end of each intermediate cycle can advantageously be accumulated locally, i.e. after a portion of the calculation cycle, the partial product results can be accumulated S90 in the device 10, 10a (at the output of the crossbar array structure 15), i.e. the accumulations are performed successively, with a view to reconstructing the final result later. The final result is obtained on the basis of the successive accumulations. The final result can for example be returned to the external storage device 2 after completing a certain number of calculation cycles. Interestingly, further matrix-vector calculation cycles can be performed uninterruptedly while continuing to accumulate partial results, since new weight values ​​can possibly be looked ahead S115 in between.

[0062] Thanks to the partial accumulation, only the final result of the matrix-vector calculation needs to be transferred and written to the external memory, without idle times due to intermediate transfers of updated weights and partial results. If a matrix-matrix multiplication is performed, the final result of each matrix-vector product can possibly also be stored locally in the device 10. After the matrix-matrix multiplication, only the result has to be returned to the external memory device 2. In both cases, based on the successive accumulations, a partial result is obtained locally, prior to transferring the final result to the external entity. The partial results do not need to be transferred to the external entity, and due to the successive accumulations, they are in fact deleted without even needing to be stored.

[0063] Consider the example of FIG. 5A in consideration of the flow of FIG. 7. Here, the objective is to perform a matrix-matrix multiplication, which can be decomposed into K×T matrix-vector calculation cycles, as mentioned above, where T corresponds to the number of columns of one of the input matrices, and is decomposed accordingly into T input vectors. And each input vector can also be decomposed into K sub-vectors, i.e., N vectors of N components each, where each N vector is associated with a respective block matrix. That is, each input vector of K×N components is associated with K block matrices, the latter corresponding to K N×M weight sets. Then, the K×T matrix-vector calculation cycle can be performed S50-S110 as follows. First, K N×M weight sets (corresponding to K block matrices) are loaded S55 and programmed accordingly in the memory system 157, the latter needing to store K N×M weight sets. Then, for each of the T input vectors, a calculation cycle is performed (see steps S58 to S110) for each of the N vectors (i.e. for each of the K sub-vectors, see steps S60 to S100), i.e. the loop on the N vectors may be nested in a loop for the input vectors, which themselves may, if necessary, be nested in a loop for the input matrices (steps S50 to S120).

[0064] The innermost loop (for the N vectors) follows the same principle as described above: N×M active weights are enabled as currently active weights S70, which correspond to the block matrix associated with the current N vector as previously assigned. Then, a signal encoding the current N vector is applied to the N input lines 152 S82, and the crossbar array structure 15 performs a MAC operation based on this N vector and the currently active weights S84. The output signals obtained at the outputs of the M output lines 153 are also read to obtain corresponding partial values, which can advantageously be accumulated in the device 10 S90. That is, the partial values ​​obtained for each N vector (however the first one) can be locally accumulated S90 with the previously obtained partial values ​​for the previous N vectors. In this way, an updated result is obtained at each cycle. The last obtained updated result is finally returned S120 to the external memory.

[0065] Such an operation is visually illustrated in FIG. 5A, where K is assumed to be equal to 4 in this example. That is, the matrix-matrix multiplication will be calculated in four calculation cycles of T, where each of the T input vectors has 4×N components, as seen in FIG. 5A. Furthermore, the sequence of operations assumes T=4 input vectors in this example. Each block matrix is ​​stored as a single N×M weight set. The calculation unit in the IMC array calculates a partial inner product for each pair of N vectors and associated weight sets. FIG. 5A shows how the array switches contexts (i.e., weight sets) in each calculation cycle and calculates the complete result locally in the accumulator. Only after the last iteration is the final result written back to the external memory. That is, the accumulator accumulates the four partial products in succession, and finally writes the result back to the external memory. This process is repeated T times, i.e., once for each input vector.

[0066] Figure 5B shows the timing. Initially, all weight sets (WS0-WS3) are loaded consecutively into the IMC, which corresponds to step S55 in Figure 7. Then, computation cycles begin in an interleaved fashion. Only four input vectors are included (T=4), each of which is decomposed into K=4 sub-vectors of N components. Weight sets WS0-WS3 are enabled (i.e. rotated) consecutively according to each of the K portions of each of the T input vectors.

[0067] Since in this example four available weight sets are used, all needed weight sets can be preloaded in array 15 in advance without needing any look-ahead S55. That is, the decomposition assumed in the flow of Figure 7 a priori does not require weight look-ahead because the matrix-vector operation already utilizes rotations of all K weight sets in this example, leaving no room (and indeed no need) to look ahead proactively to the next weight set.

[0068] However, looking ahead may be advantageous if the input vector has to be decomposed into K or more parts. In addition, also in the context of Figures 5A and 7, new weights may possibly be looked ahead during processing of the last K sub-vectors corresponding to the very last input vector. That is, upon completing a computation cycle for any of these K sub-vectors, an instruction may be given to look ahead a new set of weights and write it in place of the previously active NxM weights. This reduces the idle period (corresponding to step S55) before starting S50 the computations related to another matrix-matrix multiplication.

[0069] Of course, Figures 5A and 7 reflect one possible mapping of operations. A variety of other computational strategies leading to different sequences of operations may be considered in step S30. Furthermore, it is worth noting that regardless of the decomposition scheme employed at run time, the core computing device 10 may generally be designed to allow weight look-ahead, if desired.

[0070] The optimal mapping of operations is determined S30 by an external processing unit 2, 13, i.e., a unit separate from the core computer array 15. Still, this external processing unit 13 may possibly be integrated with the core IMC array 15 in the device 10, 10a, as assumed in FIG. 2B. In all cases, the processing unit 2, 13 is used to determine the computation strategy (i.e., to identify and associate the sub-vectors and block matrices), which may also be called a conditional operation. In practice, this amounts to mapping S30 a given problem to a certain number of sub-vectors and K N×M weight sets. This step is performed prior to programming S55 the N×M memory system 157 accordingly, and encoding the computed vectors into the input signal, for the purpose of subsequently performing the MAC operation. The processing unit 2, 13 may possibly perform other tasks, as will be discussed later.

[0071] In an embodiment, the MAC operation is performed bit-serial, i.e., in P serial cycles, S84, where P≧2. In practice, P is typically 2 rwhere 3≦r≦6. P is assumed to be equal to 8 in the example of FIG. 4A. The value P corresponds to the bit width of each of the N vectors used at the input. Note that upon completion of each of the P cycles, partial product values ​​must then be locally accumulated S86. The calculation cycles S82-S88 are minor cycles that must be distinguished from the matrix-vector calculation cycles (S50-S100), which themselves may benefit from partial accumulation S90. That is, each inner calculation cycle S80 includes P cycles, whereas the matrix-vector calculation cycles include K cycles (themselves nested in T cycles).

[0072] In a variant, the method relies on a parallel implementation (see FIG. 4B), which does not require any parallel-to-serial conversion as in FIG. 4A. In the parallel implementation, each N vector is processed by weight multiplication in a single cycle. In a further variant, a hybrid approach with parallel feeding to bit-serial values ​​can be considered.

[0073] Another aspect of the invention concerns a computer program for in-memory processing. The computer program product includes a computer-readable storage medium having program instructions embodied therein, where the program instructions are executable by the processing means 12, 13, 14 of the in-memory processing hardware device 10, 10a to cause the latter to perform the steps described above, including, as appropriate, the MAC operation S84, and the accumulation S86, S90 and look-ahead S115 operations. More generally, such processing means 12, 13, 14 process some (or possibly all) of the pre-processing and post-processing operations as suggested in FIG. 2B. These operations can be executed, for example, on an instruction-based processor or a dedicated accelerator with various instruction or command-based control mechanisms.

[0074] Apart from the operation S30 aimed at determining the computation strategy, the device 13 may possibly perform other tasks, for example related to element-wise or non-linear operations. For example, in a machine learning (ML) application, the device 13 may perform feature extraction to convert some input data (e.g., images, audio files, or text) into vectors, which are then used to train a cognitive model or for inference purposes, using a crossbar array structure 15, 15a. In that respect, one or more neuron layers may possibly be mapped onto the array 15, 15a according to the partition of said array. Still, the devices 12-14 may possibly collect outputs from the array, process such outputs (if necessary), and reinject them as new inputs into the array, for example to map multiple layers of a deep neural network that needs to be performed. ML operations (such as feature extraction) may notably need to perform depthwise convolutions, pooling / unpooling, etc. Similarly, a post-processing device 14 may be utilized to perform affine scaling of the output vectors, apply non-linear activation functions, etc.

[0075] More generally, the devices 12-14 may perform various operations, which depend on the actual application, and such operations may be partly performed in the client device 3 and in the intermediate device 2. Various computation strategies can be devised, which may depend on the application.

[0076] 1-4B, further aspects of the invention relating to an in-memory processing hardware device 10, 10a will now be described in detail. The functional and structural features of this device have already been described with respect to the present method. Such features will only be briefly described below.

[0077] To accommodate the method, the device 10, 10a comprises a crossbar array structure 15 as shown in Fig. 2A, for example. That is, the array 15 includes N input lines 152 and M output lines 153 interconnected at cross points defining N x M cells 155. Each cell 155 includes a corresponding memory system 157, each designed to store K weights. The array 15 is designed to perform a MAC operation.

[0078] The device 10, 10a further includes a selection circuit 159, shown (partially) in FIG. 3B, etc. The selection circuit 159 is connected to the N×M memory system 157. This circuit 159 is generally configured to select a weight from the K weights of the respective memory system and set the selected weight as the active weight, thereby enabling the N×M active weights for the N×M cells 155.

[0079] The device 10, 10a also includes input devices 151, 151a configured to apply signals encoding the N vectors to N input lines 152 of the array 15. This causes the array 15 to perform a MAC operation based on the N vectors and the corresponding N×M active weights, as enabled by the selection circuit 159, during the operation.

[0080] Furthermore, the readout device 154 is configured to read the output signals obtained at the output of the M output lines 153 and, if necessary, accumulate partial output values, as discussed above. Also, the readout device should be understood in a broad sense. For example, it may include accumulators 154, 154a and / or memory elements for storing such output values. In an analog implementation, the readout device may further include an analog-to-digital converter.

[0081] Each of the N×M memory systems 157 is preferably designed such that the K weights are independently programmable. As seen in FIG. 3B, the device 10, 10a may include a program circuit 158 ​​connected to each memory system 157. As mentioned above, only a portion of the program circuit 158, the portion connected to the corresponding memory system, is shown in FIG. 3B. In total, the program circuit 158 ​​is configured to program the K weights of each of the N×M memory systems. Because the K weights of each memory system are independently programmable, any of the K weights that are not currently set as an active weight may be (re)programmed in some cases, even if another of the K weights is currently set as an active weight, thereby enabling weights to be preemptively loaded (looked ahead) during operation.

[0082] That is, program circuitry 158 may advantageously be configured to look ahead q N×M weight sets that are not currently set as active weights, where 1≦q≦K−1, and accordingly program N×M memory systems 157 such that the latter stores the look-ahead weights in place of the q N×M weight sets. Thus, during operation, while crossbar array structure 15 is already performing a MAC operation based on the currently active weights, program circuitry 158 may program each of N×M memory systems 157 to change the weights that are not currently set as active weights.

[0083] It should be noted that the program circuit 158 ​​must be sufficiently independent from the computation circuit 15 so that it can proactively reprogram the currently inactive weights while the computation circuit 15 is performing the MAC operation based on the currently active weights. This independence allows it to proactively load those weights that will be needed for the next cycle of operation. The look-ahead operation may, for example, be performed for several sets of weights at a time. As mentioned above, various look-ahead schemes are possible.

[0084] The program circuit 158 ​​may, for example, connect the local storage device 11 to the configuration and control logic circuit 12, as assumed in FIG. 2B. In an analog implementation, the input lines 152 may in principle be reused to program the memory system 157, but it is preferable to provide a separate program circuit so that the memory system 157 can be reprogrammed during a calculation cycle. Similarly, digital memory cells (i.e. cells containing digital memory elements) can be connected to dedicated lines that embody word lines and bit lines, for example, for write operations in an SRAM memory device. It should be noted that, in contrast to the illustration in FIG. 3B, the selection circuit 159 may possibly reuse word lines and bit lines for read operations. Thus, the selection circuit and the program circuit may in fact overlap in part.

[0085] In the example of Figures 3A and 3B, each of the N x M memory systems 157 includes K different memory elements for simplicity. Each memory element is adapted to store a corresponding weight. In that case, the selection circuit 159 may include an N x M multiplexer. Each multiplexer is connected to all memory elements of the corresponding memory system 157 as shown in Figure 3B. Furthermore, a selection control line is connected to each multiplexer such that any of the K weights of each memory system 157 can be selected and set as the active weight during an operation. A selection bit can be conveyed through the control line to select the active weight as shown in Figure 3C.

[0086] In the example of FIG. 3C, the multiplexer is a channel multiplexer that uses inverters and logic "NAND" gates to arrive at a common output X. That is, a combinational logic circuit switches one of several input lines A, B, C, D to a single common output line X. The data lines A, B, C, D are represented by the W in FIG. 1、1、0 , W 1、1、1 , W 1、1、2 , W 1、1、3The data select lines (carrying the binary input address) correspond to the least significant bit (LSB) and most significant bit (MSB), respectively. 0 and Add 1 N×M multiplexers are used to switch the weights of the N×M memory system. In principle, there are at most 2×N×M control lines, i.e., two control lines per multiplexer 159, to allow individual control of each multiplexer. In practice, however, the control lines can possibly be shared across all multiplexers, if one wishes to simultaneously select the weight sets, as discussed below. Thus, all control lines are preferably shared, thereby allowing the same index K to be simultaneously selected for each element in the M×N memory system. In such a case, the number of control lines can be Log 2 It is possible to reduce it to the (K) line.

[0087] Similarly, the program circuit 158 ​​may include an N×M demultiplexer, where the same control bit lines are used for the entire array 15. Also, for simplicity, FIG. 3B shows a single demultiplexer 158 connected to a corresponding memory system 157. However, in practice there is an N×M demultiplexer 158 and an N×M multiplexer 159 connected to each memory system 157. In alternative embodiments, the program circuit 158 ​​and the selection circuit 159 may include other types of electronic components, where such components are disposed at or at least connected to each cell to program and select memory values ​​as required. In a further alternative embodiment, each memory system is configured to store K different values ​​at a corresponding local address, instead of being configured with K different memory elements.

[0088] As mentioned above, the required weights are preferably enabled all at once S70. To that end, the selection circuit 159 may advantageously be configured to select (at least) a subset of n×m weights from one of the K sets of N×M weights. This is most effectively achieved by concomitantly selecting the kth weight of the K weights of each memory system of the subset of n×m memory systems 157, where 2≦n≦N, 2≦m≦M, and 1≦k≦K. As indicated above, enabling a subarray of n×m weights may be advantageous for those matrix-vector calculations where not all N×M weights have to be switched, depending on how the problem is initially mapped onto the N×M cells 155. Note that switching operations may rarely have to be performed for a single cell (i.e., n=1 and m=1). In practice, however, weight selection will generally be performed simultaneously for a large subset of the N×M memory systems (i.e., n>1 and m>1), or even for all of the N×M memory systems where a particularly large operand matrix is ​​involved, as in the application examples discussed above with respect to Figures 5A, 5B, and 7. However, in a variant, selection circuitry 159 may systematically select an N×M weight set from one of the K N×M weight sets by concomitantly selecting the kth weight of each of the K weights of the N×M memory systems to systematically switch all memory systems 157 simultaneously. Thus, in general, selection circuitry 159 is configured to select an n×m weight set and set the latter as the active weight for the n×m memory systems of array 15, where 1≦n≦N and 1≦m≦M.

[0089] The device 10, 10a typically includes a sequencer circuit connected to the input devices 151, 151a and the selection circuit 159. The sequencer circuit orchestrates the operations of the input devices 151, 151a and the selection circuit 159 to successively perform several cycles of matrix-vector calculations as described above, i.e. such operations are based on N vectors. Each cycle of matrix-vector calculations includes one or more cycles of MAC operations (depending on whether the MAC operations performed are supplied bit-serial or not) and different N×M weight sets, the latter being selected from K N×M weight sets and set as N×M active weights in each cycle. The sequencer circuit, the program circuit 158 ​​and the input circuit 151 preferably form part of the same configuration and control logic, which typically includes the on-chip logic unit 12, as assumed in FIG. 2B. That is, the sequence functions are preferably performed by the logic unit 12, as are the other configuration and control functions.

[0090] Furthermore, the device 10, 10a may include an accumulator circuit 154, 154a configured to accumulate partial product values ​​obtained upon completion of each matrix-vector calculation. Also, in bit-serial applications, each cycle of the matrix-vector calculation includes several MAC cycles S80 due to bit-serial arithmetic, as in the embodiment discussed above with respect to FIG. 7. Now, a further accumulation S86 must be performed. In a variant relying on a parallel implementation (not requiring parallel-to-serial conversion 151a), the weight multiplication is performed in a single cycle on the complete input. In all cases, the active weights remain the same during each matrix-vector calculation cycle.

[0091] As can be seen in Figures 4A and 4B, an accumulator circuit 154, 154a can be arranged at the output of the output line 153. Accumulators are known per se. The accumulator circuit 154, 154a can notably form part of a readout device (not shown). In a variant, an accumulator can notably possibly be arranged at each cell in order to accumulate S86 the values ​​obtained during the bit-serial operation at the level of each cell.

[0092] As mentioned above, each of the N×M memory systems 157 preferably includes K distinct memory elements for simplicity. Each memory element is adapted to store a corresponding weight. Such memory elements may notably be digital memory elements, such as static random access memory (SRAM) devices. In a variant, the memory elements are analog memory elements. In that case, each multiplication-accumulation operation, i.e., Σ i W i、j、k x i is performed by analogy, and the output signal is transferred to the digital domain using an analog-to-digital converter (ADC) circuit (if necessary). The memory element may optionally be a non-volatile memory element. More generally, the invention accommodates various types of electronic memory devices (e.g., SRAM devices, flash cells, memristor devices, etc.). Any type of memristor device may be considered, such as phase change memory cells, resistive memory (RRAM), and electrochemical random access memory (ECRAM) devices.

[0093] In a preferred embodiment, each of the K memory elements is a digital memory element, such as an SRAM device. In that case, each cell 155 further includes an arithmetic unit 156 (including a multiplier and an adder tree) that is connected via a corresponding selection circuit portion 159 (e.g., via a multiplexer) to each of the K memory elements of a corresponding memory system 157. Note that although each cell is physically connected to a respective memory element via a selection circuit component (such as a multiplexer or other selection circuit component), it is logically connected to only one such element at a time due to the selection made by the selection circuit.

[0094] In the bit-serial implementation (FIG. 4A), each memory element is designed to store a P-bit weight. The input device 151 is configured to apply an input signal to provide N vector components bit-serially to the input lines 152 in P cycles (P≧2), each vector component corresponding to a P-bit input word. The N×M cells 155 must also be designed to perform the MAC operation bit-serially (i.e., in P cycles) S80. Thus, the hardware device 10 must include an accumulator circuit 154 for accumulating values ​​corresponding to partial bit-serial product values ​​obtained in each of the P cycles, which corresponds to step S86 in FIG. 7. Meanwhile, the selection circuit 159 must maintain an N×M weight set equal to the active weights during each of the P cycles.

[0095] As an example of an implementation, assume that (i) the crossbar array is an N×M=512×512 array, (ii) K=4, so that a total of four switchable N×M weight sets are available, and (iii) the input sample bit width (IBW) is equal to P=8 bits, so that the weight bit width (WBW) is also equal to 8 bits, as in FIG. 4A. The IMC device 10 takes in N vectors of 512 components (each component corresponds to an 8-bit input word). Each vector component is provided bit-serial in P=8 cycles. The IMC device 10 maps a total of 512×512×4 weights (each of 8 bits). Only one of the weight sets is active for processing during each of the P cycles S80. As mentioned above, the bit-serial cycle S80 should not be confused with the matrix-vector calculation cycles (S60-S100). In each cycle S80, all N input bits (one per row) are multiplied by M weights (one per column). The resulting partial products are expressed as 17-bit values: (WBW+Log 2 (N)=17), which are accumulated in accumulator 154 below IMC 15 for IBW=8 cycles to finally produce the full vector product, ie, the final result for this calculation cycle S80 (S88: YES).

[0096] Also, the arithmetic unit in the IMC array calculates partial inner products for each pair of N vectors and associated block matrices. The accumulator 154 may be further used to accumulate K partial products S90 before writing back the final result to external memory. The IMC switches context (weight set) at each matrix-vector calculation cycle. This process may be repeated for each input vector. For example, the programmable accumulator can be programmed to accumulate several intermediate result values, for example, obtained after shift and invert operations. Thus, when K weight sets are used in an 8-bit bit-serial IMC implementation, the accumulator 154 may internally accumulate K×8 (17-bit) values ​​that are appropriately shifted according to the repetition of the bit-serial sequence. If P=8, K=4, and N=512, the final bit width of the output accumulator is 27 bits. The 27 bits are calculated as follows: At each iteration of the bit-serial process, 512 8-bit multiplication values ​​are accumulated, which requires 17 bits to represent. The 17-bit value is shifted and accumulated over eight cycles, in which case the resultant value requires 25 bits to be fully represented. The accumulator can repeat this cycle for K=4 different weight sets, and finally requires 27 bits to represent the final result.

[0097] In general, the parameters N, M, P, and K can take on a variety of possible values. The values ​​shown above are only examples. In variants that rely on parallel implementations, such as those shown in FIG. 4B, no parallel-to-serial conversion is necessary. Rather, vector components are provided in parallel to each of the M cells of the corresponding input lines 152 via input device 151a. A complete input is performed (by weighted multiplication) in a single cycle. Accumulator 154a is used to accumulate S90 the partial products, resulting in the current N vector and associated matrix block. In that case, no intermediate accumulation is necessary for the MAC operation.

[0098] Whether based on a bit-serial or parallel implementation, the hardware device 10, 10a may incorporate a configuration and control logic 12 connected to each of the input devices 151, 151a and the selection circuit 159, as in Fig. 2B. Furthermore, a pre-data processing unit 13 may be connected to the configuration and control logic 12 (to appropriately partition the problem into N-vectors and block matrices and to command the operations). For completeness, a post-data processing unit 14 may be connected to the output of the output lines 153, for example to the output of the accumulators 154, 154a, to appropriately rearrange the output data as required and to command them to be stored in a local memory or a nearby memory (e.g. memory 11) or to return them to the external entity 2, as seen in Fig. 1.

[0099] In that respect, another aspect of the invention concerns a computing system 1. The system 1 may in particular include one or more in-memory processing hardware devices 10, 10a, as described above. The computing system may have a client / server configuration, for example as assumed in FIG. 1. That is, a user 4 may interact with a server 2 (via a personal device 3) with the aim of performing calculations. In the latter, in particular substantial matrix-matrix or matrix-vector multiplications may need to be performed, in which case the server 2 may decide to offload such calculations to a hardware device 10, 10a acting as an accelerator.

[0100] The server 2 may be considered as integrating an external memory device with an external general-purpose processing device 2, where the latter is connected to the former to read and write data to the storage device 2 during computation. Furthermore, each of the in-memory processing hardware devices 10, 10a is configured to be in data communication with the server 2, so that it can read and write data back to the storage device 2 as needed to address computational tasks sent by the server 2. It should be noted that the general-purpose processing device may be configured to optionally map an initial computational task (problem to be solved) into an N-vector and corresponding block matrix.

[0101] It should be noted that the external memory device and the general-purpose processing device form part of the same general-purpose computer 2 (i.e., a server) in the example of FIG. 1. In principle, however, the external processing device and the storage device may possibly be provided on different physical machines. In a further variant, the system 1 may also be configured as a cloud computing system, possibly using containerization techniques. That is, the present invention may in particular be embodied as a cloud computing system and may in some way be utilized as part of a cloud-based service. The system 1 may further include a composable disaggregated infrastructure, which may in particular include the hardware devices 10, 10a together with other hardware acceleration devices, e.g., ASICs and FPGAs. In all cases, the data exchange may be optimized to speed up the computation as explained above.

[0102] The above embodiments have been briefly described with reference to the accompanying drawings and may correspond to several variants. Several combinations of the above features may be conceivable. Examples are given in the following paragraphs.

[0103] 2. Specific Embodiments A particularly preferred embodiment relies on architectures such as those shown in Figures 2A, 3A and 4A with SRAM memory elements. Unlike previous crossbar arrangements, where the area is typically a high percentage of tightly coupled arithmetic units rather than memory elements and can be even more densely arranged, the area of ​​the IMC chip between arithmetic units (multipliers and adder trees) and memory elements is balanced. Apart from a better balance between memory and arithmetic area, the proposed solution (with multiple weight sets) improves the flexibility of the IMC unit 15, allowing it to natively map larger matrix-vector multiplications and to look ahead weight sets. Each weight set represents a separate block matrix. Nevertheless, the weight sets are connected to the same arithmetic unit, as can be seen in Figures 3A and 3B.

[0104] Increasingly, the proposed architecture and functionality also leads to improved efficiency (due to less interaction with external memory, see Fig. 5A and Fig. 5B) and faster execution time (due to the possibility of looking ahead weights for inactive weight sets, Fig. 6). The architecture relies in particular on an accumulator circuit, which can accumulate partial products coming from both bit-serial cycles and multiple weight sets. Thus, there is no need to write intermediate results to external memory. This reduces the amount of reads / writes significantly, i.e., from 2K-1 (in which case only one weight set can be stored locally) to 1 (in which K weight sets are stored locally).

[0105] As seen in FIG. 7, a typical computation flow is as follows, assuming a bit-serial implementation (see further FIGS. 2A-4A). A device 10 including a crossbar array structure 15 is provided in step S10, for example the device 10 is set up for data communication with a server 2. In step S20 the server 2 receives a request (from a user 4, which may be a computerized client) to perform a matrix-matrix multiplication. In step S30, a computation strategy is determined, either by the processing means of the server 2 or by the embedded processing device 13, whereby N vectors are associated with respective block matrices. In step S40, a matrix-matrix multiplication cycle is started. In step S50, the next iteration is started, whereby a given input matrix with T columns is selected. As a result of the computation strategy, K weight sets of N×M are loaded in step S55. The K weight sets correspond to the K block matrices to be used successively during the calculation cycle. Memory elements are programmed accordingly. The next input vector (of K×N elements) of the current input matrix is ​​selected in step S58 and padded if necessary. The next sub-vector of this input vector (i.e., an N-vector of N elements) is selected in step S60 along with the corresponding block matrix. In step S70, the corresponding N×M weights are locally enabled as active weights.

[0106] In step S80, a block matrix calculation is started. In step S82, a loop is started to provide the next bit (of the vector components of the current N vector) bit-serial to the N input lines of array 15. In step S84, a bit-serial MAC operation is performed. In step S86, partial results are accumulated. The process is repeated (S88: No) until all P bit-serial cycles are completed (S88: Yes). Once all P bit-serial cycles are completed, processing of the current N vector is complete.

[0107] The intermediate matrix-vector products obtained for the N vectors are accumulated S90 with previous matrix-vector products if necessary, i.e. all intermediate matrix-vector products are accumulated except for the very first one. The intermediate matrix-vector product calculation cycle (S60-S100) is repeated until all sub-vectors have been processed for multiplication by their associated block matrix (S100: yes). The loop on the input vectors (S50-S110) is repeated for all input vectors. Once all vectors have been processed (S110: yes), the final result for the current input matrix is ​​returned S120 to the calling entity 2, 13. Alternatively, this result may be stored locally until all input matrices (S50-S120) have been processed. Only then are results for all input matrices returned.

[0108] 3. Final thoughts The computerized device 10, 10a and the system 1 can be suitably designed to implement the embodiments of the invention as described herein. In that respect, it is understandable that the methods described herein are not interactive in nature, i.e., automated. The automated parts of such methods can be implemented in hardware only, or as a combination of hardware and software. In an exemplary embodiment, the automated parts of the methods described herein are implemented in software as a service or an executable program (e.g., application), the latter executed by a suitable digital processing device. However, all embodiments described herein include calculations performed by a crossbar array structure adapted to store multiple weight sets, possibly using the look-ahead and accumulation capabilities of the device 10, 10a.

[0109] Still, the methods described herein may typically include executable programs, scripts, or more generally some form of executable instructions that instruct the device 10, 10a to execute core computations. The required computer-readable program instructions can be downloaded to the processing elements, for example, from a computer-readable storage medium, over a network, for example the Internet and / or a wireless network.

[0110] Aspects of the present invention are described herein with particular reference to flowcharts and block diagrams. It will be understood that each block or combination of blocks in the flowcharts and block diagrams can be implemented by computer readable program instructions. The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices 10, 10a, as well as systems 1 including such devices, methods of operating them, and computer program products according to various embodiments of the present invention.

[0111] Although the present invention has been described with reference to a limited number of embodiments, variants, and accompanying drawings, it will be understood by those skilled in the art that various modifications may be made and equivalents may be substituted without departing from the scope of the present invention. In particular, features recited in a given embodiment (either device-like or method-like) may be combined with or substituted for other features in another embodiment, variant, or drawing without departing from the scope of the present invention. Thus, various combinations of the features described with respect to any of the above embodiments or variants may be contemplated that remain within the scope of the appended claims. Furthermore, many minor modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from the scope of the present invention. Therefore, it is not intended that the invention be limited to the particular embodiments disclosed, but rather that the invention shall include all embodiments that are within the scope of the appended claims. Moreover, many other variants other than those explicitly mentioned above may be contemplated. For example, other types of memory elements, selection circuits, and program circuits may be contemplated.

Claims

1. 1. A method of in-memory processing, comprising: providing (S10) a crossbar array structure (15) including N input lines (152) and M output lines (153) interconnected at cross points defining N×M cells (155), said cells (155) including respective memory systems (157) designed to store K weights, where K≧2, whereby said crossbar array structure (15) includes N×M memory systems storing K N×M weight sets; enabling (S70) N×M active weights for the N×M cells (155) by selecting a weight from the K weights for each of the memory systems (157) and setting the selected weight as an active weight; applying (S82) signals encoding a vector of N components to the N input lines (152) of the crossbar array structure (15) to cause the latter to perform (S84) multiply-accumulate or MAC operations based on the vector and the N×M active weights; Reading (S86, S90) the output signals obtained at the outputs of the M output lines (153) to obtain corresponding values; 23. An in-memory processing method comprising:

2. 2. The method of claim 1, further comprising: while performing a MAC operation according to the N×M weights that are currently enabled as active weights, if 1≦q≦K−1, looking ahead (S115) to q sets of N×M weights to be used next, instead of the q sets of N×M weights that were previously active, and storing the looked ahead weights in the N×M memory system (157).

3. 3. The method of claim 1 or 2, wherein the N×M active weights are enabled (S70) by concomitantly selecting a k-th weight of the K weights for each memory system (157) of at least a subset of the N×M memory systems (157), where 1≦k≦K, and setting the respective weight selected accordingly as a currently active weight.

4. The method includes performing (S58-S110) several matrix-vector calculation cycles, each of which comprises: activating (S70) a new N×M set of active weights for the N×M cells (155) by selecting a weight from the K weights for each of the memory systems (157) and setting the selected weight as an active weight; applying (S82) signals encoding a vector of N components to the N input lines (152) of the crossbar array structure (15) and causing the latter to perform (S84) a MAC operation based on the vector and the new N×M active weight set; Reading (S86, S90) the output signals obtained at the outputs of the M output lines (153) to obtain corresponding values; The method according to any one of claims 1 to 3, comprising:

5. 5. The method of claim 4, wherein each of the cycles further comprises: accumulating (S90) partial product results corresponding to the read output signals, whereby accumulation is performed successively.

6. 6. The method of claim 5, further comprising: prior to completing K of the number of matrix-vector calculation cycles, looking ahead (S115) q N×M sets of weights, for 1≦q≦K−1, and storing the latter in the N×M memory system (157) in place of the q N×M sets previously enabled as active weights.

7. 7. The method according to claim 5 or 6, further comprising, upon completion of the number of matrix-vector calculation cycles (S110: Yes), returning to an external storage device (2) (S120) a result obtained based on the successive accumulations.

8. The method includes performing (S50-S110) a K×T matrix-vector calculation cycle, where T corresponds to a number of input vectors, each of which is decomposed into K sub-vectors of N components and associated with K respective block matrices, the latter corresponding to K N×M weight sets, whereby the K×T matrix-vector calculation cycle can be loading (S55) K N×M weight sets corresponding to the K respective block matrices and programming the memory system (157) accordingly to store the K N×M weight sets; and For each of the K sub-vectors of each of the T input vectors (S60), activating (S70) an N×M active weight corresponding to an associated one of the K each block matrix as a currently active weight; applying (S82) signals encoding vectors corresponding to each of the sub-vectors to the N input lines (152) to cause the crossbar array structure (15) to perform (S84) a MAC operation based on each of the sub-vectors and the currently active weights; and 8. The method of claim 7, performed by reading the output signals obtained at the outputs of the M output lines (153) to obtain corresponding fractional values.

9. Reading the output signal includes accumulating (S90) the partial values ​​obtained for each sub-vector with previously obtained partial values ​​for previous ones of the K sub-vectors, if any, to obtain an updated result; The method of claim 8 , further comprising: returning (S120) a result obtained based on the last obtained updated result.

10. In the external processing device, Prior to programming (S55) said N×M memory system (157) according to K N×M weight sets and encoding a certain number of sub-vectors into input signals for the purpose of applying (S82) such input signals to said N input lines (152) to perform said number of matrix-vector calculation cycles, The method of claim 4 , further comprising mapping (S30) a given challenge to the sub-vectors and the K N×M weight sets.

11. the N×M memory system (157) being a digital memory system; each of the N×M cells (155) further comprising an arithmetic unit (156) connected to a corresponding one of the N×M memory systems (157); 11. A method according to any one of claims 1 to 10, wherein the MAC operation is preferably performed bit-serial over P cycles, P≧2 (S84), where P corresponds to the bit-width of each of the N components of each of the vectors used at input, whereby partial product values ​​are accumulated on completing each of the P cycles (S86).

12. 12. A computer program for in-memory processing, said computer program product comprising a computer readable storage medium having embodied thereon program instructions, said program instructions being executable by processing means of an in-memory processing hardware device (10) to cause the latter to perform the steps of any one of claims 1 to 11.

13. An in-memory processing hardware device (10), comprising: a crossbar array structure (15) including N input lines (152) and M output lines (153) interconnected at cross-points defining N×M cells (155), where N≧2 and M≧2, said cells (155) including respective memory systems (157) each designed to store K weights, where K≧2, whereby said crossbar array structure includes N×M memory systems (157) adapted to store K N×M sets of weights for performing multiply-accumulate or MAC operations; a selection circuit (159) connected to the N×M memory systems (157), the selection circuit (159) configured to select a weight from the K weights for each of the memory systems and set the selected weight as an active weight to enable N×M active weights for the N×M cells (155); an input device (151) configured to apply signals encoding a vector of N components to the N input lines (152) of the crossbar array structure (15) to cause the latter to perform a MAC operation based on the vector and the N×M active weights enabled by the selection circuit (159); a readout device (154) configured to read out the output signals obtained at the outputs of said M output lines (153); An in-memory processing hardware device (10) comprising:

14. Each of the N×M memory systems (157) is designed such that the K weights are independently programmable; 14. The in-memory processing hardware device of claim 13, wherein the device further comprises a program circuit coupled to the respective memory system, the program circuit configured to program the K weights of the N×M memory system.

15. The program circuit (158) Look ahead to a set of q N×M weights that are not currently set as active weights; 15. The in-memory processing hardware device (10) of claim 14, further comprising: programming said N×M memory system (157) accordingly such that the latter is configured to store the look-ahead weights in place of q number of N×M weight sets, where 1≦q≦K−1.

16. each of the N×M memory systems (157) includes K memory elements each adapted to store a corresponding one of the K weights; 16. The in-memory processing hardware device of claim 13, wherein the selection circuitry (159) includes N×M multiplexers each connected to each of the K memory elements of a corresponding one of the N×M memory systems (157), and a selection control line connected to each of the multiplexers such that any one of the K weights of each of the memory systems (157) is selectable and settable as the active weight during an operation.

17. 17. The in-memory processing hardware device (10) of any one of claims 13 to 16, wherein the selection circuitry (159) is further configured to select a subset of n×m weights from one of the K N×M weight sets by concomitantly selecting a kth weight of the K weights for a respective memory system (157) of the subset of n×m memory systems (157) of the N×M memory systems, where 2≦n≦N, 2≦m≦M, and 1≦k≦K.

18. The in-memory processing hardware device (10) further comprises a sequencer circuit and an accumulator circuit (154); the sequencer circuit is connected to the input device (151) and the selection circuit (159) to sequentially perform several cycles of matrix-vector calculation based on one or more vector sets, and organizes the operations of the input device (151) and the selection circuit (159), wherein during operation, each of the cycles of matrix-vector calculation includes one or more cycles of MAC operation, and a different N×M weight set is selected from the K N×M weight sets and set as N×M active weights in each of the cycles of matrix-vector calculation; 18. The in-memory processing hardware device (10) of any one of claims 13 to 17, wherein the accumulator circuit (154) is configured to accumulate partial product values ​​resulting from completing each MAC operation cycle.

19. 20. The in-memory processing hardware device (10) of claim 18, wherein the accumulator circuit (154) is located at an output of the output line (153).

20. 19. The in-memory processing hardware device (10) of any one of claims 13 to 18, wherein each of the NxM memory systems (157) includes K memory elements, each adapted to store a corresponding one of the K weights.

21. each of the K memory elements of each of the N×M memory systems (157) being digital memory elements; 21. The in-memory processing hardware device (10) of claim 20, wherein each of the NxM cells (155) further includes an arithmetic unit (156) coupled to each of the K memory elements of a corresponding one of the NxM memory systems (157) via a corresponding portion of the selection circuitry (159).

22. each of the K memory elements of each of the N×M memory systems (157) is designed to store a weight of P bits; said input device (151) is configured to, by applying said signal, bit-serially supply a vector of N components to said input lines (152) in P cycles, each of said N components corresponding to a P-bit input word, where P≧2; the N×M cells (155) are configured to perform a MAC operation bit-serially in the P cycles; The hardware device (10) further includes an accumulator circuit (154) configured to accumulate values ​​corresponding to partial bit-serial product values ​​obtained in each of the P cycles; 22. The in-memory processing hardware device (10) of claim 21, wherein the selection circuit (159) is configured to maintain the same NxM set of weights as active weights during each of the P cycles.

23. The in-memory processing hardware device (10) comprises: a configuration and control logic (12) connected to each of said input device (151) and said selection circuit (159); a pre-data processing unit (13) connected to said configuration and control logic (12); a post-data processing device (14) connected at the output of said output line (153); The in-memory processing hardware device (10) of any one of claims 13 to 22, further comprising:

24. A computing system (1) comprising one or more in-memory processing hardware devices (10), each as described in any one of claims 13 to 23.

25. The system further comprises a storage device (2) and a general-purpose processing device (2) connected to the storage device and reading and writing data from and to the storage device (2), Each of the in-memory processing hardware devices (10) is configured to read and write data to the storage device (2); 25. The computing system of claim 24, wherein the general purpose processing unit (2) is configured to map a given computational task to vectors and weights to a memory system (157) of the one or more in-memory processing hardware devices (10).

Images