In-memory processing of vector-matrix multiplications by a hardware accelerator using local multimode decompression of matrix weights

Abstract

The invention is notably directed to a method of processing data at a hardware accelerator device (or "accelerator" for short). The accelerator (1) includes processing means (2) and an in-memory computing (IMC) device (10). The IMC device has a crossbar array structure of cells, which are designed to store weight values. The method comprises the following steps, which are performed at the accelerator. To start with, data encoding decompression parameters and weight values are received at the accelerator. The weight values are assumed to have been compressed according to a multimode compression scheme, whereby the received data include distinct subsets of the weight values that have been compressed according to respective compression modes. Next, the distinct subsets of the weight values are decompressed using said processing means (2), based on respective decompression modes that are selected in accordance with the decompression parameters. The decompression modes correspond to said compression modes. I.e., a given decompression mode can be regarded as the reciprocal of the corresponding compression mode. The decompressed weight values are subsequently stored across the cells. Eventually, vector-matrix multiplications are performed through to the crossbar array structure (10) as multiply-accumulate operations based on the stored weight values. The above solution leads to reduced data transmission and local storage requirements. Compressed data are transmitted more efficiently and more rapidly to the accelerator, something that is notably beneficial to machine learning (ML) applications. Also, the accelerator can locally store more compressed data (all things being otherwise equal), prior to decompressing such data. Interestingly, the received data can be proactively decompressed in situ, while the IMC device performs VMMs based on weight values that are currently active. Furthermore, the VMMs are efficiently performed, in-memory, by the crossbar array structure of the IMC device. The invention is further directed to related accelerator devices and computer program products.

Description

[0001] IN-MEMORY PROCESSING OF VECTOR-MATRIX MULTIPLICATIONS BY A HARDWARE ACCELERATOR USING LOCAL MULTIMODE DECOMPRESSION OF MATRIX WEIGHTS

[0002] TECHNICAL FIELD

[0003] The invention relates in general to the field of in-memory processing methods, devices, and computer program products. In particular, it is directed to methods wherein data encoding decompression parameters and weight values are streamed to a hardware accelerator device, which decompresses the weight based on a multimode decompression scheme, the modes of which are selected in accordance with the decompression parameters, with a view to performing vector-matrix multiplications at the hardware accelerator device.

[0004] BACKGROUND

[0005] Artificial neural networks (ANNs) such as deep neural networks (DNNs) have revolutionized the field of machine learning (ML) by providing unprecedented performance in solving cognitive data-analysis tasks. ANN operations mostly involve vector-matrix multiplications (VMMs), which account for 70 to 90% of the total neural network operations, irrespective of the ANN architecture. VMM operations pose multiple challenges, because of their recurrence, universality, compute, and memory requirements. Traditional computer architectures are based on the von Neumann computing concept, according to which processing capability and data storage are split into separate physical units. Such architectures suffer from congestion and high-power consumption, as data must be continuously transferred from the memory units to the control and arithmetic units through interfaces that are physically constrained and costly.

[0006] One possibility to accelerate VMMs is to use dedicated hardware acceleration devices, such as in-memory processing circuits having a crossbar array structure. This type of circuit includes input lines and output lines, which are interconnected at cross-points defining cells. The cells contain respective memory devices (or sets of memory devices), which are designed to store respective matrix coefficients. Vectors are encoded as signals applied to the input lines of the crossbar array to perform the VMMs by way of multiply-accumulate (MAC) operations. Such an architecture can map VMMs simply and efficiently. The weights are updated (i.e., replaced) by reprogramming the memory elements, to perform successive VMMs. Such an approach breaks the “memory wall” as it fuses the arithmetic- and memory unit into a single in-memory- computing (IMC) unit, whereby processing is done much more efficiently in or near the memory (i.e., the crossbar array). Plus, an IMC unit provides a solution to the Von-Neumann Bottleneck on the instruction interface as a single instruction may suffice to operate VMMs over multiple cycles.

[0007] Such devices are particularly suited for accelerating the execution of ANNs. However, neural networks (NNs) sometimes include millions to billions of parameter values, leading to significant memory and bandwidth requirements. Now, as the present inventors have realized, weight matrices of large NNs are typically sparse. They can also be made sparse without losing application accuracy, using pruning techniques. Moreover, mapping large NNs onto large IMC devices sometimes leads to severe mapping inefficiencies. Thus, the present inventors set themselves the challenge of optimizing hardware accelerator devices having such IMC units, and methods of operation of such accelerator devices.

[0008] SUMMARY

[0009] According to a first aspect, the present invention is embodied as a method of processing data at a hardware accelerator device (or “accelerator” for short). The accelerator includes processing means and an in-memory computing (IMC) device. The IMC device has a crossbar array structure of cells, which are designed to store weight values. The method comprises the following steps, which are performed at the accelerator. To start with, data encoding decompression parameters and weight values are received at the accelerator, e.g., from an external computer connected to the accelerator. The weight values are assumed to have been compressed according to a multimode compression scheme, whereby the received data include distinct subsets of the weight values that have been compressed according to respective compression modes. Next, the distinct subsets of the weight values are decompressed using said processing means, based on respective decompression modes that are selected in accordance with the decompression parameters. The decompression modes correspond to said compression modes. I.e., a given decompression mode can be regarded as the reciprocal of the corresponding compression mode. The decompressed weight values are subsequently stored across the cells. Eventually, vector-matrix multiplications are performed through to the crossbar array structure as multiply-accumulate operations based on the stored weight values. The above solution has multiple advantages. First, it leads to reduced data transmission and improved storage capacity: compressed data are transmitted more efficiently and more rapidly to the accelerator, something that is notably beneficial to machine learning (ML) applications, where new weight values may have to be frequently transmitted to the accelerator. And since the data received by the accelerator are compressed, the accelerator can receive and store more data locally (all things being otherwise equal), prior to decompressing such data; this reduces the frequency of exchanges with the external device. Second, the decompression is performed by processing means of the accelerator, i.e., close to the crossbar array structure, in which the decompressed weight values are eventually stored. I.e., the decompressed weight values can accordingly be efficiently transferred to the crossbar array structure . Third, the received data can be proactively decompressed in situ, while the IMC device performs VMMs based on weight values that are currently active (i.e., those values that were previously stored across the cells), as discussed below. Thus, newly decompressed weight values can be immediately stored in the cells, as soon as needed. Fourth, the VMMs are efficiently performed, in-memory, by the crossbar array structure of the IMC device. For completeness, the present approach lends itself well to an offline exploration of optimal compression parameters and allows flexibility in the number of compression schemes that can be used.

[0010] As evoked above, in embodiments of the method, the distinct subsets of the weight values are proactively decompressed by the processing means, while performing VMMs, through to the crossbar array structure, based on previously decompressed weight values. This makes it possible to avoid stall times between successive computation cycles by the IMC device.

[0011] Preferably, the set of weight values included in the received data includes N * M weight values, the crossbar array structure contains an array of N x M cells, which are designed to store the N x M weight values received, for the stored weight values to span a 2D array of N x M values, where N > 2 and M > 2 . Said distinct subsets of the weight values correspond to respective, distinct areas of the 2D array, where each of said distinct areas contains a plurality of contiguous values. Now, such areas may advantageously benefit from distinct compression schemes, hence the benefit of the present approach.

[0012] In preferred embodiments, the N x M cells are designed to store K sets of N x M weight values, wherein the K sets map K respective 2D arrays of N x M values, where K > 1, N> 2, and M> 2. The N x M weight values included in the received set of weight values are decompressed and stored as one of the K sets of N x M weight values, so as to map said 2D array of TV x M values, which corresponds to a respective one of said 2D arrays of N x M values. In that case, the method further comprises enabling one of the K sets of N x M weight values as active weight values, prior to performing said VMMs based on the active weight values. Dynamically switching between weight sets at the accelerator makes it possible to reduce the frequency of data exchanges with external devices.

[0013] As noted in the background section, mapping large neural networks (NNs) onto large weight- stationary matrix-vector multipliers can lead to severe mapping inefficiencies, notably if the mapped neural layers do not fully match the characteristics of the array(s). Some mapped layers might need less input / output channels, therefore multiple layers can be mapped into the same IMC array. To manage this, the mapped layers can be padded, e.g., zero-padded, resulting in large swaths of structured data (e.g., zeros) that further increase the required memory storage, in addition to unstructured data, hence the need of optimal compression strategies.

[0014] In that respect, the present compression modes preferably include two modes (hereafter called a first mode and a second mode), which have the following properties. The first mode is a nocompression mode, whereby one or more first subsets of said subsets of the weight values are not compressed. Said one or more first subsets respectively correspond to one or more first areas of said distinct areas. They comprise weight values whose sparsity is less than a predetermined threshold value. The second mode is a net compression mode, whereby one or more second subsets of said subsets of the weight values are compressed according to said second mode, with a compression ratio that is strictly larger than 1. This can notably be advantageous where padding values are needed. Consistently, the decompression modes include a first decompression mode and a second decompression mode, respectively corresponding to said first mode and said second mode. As a result, the one or more first subsets of weight values are not decompressed, while the one or more second subsets of weight values are decompressed according to the second decompression mode.

[0015] As noted above, the received data may possibly contain padding values, each equal to a given constant (e.g., zero). In that case, the data have a high structured sparsity. Therefore, the second mode can be optimized specifically for compressing padding values. For example, the second mode can be based on a run length encoding scheme, e.g., a zero-run length encoding scheme where said given constant is equal to zero, which is very efficient at compressing sets of data with high structured sparsity. The run length encoding scheme preferably has a variable run length. In preferred embodiments, said compression modes further include a third mode, which is a net compression mode distinct from the second mode, whereby one or more third subsets of said subsets of the weight values are compressed according to said third mode, with a compression ratio that is strictly larger than 1. Consistently, said decompression modes further include a third decompression mode corresponding to said third mode, whereby the one or more third subsets of weight values are decompressed according to the third decompression mode. Having distinct net compression modes makes it possible to optimize datasets having different sparsity patterns and, in particular, adequately handle weight value subsets presenting both structured and unstructured sparsity.

[0016] Preferably, the third mode is a frequent value compression scheme, whereby the one or more third subsets encode respective masks of binary values as well as respective, non-zero values, and the third decompression mode causes to decode the one or more masks of binary values, so as to decompress weight values in the one or more third subsets in accordance with the one or more decoded masks of binary values. This algorithm compresses very efficiently values having a high unstructured sparsity.

[0017] In embodiments, the one or more third subsets of weight values are decompressed using an iterative process, which offers a more tractable implementation of the third decompression mode. This iterative process includes several iterations. Each iteration of the several iterations comprises: buffering a data chunk of said one or more third subsets; proactively decoding binary values of a mask of binary values for a next one of the several iterations; and decoding data in a data chunk as buffered during a preceding iteration of said several iterations, based on a mask of binary values as decoded during said preceding iteration.

[0018] Preferably, said compression modes consist of the first mode, the second mode, and the third mode. This, in practice, makes it possible to satisfactorily handle datasets having diverse sparsity structures.

[0019] In preferred embodiments, the received data include one or more headers and several body sections, the latter corresponding to respective ones of the distinct subsets of the weight values. The weight values are encoded in the body sections, while the one or more headers encode the decompression parameters corresponding to the distinct subsets of the weight values, as well as sizes of the body sections. In embodiments, the one or more headers consist of a single header (i.e., a header packed in a single address), which encodes the sizes of the body sections, in addition to the decompression parameters. Preferably, though, the headers include a main header and several subheaders, i.e., headers in different addresses. In that case, each of the several subheaders precedes a respective body section of the body sections and encodes decompression parameters pertaining to said respective body section. These are two different examples of data formats, thanks to which the compressed data can hold in the local memory of the accelerator.

[0020] In that respect, the method preferably comprises storing the data received in a local memory device of the accelerator, prior to decompressing the distinct subsets of the weight values, using said processing means. The processing means read the distinct subsets of the weight values directly from the local memory device, for the purpose of decompressing the weight values.

[0021] According to another aspect, the invention is embodied as a computer program for in-memory processing. The computer program product comprises a computer readable storage medium having program instructions embodied therewith, where the program instructions are executable by processing means of an accelerator. The latter further includes an IMC device having a crossbar array structure of cells designed to store weight values. In operation, executing the program instructions causes the hardware accelerator device to perform steps according to the present methods, i.e., receiving data encoding decompression parameters and weight values, decompressing the distinct subsets of the weight values based on respective decompression modes, storing the decompressed weight values across the cells, and performing VMMs as MAC operations based on the stored weight values.

[0022] According to a final aspect, the invention is embodied as an accelerator, which includes processing means, an IMC device having a crossbar array structure of cells designed to store weight values, and an interface unit, which is configured to receive data from an external entity. The processing means are configured to cause the accelerator to access data received by the interface unit, wherein, in operation, the accessed data encode decompression parameters and weight values, the latter compressed according to a multimode compression scheme, whereby the received data include distinct subsets of the weight values that have been compressed according to respective compression modes. In addition, the processing means are configured to cause the accelerator to decompress, using said processing means, the distinct subsets of the weight values based on respective decompression modes selected in accordance with the decompression parameters received, the decompression modes corresponding to said compression modes. The processing means are further configured to cause the accelerator to instruct to store the decompressed weight values across the cells and subsequently perform, through to the crossbar array structure, VMMs as MAC operations based on the stored weight values.

[0023] In embodiments, the processing means are further configured to cause the accelerator to proactively decompress said distinct subsets of the weight values, while the crossbar array structure performs VMMs based on previously decompressed weight values.

[0024] Preferably, the accelerator further comprises a memory device, whereby the data received are stored in the memory device, in operation. The processing means are further configured to directly read the data stored in the memory device to subsequently decompress the distinct subsets of the weight values, in operation.

[0025] In preferred embodiments, the accelerator is designed as a system on a chip, in which the memory device, the interface unit, the processing means, and the IMC device, are all cointegrated.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. The illustrations are for clarity in facilitating one skilled in the art in understanding the invention in conjunction with the detailed description. In the drawings:

[0028] FIG. 1 is a diagram schematically illustrating selected components of a hardware accelerator device including an in-memory compute (IMC) unit, according to embodiments;

[0029] FIG. 2A is a high-level schematic of the IMC unit of the hardware accelerator device of FIG. 1, as in embodiments. FIG. 2B illustrates the crossbar array structure of the IMC unit of FIG. 2A, where the crossbar array structure contains an array of N x M cells designed to store four sets of N x M weight values, as in embodiments. The four sets of N x M weight values map four 2D arrays of N x AT weight values, as illustrated in FIG. 2C;

[0030] FIG. 3 is a flowchart illustrating high-level steps of a method of processing data at a hardware accelerator device such as depicted in FIGS. 1, 2A, and 2B, according to embodiments;

[0031] FIG. 4 is a flowchart illustrating high-level steps of a decompression scheme implemented at the hardware accelerator device of FIG. 1, where several decompression modes are decoded, including a no-decompression mode and two net decompression modes, as in embodiments; FIGS. 5A and 5B are diagrams schematically illustrating how distinct subsets of the weight values transmitted to the hardware accelerator device can be mapped onto respective areas of a 2D array of values, which are eventually stored in the N x M cells of the crossbar array structure, where the distinct subsets of weight values are initially compressed according to respective compression modes, as in embodiments. The subsets of weights have different sparsity properties, hence the benefit of applying different compression modes, prior to transmitting the weight values to the hardware accelerator device. FIGS. 6A and 6B show the respective compression ratios that can be achieved in each case, according to various compression strategies; and

[0032] FIGS. 7 and 8 show examples of data structures of the data transmitted to the hardware accelerator device, as in embodiments. In the example of FIG. 7, the data structure comprises a main header, as well as several subheaders preceding respective data substreams. In the example of FIG. 8, the data structure comprises a single header, followed by several data substreams. Each substream corresponds to a respective weight subset, according to embodiments.

[0033] The accompanying drawings show simplified representations of devices or parts thereof, as involved in embodiments. Technical features depicted in the drawings are not necessarily to scale. Similar or functionally similar elements in the figures have been allocated the same numeral references, unless otherwise indicated.

[0034] Devices, methods, and computer program products embodying the present invention will now be described, by way of non-limiting examples.

[0035] DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

[0036] The following description is structured as follows. General embodiments and high-level variants are described in section 1. Section 2 addresses particularly preferred embodiments. Section 3 concerns technical implementation details. Note, the present method and its variants are collectively referred to as the “present methods”. All references Sn refer to methods steps of the flowcharts of FIGS. 3 and 4, while numeral references pertain to devices, components, and concepts involved in embodiments of the present invention. 1. General embodiments and high-level variants

[0037] A first aspect of the invention is now described in detail in reference to FIGS. 1 - 4, 7, and 8. This aspect concerns a method of processing data at a hardware accelerator device (hereafter referred to as an “accelerator”) 1. As seen in FIG. 1, the accelerator 1 basically includes processing means 2 (e.g., a digital processing unit) and an in-memory computing (IMC) device 10. The IMC device 10 includes a crossbar array structure 15 of cells 154 that are designed to store weight values. The crossbar array structure of an IMC device is known as such: it includes input lines and output lines 151, 152, which are interconnected at cross-points (i.e., junctions), which define cells 154. The cells 154 include respective memory systems 156. Such a structure allows in situ computations, such that the accelerator can be regarded as an in-memory processing device. The accelerator 1 actually concerns another aspect of the invention, which is described later in detail.

[0038] The method includes a series of steps, which are performed at the accelerator 1. To start with, the accelerator 1 receives data 100, which encode decompression parameters and weight values, see step SI 8, FIG. 3. The weight values included in the data 100 received by the accelerator 1 are values that have been compressed according to a multimode compression scheme. That is, the received data includes distinct subsets of the weight values that have been compressed according to respective compression modes, see FIGS. 7 and 8. The multimode compression is performed offline, i.e., prior to sending the data to the accelerator 1, as explained later in detail.

[0039] Such data are received from an external entity 20 (e.g., a connected computer), which is distinct from the accelerator 1, see FIG. 1. The accelerator 1 typically comprises an interface unit 4, which interfaces with and receives data from this external entity 20. Preferably, the interface unit 4 writes the received data to a local memory device 6 of the accelerator 1, see FIG. 1, such that the processing means 2 can read data directly from this local memory 6.

[0040] Next, the distinct subsets of the weight values are decompressed (step S21) using the processing means 2 of the accelerator 1. That is, the processing means 2 implement a decompression engine at the level of the accelerator 1. The decompression may for instance be triggered automatically upon receiving the data 100. The decompression is performed based on respective decompression modes, where the decompression modes are selected (see step S212 in FIG. 4), on the fly, in accordance with the decompression parameters included in the received data. The decompression modes correspond to the compression modes used to compress the data, initially (i.e., offline, prior to transmitting such data to the accelerator). That is, any selected decompression mode DM, can be regarded as the reciprocal of a corresponding compression mode CM,. In other words, this decompression mode DM, performs an operation that is the reciprocal of this compression mode CM,. As a result, an initial data chunk Do that is compressed through CM, and then decompressed through DM, results in Do.

[0041] The decompressed weight values are then stored S23 across the cells 154. Eventually, vectormatrix multiplications (VMMs) are performed S26 through to the crossbar array structure, as multiply-accumulate MAC operations. Such operations are performed based on the weight values as stored across the cells 154 of the crossbar array structure 15. That is, the stored weight values act as matrix coefficients. The VMMs are typically triggered by injecting input signals encoding input values (capturing vector components) into input lines 152 of the crossbar array structure 15.

[0042] The present solution revolves around applying a multimode decompression scheme to data received at the accelerator device, with a view to subsequently performing VMMs based on the decompressed weight data. The multimode decompression is performed by the processing means 2 of the accelerator, in accordance with the decompression parameters included in the received data. That is, the various decompression modes are dynamically selected (i.e., on-the- fly), according to the parameters included in the received data. To that aim, the decompression parameters are associated with respective subsets of the weight values, such that the decompression modes can be suitably selected and applied to the relevant subsets of the weight values.

[0043] Datasets may be repeatedly streamed (i.e., sent) to the accelerator device 1 for it to repeatedly and continually perform VMMs, as for instance needed for performing ML inferences. The decompression modes, which are determined by the parameters included in the datasets successively received at the accelerator 1, can actually vary from one dataset to the other. I.e., different sets of decompression modes may possibly be applied for different datasets, although the offline compression relies on a fixed, predetermined numbed of distinct compression modes. The compression modes are preferably applied in a deterministic manner, whereby the compression modes are selected in accordance with properties of the weight data, e.g., their sparsity and the locality of the data sparsity, and then accordingly applied, offline, to subsets of the weight values, prior to sending compressed data to the accelerator 1. The sparsity can be defined as the ratio of the number of zero values to the total number of values in a set of data. In the present context, a multimode compression scheme involves at least two different compression modes, where each mode refers to a specific compression algorithm. Similarly, a decompression mode refers to a decompression algorithm that can be regarded as the reciprocal of the compression algorithm used in the corresponding compression mode. In principle, any number of compression modes may be involved, depending on properties of the weight data involved. However, use is preferably made of a fixed, predetermined number of compression modes, as noted above.

[0044] In practice, the compression modes include at least one net compression mode, applied to one or more subsets of the data sent to the accelerator 1. A net compression mode means that the compression ratio is strictly larger than 1, where the compression ratio is the ratio of the uncompressed data size to the compressed data size. The corresponding subsets of weight values are identified as such in the received data and subsequently decompressed by the processing means 2, in accordance with the corresponding decompression parameters included in the received data.

[0045] Still, one of the compression modes may possibly be a no-compression mode. And this mode may concern one or more subsets of the weight values, as in embodiments. Correspondingly, the multimode decompression scheme may involve a decompression mode, according to which one or more subsets of uncompressed weights are identified as such, whereby the corresponding weight values are just pulled and eventually stored in respective cells 154 of the array, without undergoing any decompression.

[0046] Furthermore, the multimode (de)compression scheme may involve one or more additional net (de)compression modes, distinct from the first net (de)compression mode, as in embodiments. Actually, in preferred embodiments, the compression modes involved consist of exactly three modes, i.e., a no-compression mode and two net compression modes. In principle, however, any number of (de)compression modes may be contemplated, as suggested by FIG. 4.

[0047] The proposed solution takes advantage of properties of the weight data, starting with the sparsity structure of the weight data, as for instance observed for typical weight values of neural networks. As the present inventors realized, such data can advantageously benefit from a multimode compression, which is performed offline, i.e., outside of the accelerator 1 and prior to transferring the compressed weight values to the accelerator 1. The compressed data are thus more efficiently and more rapidly transmitted to the accelerator, where they are decompressed, with a view to subsequently performing VMMs that are efficiently performed as MACs by the IMC device 10.

[0048] Thus, the present solution has multiple advantages. First, the proposed solution leads to reduced data transmission: compressed data are transmitted faster from the external device 20 to the accelerator 1, which is beneficial as new weight values may have to be frequently transmitted to the accelerator. I.e., the above steps S18, S21, S23, S26 can be repeatedly performed, in particular in ML applications. Since the data received by the accelerator 1 are compressed, the accelerator can receive and store more data locally (e.g., on the local memory device 6), prior to decompressing such data. This reduces the frequency of exchanges with the external entity 20. In particular, the offline compression improves the chance to be able to transmit a weight dataset pertaining to one or more entire neural layers in one go. Second, the decompression is performed by processing means 2 of the accelerator 1, i.e., close to the crossbar array structure 15, in which the decompressed weight values are eventually stored. I.e., the time required to transfer the decompressed weight values to the crossbar array structure 15 is reduced. Third, the received data can be proactively decompressed, while the IMC device performs VMMs based on weight values that are currently active (i.e., those values that were previously stored across the cells 154), as in embodiments. Thus, newly decompressed weight values can be immediately stored in the cells 154, as soon as needed, as further discussed later in detail. Fourth, the VMMs are efficiently performed, in-memory, by the crossbar array structure 15 of the IMC device 10. In the present context, the VMMs can indeed be considered as performed in-memory, insofar as the weight values are stored where the operations take place, i.e., in the cells 154 of the crossbar array.

[0049] All this is now described in detail, in reference to particular embodiments of the invention. To start with, the weight data can be proactively decompressed, at the accelerator 1, to reduce idle times between successive computation cycles by the IMC device 10. That is, the distinct subsets of the weight values that form part of a received dataset can be proactively decompressed S21 by the processing means 2, while the IMC device 10 performs S26 VMMs through to the crossbar array structure 15 based on previously decompressed weight values, see FIG. 3. Thus, as soon as computations based on the currently active weight set (i.e., a weight set as previously stored across the cells 154) completes (step S22, FIG. 3, meaning that this set can now be replaced), freshly decompressed weight values can immediately be written S23 to the cells 154. Thus, new VMMs can henceforth be performed based on such weight values. As usual with IMC crossbars, VMMs are performed by injecting input signals encoding input values through input lines 151 of the crossbar array structure 15 and reading signals obtained in output of the output lines 152 of the crossbar array structure, see FIGS. 2A and 2B. The VMMs are performed based on weight values as stored across the cells 154, where such weight values are obtained through the multimode decompression scheme applied to the data received by the accelerator 1. The crossbar array is designed so that each cell 154 can store at least one weight value. That is, the crossbar array structure includes an array of V * AT cells 154, which can store at least N * AT weight values, where N> 2, and M> 2. In that case, N and AT correspond to the numbers of input lines 151 and output lines 152, respectively.

[0050] In embodiments, the received data include N x M weight values, which are eventually stored S23 across the N x AT cells 154. The N x AT cells 154 may be designed to store a single set of N x AT weight values. The number of decompressed weight values can be calibrated to match, and thus entirely populate, the cells. Once stored across the cells 154, the weight values map a 2D array of dimension N x M. The distinct subsets of weight values, which have been decompressed according to respective decompression modes, correspond to respective areas of this 2D array, where each of the distinct areas is formed of contiguous values of the 2D array. For example, the distinct subsets of weight values may correspond to respective areas A, B, C, D, E, F of the 2D array, as illustrated in FIGS. 5A and 6A. These areas will typically be rectangular areas, to facilitate the management of the corresponding data structures. Such areas are complementary, meaning that, as a whole, the distinct areas map the entire 2D array.

[0051] As the skilled person understands, the subsets of weight values corresponding to the distinct areas A, B, C, D, E, F may advantageously benefit from distinct compression schemes, because of the different properties the weight values have in these areas. For instance, the received data 100 may include weight values of one or several neural network layers, which may not exactly fit the 2D array enabled by the cells 154. In that case, the weight values are padded with constant values (typically zeros) to fill the residual gaps. Now, such padding values may be efficiently compressed according to a given compression scheme, while the neural network weight values may more advantageously be compressed according to another compression scheme, or not be compressed at all, as the case may be, hence the advantage of using a multimode compression scheme.

[0052] The above example assumes that each cell 154 can store a single weight value. However, in more sophisticated embodiments, the crossbar array structure 15 is designed so that each cell can store up to K weight values, where K> 2, see FIG. 2B. For example, K may be equal to 4, as assumed in FIG. 2C. That is, the crossbar array structure 15 may include TV x M cells capable of storing K sets of N x AT weight values, where K> 2, N> 2, and M > 2. As a whole, the cells 154 can store K N AT weight values, which are arranged according to K bidimensional arrays, each having a dimension N x AT, as illustrated in FIG. 2C. The advantage is that the weight values may be locally switched, i.e., at the accelerator 1, without having to first transmitting the corresponding data, i.e., without idle times. Note, new weight values may still be proactively fetched and decompressed, as background tasks, while executing VMMs based on a currently active set of weight values.

[0053] Thus, in embodiments, any set of N x M weight values (as included in a compressed form in the received set of weight values) can be decompressed S21 and stored S23 as one of the K sets of 7V x AT weight values, so as to map a respective 2D array of N x AT values. In that case, the method further comprises enabling S24 one of the K sets of N x M weight values as active weight values, prior to performing S26 VMMs based on the active weight values. That is, a set of A x M weight values map exactly one 2D array and such 2D arrays can be loaded one at a time, in one of the K sets, while performing VMMs based on a currently active array, which corresponds to a different value of K. Only one of the K sets of weight values is active at a time: the IMC device performs VMM operations based on the weight values as stored in the currently active array.

[0054] For example, as illustrated in FIG. 2C, the N x AT cells 154 may be designed to store four sets of A x AT weight values (i.e., K = 4). Each dataset newly received includes A x AT weight values. Thus, the received data span a whole weight set, which is meant to be stored in one of the K sets. This way, the weight values can be proactively loaded and decompressed, one array at a time, to avoid or reduce stall times. In each dataset received, each of the distinct subsets of weight values received corresponds to a respective area A, B, C, D, E, F in one of the 2D arrays mapped by the K sets of weight values, see FIGS. 5A and 6A. Again, these areas are distinct and complementary in each of the 2D arrays and each of the distinct areas is formed of contiguous weight values.

[0055] The above example assumes that the weight values received as part of one dataset can map one 2D array at a time. In variants, the received data may be dimensioned to map Q arrays of weight values, where 1 < Q < K. In all cases, the received weight values can still be proactively decompressed, and then timely loaded across the cells 154. At the very beginning, the equivalent of K sets of weight values may be received (in one or several steps), decompressed, and then loaded across the cells 154 so as to fill the capacity of the cells, prior to initiating VMM computation cycles.

[0056] One or more neural layers may possibly be mapped onto each 2D array of N x M values, see FIG. 2C. Since the cells 154 can, as a whole, store V sets ofN x M weight values, the cells 154 may well store weight values corresponding to multiple layers of an artificial neural network (ANN), whereby the same accelerator 1 can be used to implement several ANN layers, one after the other. That said, the mapping of the neural layers depends on the dimensions of the neural layers relative to the dimensions of the crossbar array structure. For instance, only portions of ANN layers may be mapped onto the N x M cells 154, depending on the dimensions of the problem at hand. More generally, several mapping strategies can be contemplated.

[0057] In embodiments, the compression modes include two modes, i.e., a first mode and a second mode, which respectively correspond to a no-compression mode and a net compression mode. The first mode (no-compression) concerns one or more first subsets of the weight values, which are not compressed. The first subsets of the weight values correspond to respective first areas in the 2D array (if K = 1) or in each of the 2D arrays (should K be strictly larger than 1). The sparsity properties of the weight values of these first subsets may happen to be such (e.g., the sparsity is less than a predetermined threshold value) that the best compression approach may turn out not to compress these values at all. I.e., the no-compression mode is applied to those data, whose properties do not lead to a significant compression. Conversely, the second mode (net compression) concerns one or more second subsets of the weight values, the sparsity properties of which may advantageously call for a net compression mode, i.e., with a compression ratio that is strictly larger than 1. Consistently, the decompression modes include a first decompression mode S213 and a second decompression mode S214 (see FIG. 4), which respectively correspond to the first compression mode and the second compression mode. Thus, in operation, the first subsets of weight values are not decompressed (consistently with the no-compression mode), while the second subsets of weight values are decompressed according to the second decompression mode S214.

[0058] In embodiments, the compression modes are initially determined by assessing (offline) the sparsity and the locality of the sparsity of the data. In turn, optimal compression schemes can be determined in accordance with the determined sparsity structures. The optimal compression modes are preferably determined ex ante, once and for all, in accordance with the different types of sparsity structures identified. Thus, there is no need to systematically apply and compare compression modes online, before choosing and applying such compression modes. A convenient approach is to select the compression modes in accordance with the local sparsity value of local subsets of weight values. As noted above, it may be preferable not to compress the values should their sparsity be less than a predetermined threshold value. This threshold value is predetermined, in accordance with expected benefits of compression. In some cases, the sparsity may not enable any meaningful compression. I.e., applying a compression scheme in such cases may not significantly improve or may even degrade the effective compression ratio, e.g., give rise to compression ratios that are less than 1, meaning that the data inflate instead of being effectively compressed.

[0059] Conversely, the received data may contain padding values, each equal to a given constant, which calls for a net compression. In that case, the padding values are compressed according to a net compression mode (second mode). The second mode may for instance be based on a run length encoding scheme. Preferably, the second compression mode is based on a zero-run length encoding scheme, should the padding value (the given constant) be equal to zero. In that case, the compressed data is composed by the value (zero) of the run length encoding algorithm and the length of the zero data sequence. More preferably, the second compression mode is a modified zero-run length encoding scheme, having a variable run length, where the maximum run length can for instance be set to the maximum value by the decompression engine.

[0060] In embodiments, the compression modes further include a third mode, which is again a net compression mode, albeit distinct from the second compression mode. As a result, one or more third subsets of the received weight values are compressed according to this third compression mode, with a compression ratio that is again strictly larger than 1. Consistently, the decompression modes further include a third decompression mode S215, which correspond to the third compression mode. In operation, the third subsets of weight values are decompressed according to the third decompression mode S215. Like the first and second subsets of weight values, the third subsets may correspond to any of the subsets of the weight values received. Note, the terminologies “first subset”, “second subset”, and “third subset”, do not imply any particular processing order. The data subsets can be decompressed in any order, or even in parallel, the processing means 2 permitting.

[0061] In embodiments, the third compression mode is a frequent value compression (FVC) scheme. In that case, the third subsets encode respective masks of binary values as well as respective, non-zero values. The third decompression mode S215 causes to decode the one or more masks of binary values, so as to decompress weight values in the one or more third subsets of weight values received, in accordance with the one or more decoded masks of binary values. As known per se, each of the binary values indicates a presence of a zero-weight value or a non-zero weight value. The masks of binary values are included (i.e., mixed) in the weight data.

[0062] As further seen in FIG. 4, the third decompression mode is preferably implemented as an iterative process. I.e., the third subsets of weight values are decompressed S215 thanks to this iterative process. Each iteration of this process comprises several steps S2152, S2154, and S2156. During the first step S2152, a data chunk of the third subsets is buffered. Next, step S2154, binary values are proactively decoded. Such values concern the mask of binary values to be used during the next iteration. At step S2156, the data contained in the data chunk that was buffered during the previous iteration are decoded based on the mask of binary values as decoded S2154 during the previous iteration.

[0063] The above examples involve three particular compression modes. In principle, however, the present approach may involve any number of compression modes. That said, preferred embodiments rely on exactly three compression modes, i.e., the first mode, the second mode, and the third mode, as described above. Relying on these three modes is a good trade-off between simplicity and efficiency in practice. The performance of the multimode decompression approach proposed here can be appreciated in FIGS. 5B and 6B. This approach outperforms other compression methods tested for the simple weight area partitions shown in FIGS. 5 A and 6 A. The relative improvements obtained in terms of compression ratios range from 8% to 64%, when compared with standard run-length encoding (RLE) compression schemes RLE / ? (where n = 4, 8, and 16) and FVC compression methods.

[0064] Again, the fact that a fixed number of compression modes is provided does not imply that the data received will systematically be compressed according to exactly three compression modes. Indeed, the compression modes selected for the offline compression may depend on properties of the actual data. As a result, some of the datasets received may happen to be compressed according to only one or two compression modes. That said, the majority of the datasets received will typically happen to involve data compressed according to two, if not three, different modes.

[0065] As illustrated in FIGS. 7 and 8, the received data 100 may include one or more headers and several body sections S 1 - S3, where the body sections S 1 - S3 correspond to respective subsets of the weight values. The weight values are encoded in the body sections SI - S3, while the one or more headers encode the decompression parameters corresponding to the distinct subsets of the weight values, as well as sizes of the body sections SI - S3. Note, the number of body sections SI - S3 contained in each transmitted dataset will typically exceed the number of compression modes, because several sections of weight values may possibly be compressed according to the same compression mode.

[0066] The headers may for example decompose into a main header and several subheaders, as illustrated in FIG. 7. In this example, each subheader precedes a respective body section and encodes decompression parameters pertaining to that body section. This option is easier to implement in practice. For instance, the main header may include a parameter value (“Stream length”) that determine the length of the dataset (referred to as a stream in FIG. 7), while each subheader may include decompression parameter values, e.g., pertaining to the length of the respective compressed subset (“Comp, substream length”), the length of the corresponding, uncompressed subset (“Uncomp, substream length”), as well as other decompression parameter values of the associated decompression scheme (“Scheme 1”, “Scheme 2”, “Scheme 3”, etc.), if necessary.

[0067] Alternatively, the headers may comprise a single header, as assumed in FIG. 8. In this case, the single header encodes the sizes of the body sections SI - S3, in addition to the corresponding decompression parameters (“Substream 1 info”, “substream 2 info”, ..., substream n info”). This header may further include the overall dataset length (“Stream length”), as assumed in FIG. 8. This option reduces the overall data size. In each case (FIGS. 7 and 8), a body section S / include X, data chunks, referred to as “Substream z data where j = 1 to X.

[0068] The data structure examples shown in FIGS. 7 and 8 involve a single header or a main header followed by subheaders corresponding to respective body sections. However, it will be apparent to the one skilled in the art that various alternative implementations can be contemplated.

[0069] Next, according to another aspect, the invention can be embodied as a computer program product for in-memory processing. The computer program product comprises a computer readable storage medium having program instructions embodied therewith, where the program instructions are executable by processing means 2 of the accelerator 1. As explained above, the accelerator 1 further includes an IMC device 10 having a crossbar array structure of cells 154 designed to store weight values. In operation of the accelerator, executing the program instructions causes the accelerator to perform steps of the present methods, i.e., receive data encoding decompression parameters and weight values, decompress the distinct subsets of the weight values based on respective decompression modes selected in accordance with the decompression parameters received, store the decompressed weight values across the cells, and perform VMMs as MAC operations through to the crossbar array structure. Additional features of this computer program product are discussed in section 3.

[0070] Referring back to FIGS. 1 - 2C, a final aspect of the invention is now discussed in detail, which concerns an accelerator 1. As explained above in reference to other aspects of the invention, the accelerator 1 includes processing means 2, which are basically configured as a decompression engine. The accelerator further includes an IMC device 10, which has a crossbar array structure of cells 154 designed to store weight values, as well as an interface unit 4, configured to receive data from an external entity 20, see FIG. 1. Consistently with the previous aspects of the invention, the processing means 2 are configured to cause the accelerator 1 to access data received by the interface unit 4, decompress the weight values according to a multimode decompression scheme, and instruct to store the decompressed weight values across the cells to subsequently perform VMMs.

[0071] In operation, the accessed data encode decompression parameters and weight values. The weight values are assumed to have been compressed according to a multimode compression scheme. That is, the received data include distinct subsets of the weight values that have previously been compressed according to respective compression modes. At the level of the accelerator 1, the distinct subsets of the weight values contained in the received data 100 are decompressed using the processing means 2, based on respective decompression modes that are selected S212 in accordance with the decompression parameters received as part of the data 100.

[0072] Once decompressed, the accelerator 1 instructs to store the weight values across the cells 154, although this instruction may be implicit. E.g., the decompressed weight values may be written to a specific portion of a local memory and then transferred from this portion to the cells 154. In all cases, the decompressed weight values are eventually stored across the cells 154, which makes it possible to subsequently perform the VMMs as MAC operations through the crossbar array structure 15. As explained earlier, such operations are based on the stored weight values (matrix coefficients) and are typically triggered upon injecting input signals encoding input values (vector components) into the input lines 152 of the crossbar array structure 15. As illustrated in FIG. 1, the accelerator 1 preferably includes a memory device 6. In operation, the data received by the interface unit 4 are stored in the memory device 6. Next, in order to decompress the subsets of weight values, the processing means 2 can read the subsets of weight values directly from the memory device 6. I.e., the decompression engine reads data from the local memory 6, close to the IMC device 10, and not from the external entity 20. This makes it possible to quickly transfer the decompressed weight values to the cells 154. Also, the weight values can be preloaded in the memory device 6 and proactively decompressed. Thus, the processing means 2 are preferably configured to directly read the data stored in the memory device 6, with a view to subsequently decompress said distinct subsets of the weight values. Moreover, the processing means 2 can be configured to cause the accelerator 1 to proactively decompress the distinct subsets of the weight values, while the crossbar array structure performs VMMs based on previously decompressed weight values.

[0073] The above embodiments have been succinctly described in reference to the accompanying drawings and may accommodate a number of variants. Several combinations of the above features may be contemplated. Examples are given in the next section.

[0074] 2. Particularly preferred embodiments

[0075] 2.1 System on a chip and IMC device

[0076] In embodiments, the accelerator 1 is designed as a system on a chip (SoC), in which the memory device 6, the interface unit 4, the processing means (decompression engine) 2, and the components of the IMC device 10, are all co-integrated. Such components are typically integrated on the same substrate or microchip. In addition, the SoC may include memory interfaces and other interfaces, as usual with SoCs.

[0077] The IMC unit 10 may for instance be designed substantially as shown in FIG. 2A. An input / output (VO) unit 19 is used to interface the IMC device 10 with the local memory device 6 and the decompression engine 2. The crossbar array structure 15 includes N input lines 151 and M output lines 152, where N> 2 and M > 2. At least two input lines and two output lines are required to form an array. In practice, however, the number of input lines 151 and output lines 152 will typically be larger than or equal to 12 or 16, so that compression may start being beneficial when using header structures as described above, taking into account the overhead induced by the headers. That said, variants can be contemplated, in which use in made of optimized header vs. body sections. For instance, dense packing formats can be contemplated, in which the header information (and subheader information, if necessary) is encoded together in one address line. In such cases, the information needed can be encoded into a few bytes only, which ensures compression rates that are strictly larger than 1, even for a 2 * 2 matrix or a 2 * 3 matrix.

[0078] In preferred embodiments, however, the number of input lines 151 and output lines 152 will typically be on the order of several hundreds to thousands of lines. For example, arrays of 256 x 256, 512 x 512, or 1024 x 1024, may be contemplated, although TV need not necessarily be equal to M. The input lines and output lines 151, 152 are interconnected at cross-points (i.e., junctions), which define N x AT cells 154. The cells 154 include respective memory systems 156. Overall, the crossbar array structure 15 includes N x AT memory systems, which are capable of storing K sets of N x AT weights, i.e., K N AT weights in total, where K may for instance be equal to 4 (as assumed in FIG. 2C), 8, 16, or 32. In other words, the crossbar array structure 15 includes N x AT cells 154 in a crossbar configuration, where each cross-point of the crossbar configuration corresponds to a cell and each cell involves a memory system 156 capable of storing K weights. Such weights are noted Wijjc in FIG. 2B, where i runs from 1 to TV, j from 1 to AT, and k from 1 to K.

[0079] The memory system 156 stores weights that correspond to matrix elements used to perform the VMMs. Each memory system 156 may for instance includes serially connected memory elements, which store respective bits of the weight stored in the corresponding cell; the MAC operations are performed in a bit-serial manner in that case. The memory elements may for instance be static random-access memory (SRAM) devices. However, the crossbar array structure 15 may, in principle, be equipped with other types of electronic memory devices (e.g., flash cells, memristive devices, etc.). Any type of memristive devices can be contemplated, such as phase-change memory cells (PCM), resistive random-access memory (RRAM), as well as electro-chemical random-access memory (ECRAM) devices.

[0080] Such memory elements may form part of a multiply-and add circuit (as suggested in FIG. 2B), whereby each column includes N multiply-and add circuits, so as to efficiently perform the MAC operations. As per the crossbar configuration, M MAC operations are being performed in parallel, during each calculation cycle. Note, the operations performed at every cell correspond to two scalar operations, i.e., one multiplication and one addition, as suggested in FIG. 2B. Thus, the M MAC operations imply N x M multiplications and N x M additions, meaning 2 x A x M scalar operations in total. Output signals obtained in the M output lines 152 are subsequently read out to obtain corresponding values. In practice, several calculation cycles need often be successively performed, whereby weights are locally enabled (i.e., selected and set as active weights) at each cycle, prior to feeding components of an A-vector to perform MAC operations and read the output values. Such output values may correspond to partial values, which may advantageously be accumulated locally, e.g.., at the readout circuitry 16 or the near-memory processing device 18. I.e., the readout operations may not only aim at extracting the output values, but also accumulating them with previous output values, if necessary.

[0081] The crossbar array structure shown in FIG. 2B allows distinct sets of weights to be locally enabled at the crossbar array 15, which makes it possible to locally perform rotations of the weights and accordingly reduce the frequency of data exchanges with the memory device 20. This, in turn, reduces idle times of the device 10. That is, some intermediate weight updates are avoided, because up to K successive computation cycles can be performed without the need to transfer new sets of weights. Nevertheless, new sets of weights can be proactively decompressed, e.g., during each computation cycle.

[0082] In principle, the accelerator 1 may be used to perform VMMs for any type of applications, e.g., including ML or linear optimization applications. In preferred applications, the accelerator is used to execute an ANN. In that case, the crossbar array structure 15 implements up to M artificial neurons at a time. Although physically limited to M neurons, the structure 15 can nevertheless be used to map larger vector-matrix multiplications, in several steps. Once a set of A x M weights has been enabled as active weights, vector components can be injected into the crossbar array structure 15. More precisely, signals encoding a vector of N components (hereafter referred to as an A- vector) are applied by the input unit 11 to the A input lines 151 of the crossbar array structure 15. This causes the crossbar array structure 15 to perform MAC operations based on the A- vector and the A x M active weights as currently enabled. The MAC operations result in that the values encoded by the signals fed into the A input lines are multiplied by the currently active weight values.

[0083] Signals are obtained in output of the M columns 152 of the crossbar 15. The obtained signals encode the neuron output values (or first values). The corresponding values are read out by a dedicated circuitry 16 and passed to the near-memory processing unit(s) units 18. One or more near-memory processing units 18 may actually be arranged in output of the output lines 152 to efficiently evaluate mathematical functions (such as activation functions) applied to the neuron outputs. The process repeats for each successive neural layer. If the current neural layer is the last layer, then the algorithm may return the last activation function values with a view to forming an inference result.

[0084] In principle, the processing means 2 may include, or consist of, any digital processing unit. Preferably, however, the processing unit 2 is a custom digital accelerator designed to implement a combination of combinatorial and sequential logic, and optimized for high-throughput decompression, e.g., of up to 8 - 1024 bits of decompressed data per cycle. The memory device 6 can be a fast on-chip memory, preferably a volatile, random access memory device like an SRAM (Static Random Access Memory) device, or an embedded DRAM (Dynamic Random Access Memory) device. The device 6 may also be a non-volatile memory, e.g., an embedded Flash device, an RRAM (Resistive Random Access Memory) device, or an STT-MRAM (Spin- Torque Transfer Magnetic Random Access Memory) device. More generally, this device 6 can consists of, or include, any memory device based on other read- and writable, embedded on- chip memory technology.

[0085] 2.2 Preferred flows

[0086] 2.2.1 FIG. 3

[0087] As seen in FIG. 3, all necessary weight data are initially compressed at an external entity 20 (the external computer performing the offline compression). At step S12, uncompressed weight values are loaded in the external computer 20 and then compressed S14 according to a multimode compression scheme. Step S14 may include a sub-step of scanning the weight values to assess the sparsity structure of the data and accordingly select suitable compression schemes for subsets of the weight values. At step SI 6, a datafile is created, which encodes decompression parameters and the compressed weight values. The datafile is subsequently transferred S18 to the accelerator 1. Steps S12 - S18 can be repeatedly performed, as necessary to continually feed the accelerator 1 for it to repeatedly perform VMMs. Note, steps S12 - S18 can thus be performed concomitantly with the steps S20 - S29 described below.

[0088] At the level of the accelerator device 1, the datafile is received by the interface unit 4 and the data contained in the received datafile are written to the local memory device 6, step S20. The weight values are then proactively decompressed (step S21) by the processing means 2 according to a multimode decompression scheme, thanks to decompression parameters contained in the received data 100, by reading and extracting all the necessary data directly from the local memory device 6. Meanwhile, the processing device 2 monitors S22 (Yes / No) the VMMs performed by the IMC device 10. Once computations based on the currently active set of weight values have completed (S22: Yes), the processing means 2 instructs to write S23 the freshly decompressed weight values in the cells 154, in place of a previously enabled set of weight values that are now disused; this set could be any of the K sets of weight values stored in the crossbar array structure 15, other than the currently active set.

[0089] At the level of the IMC device 10, once computations based on the currently enabled set of weight values have completed (S22: Yes), the IMC device enables S24 a new set of weight values as active weight values; this set of weight values does not necessarily correspond to weight values written last to the cells 154. From this point on, the IMC device 10 can select S25 a next input vector to inject corresponding signals through the input unit 11 and accordingly perform S26 VMMs as MAC operations. The output vector is read S27 by the readout circuitry 16. Such values may possibly be accumulated at the circuitry 16 (see the loop back to S24) and, eventually, further processed by the near-memory processing unit(s) 18 (e.g., to compute activations), as the case may be. Eventually, the results are transferred S28 to the local memory device 6. Further computation cycles S24 - S27 may take place, if necessary. The final results are eventually transferred S29 to the external entity 20.

[0090] 2.2.2 FIG. 4

[0091] Compressed data 100, which are received by the accelerator 1 at step S20 (see FIG. 3), are accessed by the processing means 2 (i.e., a processing unit 2 configured as a decompression engine) to perform the decompression step S21, also shown in FIG. 3. Step S21 decomposes into several sub-steps, which are performed either iteratively or in parallel for each subset of weight values, i.e., each body section SI - S3 (see FIGS. 7 and 8).

[0092] First, the processing unit 2 analyses the header(s) to locate the body sections and the corresponding decompression parameters. This makes it possible to decode S212 the compression mode that was initially applied to each body section. Then, for each body section, the processing unit 2 chooses the decompression mode in accordance with the decompression parameters retrieved at step S212. In the example of FIG. 4, the unit 2 selects one of L decompression modes, which notably include a no-decompression mode S213, a run length decoding scheme S214, and a frequent value decompression mode S215. The frequent value decompression scheme (third mode S215) is preferably performed as an iterative process, where each iteration comprises steps S2152, S2154, and S2156, as described in section 1. The selected decompression mode is then applied to obtain uncompressed weight values. The decompressed weight values are composed at step S216 to form a 2D array of weight values 110, with a view to subsequently storing S22 - S23 this array of weight values in the cells 154 of the crossbar array structure 15.

[0093] The present approach is to compress the weight values transmitted to the cells. By compressing these values, the overall model size can be drastically reduced, enabling efficient deployment on resource-constrained devices, such as edge computing systems. Reducing the size of the weight values allows more data to be stored on, e.g., cheap and fast on-chip memory, instead of relying on expensive and slow external memories. Reducing the overall size of the data to be transmitted leads to an energy reduction, lower system latency, and higher system performance.

[0094] There is no limit in the number of substreams. The selection of the substreams and the corresponding compression schemes can be done offline based on the sparsity of the identified sections. The proposed solution does not limit the lengths of the compressed and uncompressed sub streams. This can be adapted to the capabilities of the devices used.

[0095] 3. Technical implementation details

[0096] The present accelerators 1 may form part of a larger computer system, involving a server, which interacts with clients, who may be natural persons (interacting via personal computers), processes, or machines. This system may actually include a plurality of accelerators 1. Each accelerator 1 may thus be configured to read data from, and write data to, the memory unit of the server computer in this example. Client requests are managed by the server, which may notably be configured to map a given computing task onto vectors and weights, which are then passed to the accelerators 1. The overall computer system may for instance be configured as a composable disaggregated infrastructure, which may further include other hardware acceleration devices, e.g., application-specific integrated circuits (ASICs) and / or field- programmable gate arrays (FPGAs). Of course, many other architectures can be contemplated. For example, this computer system may be configured as a standalone system or as a computerized system connected to one or more general-purpose computers. The system 1 may notably be used in a distributed computing system, such as an edge computing system.

[0097] Computerized devices 1 and systems as described herein can be designed for implementing embodiments of the present invention as described herein, starting with the methods described in sections 1 and 2. In that respect, it can be appreciated that such methods are essentially non- interactive, i.e., automated. Automated parts of such methods can be implemented in hardware only, or as a combination of hardware and software. In exemplary embodiments, automated parts of the methods described herein are implemented in software, as a service or an executable program (e.g., an application), the latter executed by suitable digital processing devices, including the accelerators 2 and external entities 20.

[0098] All embodiments described herein involve computational steps that are performed thanks to non-conventional hardware such as an IMC device 10 having a crossbar array structure 15. However, the methods described herein may further involve executable programs, scripts, or, more generally, any form of executable instructions, which are executed by the processing means 2, here configured as a decompression engine, and an external entity 20, which can be a conventional computer. The required computer readable program instructions can for instance be downloaded to processing elements from a computer readable storage medium, via a network, for example, the Internet.

[0099] Aspects of the present invention are described herein notably with reference to a flowchart and block diagrams. It will be understood that each block, or combinations of blocks, of the flowchart and the block diagrams can be implemented through computer readable program instructions. The flowchart and the block diagram in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of accelerators 2, and methods of operating them, according to various embodiments of the present invention.

[0100] While the present invention has been described with reference to a limited number of embodiments, variants, and the accompanying drawings, it will be understood by those skilled in the art that various changes may be made, and equivalents may be substituted without departing from the scope of the present invention. In particular, a feature (device-like or method-like) recited in a given embodiment, variant or shown in a drawing may be combined with or replace another feature in another embodiment, variant, or drawing, without departing from the scope of the present invention. Various combinations of the features described in respect of any of the above embodiments or variants may accordingly be contemplated, that remain within the scope of the appended claims. In addition, many minor modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from its scope. Therefore, it is intended that the present invention is not limited to the particular embodiments disclosed, but that the present invention will include all embodiments falling within the scope of the appended claims. In addition, many other variants than explicitly touched above can be contemplated. For example, other types of memory elements and processing units can be contemplated.

Claims

CLAIMS1. A method of processing data at a hardware accelerator device (10) that includes processing means (2) and an in-memory computing device (15), the latter having a crossbar array structure of cells (154) designed to store weight values, wherein the method comprises, at the hardware accelerator device (10): receiving (SI 8) data (100) encoding decompression parameters and weight values, wherein the weight values have been compressed according to a multimode compression scheme, whereby the received data include distinct subsets of the weight values that have been compressed according to respective compression modes; decompressing (S21), using said processing means (2), the distinct subsets of the weight values based on respective decompression modes selected (S212) in accordance with the decompression parameters, the decompression modes corresponding to said compression modes; storing (S23) the decompressed weight values across the cells (154); and performing (S26), through to the crossbar array structure, vector-matrix multiplications (VMMs) as multiply-accumulate (MAC) operations based on the stored weight values.

2. The method according to claim 1, wherein said distinct subsets of the weight values are proactively decompressed (S21) by the processing means (2), while performing (S26) VMMs through to the crossbar array structure, based on previously decompressed weight values.

3. The method according to claim 1 or 2, wherein the set of weight values included in the received data includes N x M weight values, the crossbar array structure contains an array of N x M cells (154), which are designed to store the N x M weight values received, for the stored weight values to span a 2D array of N x M values, where N> 2 and M> 2, andsaid distinct subsets of the weight values correspond to respective, distinct areas of the 2D array, where each of said distinct areas contains a plurality of contiguous values.

4. The method according to claim 3, wherein the N x M cells (154) are designed to store K sets of N x AT weight values, the K sets mapping K respective 2D arrays of N x AT values, where K > 1, N> 2, and M > 2, the TV x A / weight values included in the received set of weight values are decompressed (S21) and stored (S23) as one of the K sets of N x AT weight values, so as to map said 2D array of N x AT values, corresponding to a respective one of said 2D arrays of TV x AT values, and the method further comprises enabling (S24) one of the K sets of N x AT weight values as active weight values, prior to performing (S26) said vector-matrix multiplications based on the active weight values.

5. The method according to claim 3 or 4, wherein said compression modes include: a first mode, which is a no-compression mode, whereby one or more first subsets of said subsets of the weight values are not compressed, said one or more first subsets respectively corresponding to one or more first areas of said distinct areas, and comprising weight values whose sparsity is less than a predetermined threshold value; and a second mode, which is a net compression mode, whereby one or more second subsets of said subsets of the weight values are compressed according to said second mode, with a compression ratio that is strictly larger than 1, and said decompression modes include a first decompression mode (S213) and a second decompression mode (S214), respectively corresponding to said first mode and said second mode, whereby the one or more first subsets of weight values are not decompressed, while the one or more second subsets of weight values are decompressed according to the second decompression mode (S214).

6. The method according to claim 5, whereinthe received data contain padding values, each equal to a given constant, the padding values compressed according to said second mode, and said given constant is preferably equal to zero.

7. The method according to claim 6, wherein said second mode is based on a run length encoding scheme, which preferably has a variable run length, and said second mode is more preferably based on a zero-run length encoding scheme, said given constant being equal to zero.

8. The method according to claim 6, wherein said compression modes further include a third mode, which is a net compression mode distinct from the second mode, whereby one or more third subsets of said subsets of the weight values are compressed according to said third mode, with a compression ratio that is strictly larger than 1, and said decompression modes further include a third decompression mode (S215) corresponding to said third mode, whereby the one or more third subsets of weight values are decompressed according to the third decompression mode (S215).

9. The method according to claim 8, wherein the third mode is a frequent value compression scheme, whereby the one or more third subsets encode respective masks of binary values as well as respective, non-zero values, and the third decompression mode (S215) causes to decode the one or more masks of binary values, so as to decompress weight values in the one or more third subsets in accordance with the one or more decoded masks of binary values.

10. The method according to claim 9, wherein the one or more third subsets of weight values are decompressed (S215) using an iterative process of implementation of the third decompression mode, said iterative process includes several iterations, and each iteration of the several iterations comprises: buffering (S2152) a data chunk of said one or more third subsets proactively decoding (S2154) binary values of a mask of binary values for a next one of the several iterations, anddecoding (S2156) data in a data chunk as buffered during a preceding iteration of said several iterations, based on a mask of binary values as decoded (S2154) during said preceding iteration.

11. The method according to claim 8, 9, or 10, wherein said compression modes consist of the first mode, the second mode, and the third mode.

12. The method according to any one of claims 1 to 11, wherein the received data (100) include one or more headers and several body sections (SI - S3), the latter corresponding to respective ones of the distinct subsets of the weight values, the weight values encoded in the body sections (SI - S3), and the one or more headers encode the decompression parameters corresponding to the distinct subsets of the weight values, as well as sizes of the body sections (SI - S3).

13. The method according to claim 12, wherein the one or more headers consist of a single header, which encodes the sizes of the body sections (SI - S3), in addition to the decompression parameters.

14. The method according to claim 12, wherein the one or more headers include a main header and several subheaders, and each of the several subheaders precedes a respective body section of the body sections (SI - S3) and encodes decompression parameters pertaining to said respective body section.

15. The method according to any one of claim 1 to 14, wherein the method further comprises storing the data received in a memory of the hardware accelerator device (10), prior to decompressing the distinct subsets of the weight values, using said processing means (2), the latter reading the distinct subsets of the weight values directly from the memory.

16. A computer program for in-memory processing, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by processing means (2) of ahardware accelerator device (10), which further includes an in-memory computing device (15) having a crossbar array structure of cells (154) designed to store weight values, to cause the hardware accelerator device to: receive data encoding decompression parameters and weight values, the latter compressed according to a multimode compression scheme, whereby the received data include distinct subsets of the weight values that have been compressed according to respective compression modes; decompress, using said processing means (2), the distinct subsets of the weight values based on respective decompression modes selected (S212) in accordance with the decompression parameters received, the decompression modes corresponding to said compression modes; store the decompressed weight values across the cells (154); and perform, through to the crossbar array structure, vector-matrix multiplications (VMMs) as multiply-accumulate (MAC) operations based on the stored weight values.

17. A hardware accelerator device (10), which includes processing means (2), an in-memory computing device (15) having a crossbar array structure of cells (154) designed to store weight values, and an interface unit (4), configured to receive data from an external entity (20), wherein the processing means (2) are configured to cause the hardware accelerator device (10) to: access data received by the interface unit (4), wherein, in operation, the accessed data encode decompression parameters and weight values, the latter compressed according to a multimode compression scheme, whereby the received data include distinct subsets of the weight values that have been compressed according to respective compression modes; decompress, using said processing means (2), the distinct subsets of the weight values based on respective decompression modes selected (S212) in accordance with the decompression parameters received, the decompression modes corresponding to said compression modes; and instruct to store the decompressed weight values across the cells (154) and subsequently perform, through to the crossbar array structure, vector-matrix multiplications (VMMs) as multiply-accumulate (MAC) operations based on the stored weight values.

18. The hardware accelerator device (10) according to claim 17, wherein the processing means (2) are further configured to cause the hardware accelerator device (10) to proactively decompress said distinct subsets of the weight values, while the crossbar array structure performs vector-matrix multiplications based on previously decompressed weight values.

19. The hardware accelerator device (10) according to claim 17 or 18, wherein the hardware accelerator device (10) further comprises a memory device (6), whereby the data received are stored in the memory device (6), in operation, and the processing means (2) are further configured to directly read the data stored in the memory device (6), with a view to subsequently decompress said distinct subsets of the weight values, in operation.

20. The hardware accelerator device (10) according to claim 19, wherein the hardware accelerator device (10) is designed as a system on a chip, in which the memory device (6), the interface unit (4), the processing means (2), and the IMC device, are all co-integrated.

Images