Methods and systems for compressing sparse matrices

Abstract

Described is an improved method of for compressing a sparse matrix. The method comprises reading a first subset of one or more values from the sparse matrix comprising at least one zero and performing a logical bitwise operation on each value to identify the at least one zero. The method comprises writing the first subset of values to an output array, and incrementing an offset counter based on a number of identified non-zeros. The method then comprises reading a second subset of one or more values from the sparse matrix containing at least one non-zero value and performing a logical bitwise operation on each value in the second subset to identify the at least one non-zero value, and writing the second subset to the output array at a position defined by the offset counter, comprising overwriting at least one zero in the output array with at least one non-zero value.

Description

[0001] METHODS AND SYSTEMS FOR COMPRESSING SPARSE MATRICES

[0002] FIELD OF THE INVENTION

[0003] The present disclosure relates to the field of compressing sparse matrices, in particular to efficient methods of compressing sparse matrices into the compressed sparse row (CSR) format.

[0004] BACKGROUND

[0005] In the last few years, deep neural networks (DNNs) have become the de facto and state-of-the-art technique for use in many machine learning and artificial intelligence (Al) tasks. Hence, DNNs have become more and more popular on a variety of devices such as high-performance computing, mobile and loT (internet of things) devices. DNNs comprise several layers, where each layer is configured to learn a distinct set of features. The data that passes between successive layers is called “activations” or “features”. DNNs are computationally intensive and consume a significant amount of energy due to memory access, and the size of some DNNs (which may comprise a large number of layers, where each layer may comprise a huge number of activations).

[0006] Generally, the performance of a network may be improved by increasing the number of layers. For example, the capability of a DNN to learn non-linear features can increase with the number of hidden layers in a network. However, DNN size comes at the cost of increased latency and power consumption. The computations for performing inference and / or training of a DNN are dominated by matrix multiplications (including performing convolution layers, which are also computed, in practice, by matrix multiplication).

[0007] These costs become more and more expensive and in some cases cause a barrier for real-time applications due to the increased time of executing matrix multiplications. There are several known approaches to tackle this issue;

[0008] Quantization: This method approximates a DNN by using a more continuous representation of weights and activations (which are originally represented using floating point values) into a finite set of values, for example, and ‘int8’ representation. This reduces the overall memory usage and thus decreases latency and speeds up runtime. However, quantization comes are the cost of reduced accuracy.

[0009] Weight pruning (offline): Portions of the model weights (which are typically constants, unlike activation which are dynamically produced during runtime) are zeroed in order to reduce the size of the model and thus speed up the model’s runtime. This may be done offline when the model weights are constant.

[0010] Activation pruning (online): This method modifies the activations produced by layers of the model by ‘zeroing’ activation values. This is necessarily performed online since each execution is different. The result can be higher accuracy. However, activation pruning introduces a latency penalty during run time. For example, it can take time to determine which activations to modify to zero.

[0011] Other improvements take advantage of the inherent characteristics of the model, for example, zeroing activations using an activation function. Activation functions determine the output of a layer and are embedded into all DNNs to achieve nonlinearity mapping of the input. Activation functions can act as a high pass filter, and thus applying an activation function to layers can result in data that contains a lot of zeros. For example, a rectified linear unit (ReLU), which performs (x) = max(x, 0)), is a popular activation layer which, on average, results in the zeroing of 50% of the activations. With respect to the results, multiplying zeroes provides no new information and thus wastes computational time and resources. Various methods are known in the art which exploit the properties of sparse matrices (matrices having a large number of zeroes) to speed up the multiplication of sparse matrices. However, sparse matrices are typically first compressed into a convenient compressed form, such as compressed sparse row (CSR) form.

[0012] Although the use of CSR form speeds up matrix multiplication, the compression algorithm itself to form the CSR can be a slow process. It is self-evident that the speed-up benefit of using a CSR form must outweigh the latency that will be caused by performing the CSR compression. In some cases, however, this balance cannot be achieved. Moreover, generally, in the case of activation pruning (online) DNNs, since the activations are always created anew during each application of an activation function, the compression process to form a compressed matrix of the activation layer should be efficient since it represents a bottleneck in the runtime of the DNN. In other words, in order to exploit the efficiency of sparse matrix multiplication, especially for DNNs, compression of sparse matrices into a compressed representation needs to be efficient since the compression is done during runtime.

[0013] The present disclosure provides methods and systems that solve at least the aforementioned problems of efficiently compressing sparse matrices into a CSR form.

[0014] SUMMARY

[0015] In view of the scenarios above, improved compression algorithms are needed which allow the processor to deterministically iterate over a sparse matrix in order to provide a compressed sparse matrix. This can be done by providing a branchless implementation. This is advantageous because it avoids the processor restarting an instruction pipeline after decoding a branch, which wastes computing resources and in effect stalls the processing of the algorithm while the processor fetches new data and instructions for a new branch.

[0016] A first aspect of the present disclosure provides a computer-implemented method for compressing a sparse matrix into a compressed format that identifies non-zero values of the sparse matrix, the method comprising: reading a first subset of one or more values from the sparse matrix, wherein the first subset comprises at least one zero; performing a logical bitwise operation on each value in the first subset to thereby identify the at least one zero; writing the one or more values of the first subset to an output array, the output array defining a portion of the compressed format; incrementing an offset counter in dependence on a number of identified non-zeros in the first subset; reading a second subset of one or more values from the sparse matrix, wherein the second subset contains at least one non-zero value; performing a logical bitwise operation on each value in the second subset to thereby identify the at least one non-zero value; writing the second subset to the output array at a write position defined by the offset counter, wherein writing the second subset comprises overwriting the at least one zero in the output array with at least one non-zero value from the second subset. Advantageously, the logical bitwise operation is deterministic and can be implemented without a branch. In other words, the logical bitwise operation and indeed all steps of the method are performed unconditionally. Thus, the algorithm is branchless from the point of view of the processor, and a pipeline of instructions to be followed by the processor can be determined and executed deterministically and thus far more efficiently.

[0017] In examples of the first aspect, the output array is stored in cache memory, wherein the writing of the one or more values of the first subset to the output array and the writing of the second subset to the output array comprises writing values to the output array stored in the cache memory. Furthermore, in example of the first aspect writing the first subset and the second subset at the output array is unconditional with respect to the value of the offset counter, and wherein a write location of writing the second subset at the output array is conditional only on the value of the offset counter.

[0018] Writing to the cache memory is advantageous because cache is proximate to the processor and can be accessed with very low latency. Thus, the cache can be written to, and read from, by the processor efficiently.

[0019] In examples of the first aspect, the method is performed by one or more processors, and wherein performing each logical bitwise operation on each value comprises applying a pre-determined bitwise operation on the value to obtain an indication value that identifies whether the value is equal to zero without causing the one or more processors to branch during execution of the logical bitwise operation. In examples, the logical bitwise operation is encoded as a deterministic operation without an if statement, thereby forming contiguous set of machine-level instructions without branching. The indication value may be a binary value, e.g., a 0 or 1, wherein for example a 0 indicates a zero value and a 1 indicates a non-zero value.

[0020] In examples of the first aspect, the first subset and second subset each contain a single value, and wherein performing the logical bitwise operation on each value comprises generating a binary indication whose value indicates whether the value is equal to zero, wherein the method further comprises, in response to determining that the value in the first subset is equal to zero: incrementing the offset counter in dependence on a value of the binary indication, wherein the offset counter defines an offset position for a subsequent write operation to the output array; writing the non-zero value of the second subset to the output array at the offset position defined by the offset counter thereby causing the overwriting of the zero in the output array with the non-zero value from the second subset.

[0021] Preferably, the logical bitwise operation is performed without a processor determining whether each value read from the first subset is zero or not. In other words, the one or more processors performing the logical bitwise operation perform the same set of operations, irrespective of the value read from the first subset. Thus, preferably, the logical bitwise operation is performed without causing the one or more processors to branch during execution of the logical bitwise operation.

[0022] In this example the first subset and second subset each contain a single value, therefore the first subset contains exactly one ‘zero’ value and the second subset contains exactly one non-zero value. In other words, the reading from the sparse matrix comprises reading one value at a time.

[0023] Incrementing the offset counter in dependence on a value of the binary indication has the advantage that the offset counter is incremented unconditionally. For example, where no non-zero values exist in the first subset, the offset counter is incremented by zero thus leaving the value of the offset counter unchanged. Incrementing the offset counter may thus comprise incrementing the counter by zero based on the value of the binary indication indicating whether the value in the first subset is zero. As a result of the zero increment, the next value is written at the same location as the zero in the output array, thus the zero becomes overwriten by the next, non-zero, value. This method is unconditional, which has the advantage that the processor pipeline can be formed without branching which significantly reduces latency.

[0024] In other examples: in response to determining that the value in the second subset is not equal to zero: incrementing an offset counter by one in dependence on a value of the binary indication generated for the non-zero value. In other words, in response to reading a non-zero value: incrementing the offset counter by one.

[0025] In examples of the first aspect, the first subset and the second subset each comprise a vector of values read from the sparse matrix, wherein performing the logical bitwise operation comprises generating a mask vector identifying positions of non-zero values, the mask vector having the same dimension as the vector of values read from the sparse matrix. Preferably, the mask vector has the dimension as both the first subset and the second subset of values. The vector may comprise 4 values, where each value may be represented by four bytes of data and thus each vector is stored using 16 bytes of data. In other example, a vector may comprise 8, 16, 32, or even 64 values.

[0026] Examples of the first aspect may further comprise: identifying a position and a number of non-zero values in the first subset in dependence on the mask vector generated for the first subset; prior to writing the first subset of values to the output array, re-ordering the first subset to obtain a re-ordered first subset in which non-zero values are leading values; writing the re-ordered first subset to the output array; incrementing an offset counter in dependence on the number of non-zero values in the first subset, wherein the offset counter defines an offset position for a subsequent write operation to the output array; writing the at least one non-zero value of the second subset to the output array at the offset position defined by the offset counter thereby causing overwriting of one or more zeroes in the output array with the at least one non-zero value of the second subset.

[0027] Examples of the first aspect may further comprise: identifying a position of non-zero values in the second subset in dependence on a mask vector generated for the second subset; re-ordering the second subset to obtain a re-ordered second subset in which the at least one non-zero values are leading values; writing at least the leading values of the re-ordered second subset to the output array at the offset position defined by the offset counter thereby causing the overwriting of one or more zeroes in the output array.

[0028] This has the advantage of facilitating the overwriting operations of the compression. By re-ordering the vector such that nonzero values are leading, the zero values can straightforwardly be overwriten at a subsequent iteration by instructing the processor to write the next one or more non-zero values at an entry defined by the offset counter, thereby overwriting the reordered zero values in the previously writen vector. In examples of the first aspect, re-ordering the first subset and / or the second subset comprises: determining a number uniquely defined by the mask vector generated for the first and / or second subset; using the uniquely defined number to determine a re-order vector defining a unique permutation of non-zero values in the first subset and / or the second subset; independence on the re-order vector, overwriting one or more leading values of the first subset and / orthe second subset using one or more non-zero values. Determining the re-order vector defining a unique permutation of non-zero values may comprise using a pre-defined lookup table comprising all unique permutations of non-zero values for a vector having the same dimension as the first subset and / or the second subset.

[0029] In examples of the first aspect, the method comprises reading at least the first and second subsets from the sparse matrix concurrently, and concurrently processing each vector defining at least the first and second subsets by one or more processors using single instruction multiple data, SIMD, processing. Concurrent, or vectorised, processing is advantages faster as it allows the processor to perform at least some operations (e.g., any of read, write, or an arithmetic logic operation) simultaneously within the processing pipeline.

[0030] In examples of the first aspect, the method further comprises: sending a prefetch instruction prior to processing the first subset of values, the prefetch instruction causing one or more processors to read at least the second subset; in response to the prefetch instruction, obtaining the vector of values defining the second subset prior to a point in time at which the one or more processors commence processing the second subset. The advantage is that the prefetch thereby prevents the one or more processors from stalling temporarily. In other words, in the intervening time between sending the prefetch instructions and obtaining the vector of values defining the second subset the one or more processors are able to continuously perform processing operations involving the first subset. The processors are therefore wholly or substantially wholly utilised throughout the entirety of the method.

[0031] In examples of the first aspect, obtaining the vector of values defining the second subset in response to the prefetch instruction comprises reading the vector of values into a cache coupled with at least one processor of the one or more processors.

[0032] In examples of the first aspect, the sparse matrix encodes a set of activation values and / or weight values for use in a neural network, NN.

[0033] According to a second aspect the present disclosure provides a computer program comprising a program code for performing the method according to any preceding claim.

[0034] According to a third aspect the present disclosure provides an apparatus for compressing a sparse matrix into a compressed format that identifies non-zero values of the sparse matrix, the apparatus comprising one or more processors and a memory storing data in non-transient form defining program code executable by the one or more processors for compressing the sparse matrix, the apparatus being configured to: read a first subset of one or more values from the sparse matrix, wherein the first subset comprises at least one zero; perform a logical bitwise operation on each value in the first subset to thereby identify the at least one zero; write the one or more values of the first subset to an output array, the output array defining a portion of the compressed format; increment an offset counter in dependence on a number of identified non-zeros in the first subset; read a second subset of one or more values from the sparse matrix, wherein the second subset contains at least one nonzero value; perform a logical bitwise operation on each value in the second subset to thereby identify the at least one non-zero value; write the second subset to the output array at a write position defined by the offset counter, wherein writing the second subset comprises overwriting the at least one zero in the output array with at least one non-zero value from the second subset.

[0035] BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present disclosure is described by way of example, with reference to the accompanying drawings, in which:

[0037] Fig. 1 shows an example matrix and its corresponding compressed sparse row (CSR) form;

[0038] Fig. 2 is a flowchart illustrating a compression algorithm that operates on scalars from a sparse matrix;

[0039] Fig. 3 is a flowchart illustrating a compression algorithm that operates on vectors from a sparse matrix;

[0040] Fig. 4 is an example of a look-up table used by the algorithm shown in Fig. 3 to re-order vectors;

[0041] Fig. 5 shows a specific example of the processing of a vector according to the algorithm shown in Fig. 3 ;

[0042] Fig. 6 is a flowchart illustrating a variant of the compression algorithm in Fig. 3 including prefetching a set of values; and

[0043] Fig. 7 is a flowchart illustrating an algorithm combining the algorithms shown in Figs. 2 and 3.

[0044] DETAILED DESCRIPTION

[0045] The compressed sparse row (CSR) form is ubiquitous in many linear algebra frameworks, for example in the GNU Sparse Library, and Eigen. These frameworks, however, are implemented for efficient multiplication of matrices that are already in the CSR form. Generally, the compression algorithm itself that forms the CSR data from a sparse matrix has not been optimised and can introduce unwanted, or prohibitive, latencies into programs or neural networks whose computations predominantly involve matrix multiplication.

[0046] Fig. 1 illustrates a matrix 102 and its corresponding CSR form. The CSR form comprises the following information: a value array 108. This comprises all the non-zero values of the original matrix; a column index array 106. This contains the column indices of corresponding non-zero values; a row pointer array 104. This contains the starting index for each row in the values array and also indicates the number of non-zero elements in each row.

[0047] For example, in Fig. 1, the first two values, [0, 2], in the row pointer array 104 indicate that the non-zero values contained in the first row of the matrix begin at position 0 and end before position 2 of the values array. The first two values, [0, 4] of the column index array 106 indicate which columns of the matrix the non-zero values of the corresponding value array belong to. The number of values in the column index array will always equal the number of values in the value array. The number of values in the row pointer array is equal to the number of rows in the matrix plus 1 (or, if the leading zero is omitted for brevity, equal exactly to the number of rows of the sparse matrix).

[0048] One can easily deduce that for sparsification of less than 50% (e.g., where more than half of the values of the matrix are nonzero) this representation consumes more space than the original data. However, the compressed form allows for faster data handling because it focuses the matrix multiplication code on only the relevant data.

[0049] As mentioned above, the generation of the CSR form from a sparse matrix is a pre-processing step (i.e., prior to matrix multiplication) that impacts runtime latency when performed ‘online’ (i.e., during the runtime of matrix multiplication). An example where CSR compression will necessarily be done during runtime is when the activation layer of a DNN is compressed since the activation layers are produced using an activation function during runtime of the DNN. It is therefore highly desirable for the compression process itself, as well as the subsequent sparse matrix multiplication framework, to be as efficient as possible.

[0050] In known solutions, compression algorithms iterate over all matrix elements and conditionally insert (i.e., on the condition that the element is a non-zero value) only the non-zero elements into the values array, which is a time-consuming process. Generally, the conditional portion of code used to implement the known compression algorithms can be described with the following pseudo-code : for row in matrix: forva / z / e in row: if (value > 0; OR value < 0): append value to output array else: proceed to next iteration end for end for

[0051] The problem arises from the uncertain way in which the processor interprets the ‘if’ statement. More precisely, latency arises from the machine code (i.e., that is read by the processor) that results from the compilation of the above code. Generally, an ‘if’ statement, when expressed at the machine-code level, causes a CPU to ‘branch’ . A branch splits the flow of a program into two parts based on runtime condition evaluation. In other words, the high-level code is re-written, at the machine / processing level, into machine instructions that the processor itself follows in which the instructions for an ’if’ statement form two separate portions of code. The processor cannot know ahead of time which branch will be taken because it depends on the arguments used by the ‘if’ statement.

[0052] The consequence of branching is that new instructions need to be read depending on which branch is taken, and new values need to be read into the processor’s cache depending on which branch is taken. It is not generally possible or practical to preload the instructions or values ahead of time. Thus, during a branch, a processor fetches the upcoming instruction from memory, and when a processor decodes a branch instruction it has to restart the instruction pipeline. ‘If’ statements therefore generate latency that is attributable to delays in fetching instructions and values on-the-fly without any way of predicting which branch will be taken.

[0053] Another problem is the delay associated with reading new values from the matrix during the iterations. When a read instruction is executed, the processor may stall until the necessary data is ready, which degrades the overall performance. In some cases, read operations consume about 22% of the total execution time.

[0054] In this specific case, the low-level machine code will contain two branches which represent the two possibilities that the ‘value ’ is either equal to zero or not equal to zero. The ‘if’ statement must be performed for every single value in the matrix, so a large number of branches are necessarily handled during the compression of a sparse matrix, for typical compression algorithms. In some exemplary cases, branching in the processor pipeline consumes about 24% of the total execution time of the matrix compression.

[0055] Branchless programming is therefore desired for the processor instruction pipeline in order to achieve high performance and low latency for compressing a sparse matrix.

[0056] Fig. 2 shows an example of an algorithm 200 for compressing a sparse matrix into CSR form which avoids branching of the algorithm at the processor level.

[0057] At step S 100, an offset value (which would usually be an integer value) is initialised to zero. The offset value, also referred to as an ‘offset counter’, defines the position at which the next write operation will take place in the output array. The value of ‘offset’ is generally initialised to zero, producing the result that the first value read from the matrix will be written to the initial (or ‘zeroth’) entry in the output array.

[0058] At step S102, a row of the matrix is read (i.e., the first row in the first instance). Alternatively, the matrix may be stored in some alternative format (e.g., a linear array) in which case a predetermined number of values from some address in memory may be read where those predetermined number of values correspond to a row or part of a row of a matrix. It would be apparent to the skilled person that step SI 02 is merely one example, and that no particular storage regime of the sparse matrix is required for the efficient operation of the compression method, nor is it essential for an entire row of a matrix to be read at a time.

[0059] At step S104, a single element from the row is read. The full row of the matrix may (and preferably is) already be stored in a cache of the processor. Thus, the meaning of ‘read’ in this case, and in general, may merely indicate that the processor is currently handling or preparing to handle a particular value already held in the cache at some level.

[0060] At step S 106, the value read at step S 104 is written into the output array at the position defined by the ‘offset’, i.e., initially at position [0] . The writing of the value into the output array is not conditional on the value. In other words, both zero and nonzero values will initially be written to the output array. The advantage of this is that the writing step is deterministic and is not conditional. Thus, no branching of the CPU is required. Moreover, since no branching is required, the processor will be able to determine its pipeline of operations with certainty, thereby mitigating delays. Preferably, the write operation takes place in the cache; i.e., preferably the output array is temporarily stored in a cache.

[0061] At step S108, another write is performed into the column index array indicating, again at the current ‘offset’ position, indicating the matrix column from which the current element has been read.

[0062] At step S 110, the offset is incremented based on a deterministic Boolean operation. In other words, a logical bitwise operation is performed on the value. The result of the logical bitwise operation will be an indication, preferably a binary value. The value of that indication / binary value is used to increment the offset. For example, the offset may be incremented as follows: offset = offset + (value not_equal 0)?l :0

[0063] Thus, if value equals zero the result of the logical bitwise operation will be a ‘O’, and the offset will be incremented by a value of zero (in other words, the value of the offset will not change). Alternatively, if value is non-zero, the logical bitwise operation will return a value of 1, and offset will be incremented by a value of 1. The result of ‘" value not_equal 0)?l:0” is therefore a value of 0 or 1. This result thus represents an ‘indication value’ that indicates whether the value read from the sparse matrix is a zero or non-zero.

[0064] Step SI 10 thus provides an advantage of the overall method, in that it overcomes the problem of conditional ‘if’ statements for determining whether (or not) to write a value into the output array. The operation of SI 10 is performed irrespective of the value of ‘value". Perhaps surprisingly, eventhough an apparently unnecessary increment may be calculated (i.e., incrementing ‘offset’ by a value of zero), the advantage of incrementing ‘offset’ unconditionally is that the processor is able to guarantee what instructions it needs to perform and what data it needs access to. This confers a significant advantage since no branching is necessary, and there is no uncertainty in the processing pipeline. Thus, the overall runtime can be significantly reduced due to the deterministic nature of this algorithm.

[0065] Steps S104 to SI 10 are then iterated for all values in the row. It will therefore be appreciated that all values read from the matrix are initially written to the output array at the ‘offset’ position unconditionally. For example, take the first row of matrix 102 in Fig. 1, [10, 0, 0, 0, -12], The first value,

[0010] , will be written to the output array and the ‘offset’ will be increment by 1. The next three values are all zero, and so all three will be written to the second position of the output array. In other words, each successive zero will overwrite the previously read zero since the offset will be not incremented (more precisely, will be incremented by zero). After the first four values have been processed, the output array will be [10, 0], Upon reading the fifth value, -12, the fifth value will again be written to the second position in the output array since the ‘offset’ still has not been incremented. The output array will thus be [10, -12], At this point, i.e., step SI 10, the value of ‘offset’ will be incremented by +1, since -12 is a non-zero value. The next value read from the matrix will thus be written to the output array at the third position. Consequently, the values overwritten as a result of the offset not being incremented (i.e., being incremented by zero) will not be accessible at subsequent iterations because they will have been overwritten.

[0066] In summary, each value read from the matrix is written under any condition (whether it is zero or non-zero). Furthermore, the column index array can be updated using the same ‘offset’ value used to define the write position for the output array. It will be understood that the order of steps in Fig. 2 is not restrictive but merely exemplary. For example, steps S106 and S108 could be performed in a different order, or concurrently. Other possibilities would be within the remit of the skilled person to determine.

[0067] Preferably, the output array is temporarily stored and accumulated in the cache so that the processor can perform all write operations (at S106) at the cache level. This has the advantage of significantly increasing processing speed. Once the output array has been completed, or a storage capacity of the cache is reached and the output array needs to be evicted, the output array may be written to another memory location by the processor.

[0068] Pseudo-code corresponding to algorithm 200 in Fig. 2 may be represented as follows (where text following a “\\” indicates commentary on the code and not actual code instructions): offset = 0 for row in matrix: row_pointer[row] = offset for column _idx in row: value = Matrix[row][column_idx] / / S104 outpiit irray | / sv / | = value / / S106 coliimn_id.\_array| / / se / | = column idx / / S108 offset = offset + (value not_equal 0)?l :0 / / SI 10 end for end for

[0069] In some examples, the actual machine-level instructions that perform the logical bitwise operation may be split into two sequential instructions:

[0070] 1) assert a ‘not_equal’ flag according to the condition of "value not equal to O’;

[0071] 2) offset is conditionally incremented by 1 according to the value of the ‘not_equal’ flag.

[0072] In terms of latency, perhaps surprisingly, it is more beneficial to perform three write commands (values array, column array, and offset) instead of performing a branch at the processor level. The algorithm shown in Fig. 2 therefore confers an immediate advantage over known compression methods since it avoids time-consuming CPU branching. It also enables all read and write operations to be performed at the cache level much more easily since the processor can determine ahead of time which instructions and values it will need, and because all values of the matrix are unconditionally written into the output array at some point. In addition to significantly reducing the latency arising due to processor branching, the compression method disclosed herein (both in respect of Fig. 2 and the following examples) also reduces the power consumed by computing resources when performing sparse matrix compression. This confers a particular benefit for computing resources with low power requirements, though the advantage is still obtained for large computing clusters or servers performing embodiments of the compression algorithm disclosed herein.

[0073] Fig. 3 illustrates another branchless algorithm 300 that contains yet further improvements on the sparse matrix compression algorithm. This variant avoids the loop-dependency of the ‘offset’ value and thus enables the processor more effectively vectorise the code. In other words, the algorithm of Fig. 3 operates concurrently (though not necessarily simultaneously) on the values within a vector which further improves efficiency.

[0074] At step S200, an offset value (which would usually be an integer value) is initialised to zero. The first write will thus start at the first entry in the output array since ‘offset’ is initialised to zero. At step S202, a vector of values from the matrix is read. This vector of values may or may not correspond to the number of values in a row. In some examples, a vector of four values is read, and each value is handled concurrently.

[0075] At step S204 a bitwise operation is applied to produce a masked vector (or vector mask) which holds the indications) of whether a non-zero value exists in each location. Thus, acquiring the vector mask represents applying four logical bitwise operations (1 for each value of the vector), where each value of the vector mask represents (or actually is) a value of 0 or 1. Each value of the vector mask thus represents an ‘indication value’ that indicates whether the value read from the sparse matrix is a zero or non-zero.

[0076] At step S206 the number of non-zero values within the vector mask are counted. Counting the number of non-zero values in the vector mask is a proxy to counting the number of non-zero values in the vector read at step S202. It is generally more efficient to count the number of non-zero elements in the mask (compared to counting using the originally read vector) because the set of values that can be present in the mask is restricted (preferably restricted to a binary value representing either a zero or a non-zero value). Counting the number of non-zero values in the vector mask, ratherthanthe vector, thus allows the counting to be performed in a deterministic way, i.e., without using a conditional ‘if’ instruction. Detailed examples of the vector mask and methods of counting the non-zero values are provided below. At the point step S208 is reached, the position and number of non-zero values in the vector are known. Based on this information, at step S208 the vector is re-ordered such that the non-zero values (if present) are moved to occupy the first entries in the vector. In other words, the non-zero values are moved to be the leading entries of the vector. If ‘nz’ is deemed the number of non-zero values, in practice, the re-ordering may comprise simply overwriting the ‘nz’ leading values of the vector with the non-zero values. For example, if the vector read is v = [10, 0, 0, -12], the re-ordering may comprise overwriting the second entry, [0], with the -12 to obtain [10, -12, 0, -12], The values of the final two entries are irrelevant for the further progression of the algorithm since they will be overwritten by subsequently read non-zero values. Alternatively, the vector may be rearranged so that the non-zero values lead, and the zeros form the trailing entries in the vector, i.e., to form [10, -12, 0, 0],

[0077] At step S210, the re-ordered vector (e.g., [10, -12, x, x], where x denotes a value that is unimportant and can be deemed arbitrary) is written to the output array. As with step S 106 in Fig. 2, this write operation is unconditional on the values of the re-ordered array, i.e., the write is performed irrespective of how many zeros are present in the re-ordered vector (including if all values of the vector are zero). The re-ordered vector is written to the position in the output array indicated by the accumulated value of ‘offset’ (which initially is zero). Again, as with S 106, the advantage of writing the re-ordered vector unconditionally is that the process is deterministic, and not conditional, and therefore avoids the latency of processor branching that is necessitated by conditional ‘if’ instructions.

[0078] At step S212, similarly to step S 108, the column indices (that indicate the column from which each non-zero value in the matrix was read) are written to a column index array.

[0079] At step S214, the offset counter is incremented by the number of non-zero values present in the array (as counted in step S206). It will be understood that the order of steps in Fig. 3 is not restrictive but merely exemplary. For example, the offset increment step S214 can be carried out at any point after S206 and at any point before the updated offset value is needed again (which would be step S212 at the next iteration dealing with a new vector of values). Furthermore, various steps could be performed concurrently in multi -threaded environment, or using multiple processing cores. Generally, any steps without dependencies on one another could be performed concurrently, or in a different order to that shown in Fig. 4. For example, any of steps S210, S212, and S214 could be performed concurrently.

[0080] Following step S214, a new vector of values is read from the matrix and steps S202 to S214 iterate until the matrix has been compressed into sparse form. It should be appreciated that the non-zero values of subsequently read vectors will overwrite the zero-values from previously-read vectors. This is because the ‘offset’ counter, which is determined based on the number of non-zero values in a vector, also defines the position at which each vector is written to the output array; and, because the vectors have been reordered such that non-zero values are placed in the lead entries of each vector, all zero values are eventually overwritten by non-zero values as the iterations progress. For example, the output array may contain values [10, -12, x, x] after the first iteration (where x denotes either zero or an arbitrary value). After the first iteration, the offset counter will be at nz = 2. If the next array read is [0, 13, 0, 19,], this will be re-ordered to [13, 19, x, x]. This re-ordered array will then be written starting at write position 2 in the output array (which, when indexing starts at zero, represents a third entry position in the output array). Consequently, aftertwo iterations, the output array will be [10, -12, 13, 19, 0, 0],

[0081] The following pseudo-code provides a more detailed example implementation of the vector algorithm 300 in Fig. 3. It should nevertheless be understood that this is merely one example and is in no way restrictive. In this example, a look-up table is used to re-order vectors based on the positions of the non-zero values in the vectors (text following a “ / / ” indicates commentary on the code and not actual code instructions): table_key= {1,2, 4, 8} offset = 0 for row in matrix: end for end for

[0082] In this example, step S204 involves acquiring a vector mask. This may include applying another logical bitwise function that results in a binary value resulting from each comparison. For example, the mask could apply a ‘compare equal’ instruction by comparing each value to zero. The result of applying the mask is preferably another vector, of the same dimension of the vector read from the sparse matrix, which comprises a ‘ 1’ in entries that correspond to non-zero values and a ‘0’ in entries that correspond to zero entries. For example, the vector [10, 0, 0, -12] would yield the mask vector [1, 0, 0, 1], The number of nonzero values in the original vector can then be straightforwardly counted by summing the values of the mask vector. Advantageously, this method provides a deterministic method of counting the number of non-zero values which avoids conditional instructions, and the slow branching that such conditional instructions require.

[0083] After being used to count the number of non-zero values, the vector mask serves the secondary purpose of being used in the process of re-ordering the vector. In one example, as in the pseudo-code above, a lookup table is used to determine how to reorder the values in the vector.

[0084] Fig. 4 shows a specific example of a look-up table 400. Each value in the table represents a byte offset. The table 400 in Fig. 4 corresponds to the specific case where a vector read from a sparse matrix has four entries, where each entry is represented using four bytes. Thus, each row comprises 16 values (4 x 4 = 16). Each row of the 16 rows in the table represents a different permutation of four non-zero values in a vector (i.e., each value can be zero or non-zero, and there are four values, thus the number of permutations is 24= 16). Thus, the look-up table comprises 16 rows each with 16 values. In summary, each possible permutation of non-zero values for a four-entry vector can be associated with a unique row from one of the 16 rows of the table, and each row of the lookup table defines how to re-order a vector such that the non-zero values become the leading value.

[0085] Fig. 5 shows a detailed example 500 describing how the vector mask may be used to select the correct unique row from the look-up table, and how the vector is subsequently re-ordered based on the entries selected from the look-up table. The method 500 shown in Fig. 5 thus describes a specific example of how steps S202 to S208 may be carried out.

[0086] At S300 of Fig. 5, a vector comprising 4 values is read from the matrix: [0, 0, 8, 9], A logical bitwise operation is then applied at step S302 to each value in the vector to obtain a mask vector: [0x000000, 0x000000, OxFFFFFFF, OxFFFFFFF] . The values here are represented in hexadecimal. In practice, this vector may be obtained in a two-step process: i. perform a ‘compare equal instruction’ which compares each value in the vector to zero in a bitwise fashion. A comparison of ‘0 AND 0’ returns 1, and a comparison of 0 and any non-zero value returns a 0. The result of applying a compare equal instruction to [0, 0, 8, 9] is: [OxFFFFFFF, OxFFFFFFF, 0x000000, 0x000000] ii. The result of i) is reversed to that values representing all Is (OxFFFFFFF) become all zero, and vice versa. The result of this reversal gives: [0x000000, 0x000000, OxFFFFFFF, OxFFFFFFF]

[0087] Step S304 modifies the mask vector thus obtained [0x000000, 0x000000, OxFFFFFFF, OxFFFFFFF] into a form that facilitates determining the number and position of the non-zero values. In this example, a right bitwise shift is applied which results in: [0, 0, 1, 1], A right bitwise shift is defined in this case as shifting the bit-representation of a value to the right by a specified number of bits; in doing so, excess bits shifted off to the right are discarded, and bit positions that have been vacated by the shift operation are zero-filled.

[0088] Consequently, the resulting bit mask, [0, 0, 1, 1], can be used to efficiently calculate the number of non-zero values in the vector. This is performed in step S306 (corresponding to step S206 in Fig. 3) in which the values of the shifted mask vector are summed. Moreover, the bitwise operations used to obtain this mask are very efficient to compute since they involve simple bitwise logic. The bitwise logic in this example does not comprise conditional instructions and so advantageously does not introduce latency associated with branching.

[0089] The shift mask vector is then used to determine which row of the look-up table uniquely defines the permutations of non-zero values in the vector. The pseudo-code above includes an example for performing this, i.e., by calculating: SUM(taWe_tey AND vector mask). In this example, the ‘AND ’ is a bitwise AND operation. The table key is used to determine the row in the lookup table. In this example the table_key is [1, 2, 4, 8], as indicated in Fig. 5, the reason being that any value from 1 to 15 canbe obtained by summing different combinations of 1, 2, 4, 8. Thus, the combination of the table key and the shifted mask vector yields a number from 0 to 15 (0 resulting from a combination of the table key with a mask vector that contains all zeros) which is used to select the correct from the lookup table.

[0090] Step S308 involves the calculation of a unique mask vector by performing 'table key AND vector mask’. For completeness, it should be appreciated that the ‘AND’ operation is performed at the binary level. 4 represent in binary is ‘.00100’. The shifted vector mask may in practice contain all ones or all zeros, e.g. : “[000, 000, 111, 111]”. Thus, the bitwise operation (‘ 111’ AND ‘100’) results in ‘100’: i.e., the value 4. Thus, the result of the example in Fig. 5, represented in decimal, is [1, 2, 4, 8] AND [0, 0, OxFFFFFFFF, OxFFFFFFFF] = [0, 0, 4, 8],

[0091] At step S310, the unique entry for the lookup table is calculated by summing the values of the unique mask vector: 4+8 = 12. Following this, at step S312 the twelfth row of the lookup table 400 is read, which in this case is: {8, 9, 10, 11, 12, 13, 14, 15, 0, 0, 0, 0, 0, 0, 0, 0}. As mentioned above, each value of the lookup table represents a byte offset, and each value in the vectors is represented (in this specific example) using four bytes. Thus, byte offsets {8, 9, 10, 11} refer to the bytes in position 8, 9, 10, and 11 of the original vector: i.e., the four bytes representing the value ‘8’ initially read.

[0092] Step S314 thus involves reordering the originally read vector at step S300, [0, 0, 8, 9] using the key defined by the re-order table entry determined in step S312. The byte offsets in the look-up table entries thus identify which bytes of the vector should be written to which position in the re-ordered vector. In this case, the first four-byte offsets are {8, 9, 10, 11} which indicates that value ‘8’ of the vector (which is a four-byte value stored at the indicated byte offsets) should be re-written as the leading entry. The second set of four-byte offsets, {12, 13, 14, 15}, indicates that the vector value ‘9’ (which is a four-byte value stored at the indicated byte offsets 12, 13, 14, 15) should be written as the second entry in the re-ordered vector. The remaining byte offsets are all zero, which in this case indicates that the remaining entries should be overwritten with zero. Thus, the remaining entries in the re-ordered vector are considered arbitrary, and are denoted by the ‘x’ : [8, 9, x, x] . They are considered arbitrary because they will eventually be overwritten by a subsequently read non-zero value.

[0093] Step S316 involves generally the same logic to re-order the column index key {0, 1, 2, 3 } using the same look-up table row. The result in this case is the re-ordered column index [2, 3, x, x]. Following this step, the re-ordered vector and re-ordered column index are written to the output array and the column index array, respectively, as outlined in steps S210 to S212 and in the example pseudo-code above.

[0094] Surprisingly, performing the multiple bitwise and write operations described in steps S300 to S316 provides a significant speedup compared to simply using conditional instructions to rearrange the vector, which involves time-consuming branching at the processor level. It should be appreciated that this advantage derives from the fact that the steps in Fig. 5 are, in isolation, efficient to compute and are also deterministic in the sense that each step is performed unconditionally with respect to the underlying values.

[0095] Fig. 6 shows a further vectorised algorithm 600, which is a variant of the vector algorithm 300 of Fig. 3. This variant 600 is specifically aimed at further mitigating read delays that result from an instruction requiring the processor to handle data that it has not yet cached. Furthermore, this variant 600 handles multiple vectors (preferably of 4 elements, though larger vectors may be used) at a time.

[0096] As with the previous algorithm, at step S400 the offset value is initialised. At step S402 a chunk of data is pre-fetched by the processor. This data fetched by the processor is preferably data that will be used for a subsequent iteration but not by the current iteration. Pre-fetching therefore provides the advantage that the processor can add a pre-fetch instruction into its pipeline whilst performing other operations for the current interaction, thereby causing data for a subsequent iteration to be written into a cache. Thus, at the subsequent iteration, the data will already be cached and will be immediately accessible for use by the processor.

[0097] The algorithm otherwise generally performs in a similar way to the vector algorithm 300 described in Fig. 3. However, as mentioned, the further vector algorithm 600 can also allow for multiple vectors to be handled concurrently (though not necessarily precisely simultaneously). Merely as an example, the following pseudo-code provides a detailed example of the multi -vector variant algorithm 600 which allows multiple vectors to be handled simultaneously and also pre-fetches chunks of vector data to mitigate read delays (text following a “ / / ” indicates commentary on the code and not actual code instructions): offset = 0 for row in matrix: row_pointer[row] = offset for column idx in row:

[0098] / / prefetch data / / prefetch Matrix[row][colum idx + prefetch offset} / / handle vector data comprising chunk of 16 elements / / vector 0 = Matrix[row][colum idx] vector 1 = Matrix[row][colum idx + 4} vector _2 = Matrix[row][colum idx + <8] vector _3 = Matrix[row][colum idx + 06] for vector in [vector . vector 1, vector _2, vector _5]: vector mask = mark non-zero values in vector / / S204 nz = count non-zero values in vector mask / / S206 ordered vector = vector re-ordered using reorder indices

[0099] / / step 208 end / / ordered_col_idx = {0, 1, 2, 3} ordered w / reorder indices ordered_col_id write ordered v write ordered_c offset += nz / / S214 end for end for

[0100] In this example, the prefetchoffset defines the volume of data that is prefetched: specifically, the number of matrix values. Each of [vector _0, vector l, vector_2, vector _3 ] represents a different vector of four values (preferably pre-fetched at a previous iteration).

[0101] In general, the time it takes to read from memory is highly relevant for runtime performance, because the processor may run out of things to do (i.e., stall) if it cannot obtain data ready to perform instructions. Furthermore, actual read operations are usually done using blocks of fixed size from the cache (cache line), since the cache memory is much faster (than external RAM, for example) and beneficial to use when handling live data being processed by a processor. Further, it is preferred to perform write operations in the cache because, the majority of the time, the same offset array is written to (i.e., where lots of values in the sparse matrix are zero, the offset counter will not be incremented and so the write offset will remain unchanged).

[0102] In summary, the Fig. 6 algorithm 600extends algorithm 300 (shown in Fig. 3) by prefetching and loop unrolling into chunks of 16 floating-point values. Memory prefetch speeds up execution by virtue of bringing subsequently needed memory data elements of the matrix into the cache earlier (i.e. before the processor actually needs to perform instructions on the data). The prefetch offset depends on the actual cache size. Moreover, the size of the prefetch offset may be selected to balance competing aspects: on one hand one doesn’t want to prefetch memory data too far in advance because it may need to be evicted from the cache, and on the other hand it would not be desirable to load data that it needed imminently because it is preferable to ensure that the data is ready in cache in time for the processor to use it. As an example, an offset that loads elements corresponding to about the next 4 to 8 loops / iterations is used for good performance.

[0103] Furthermore, the use of loop unroll enables the algorithm to hide the latency associated with reading by enabling the processor to execute instructions which are not dependent on data being prefetched by the memory-read instruction. For example, when using chunks of 16 elements it is possible to “hide” the read latency because the processor can execute instructions which are not required by the (prefetched) read results. For example, when implementing a loop unroll, which comprises reading more than 1 vector per loop iteration, the processors can handle other instructions while waiting for the data to be obtained in the cache (thus instead of waiting the processor can performed useful instructions such as issuing more than one read instruction. Furthermore, when implementing a loop unroll the second vector (vector_l) data will be probably have been obtained (in the cache) and be ready for use by the time the processors) begin to process it.

[0104] In other words, the processor can continue to perform other arithmetic, read, write, and / or logical operations while it waits for the values to be read from memory and written to the cache for subsequent iterations. It will be understood that 16 is merely an exemplary number: this algorithm 600 can be extended to larger chunk size (e.g., 32, 64 or even more floating-point values). For example, the chunk size may be chosen on a case-by-case basis in order to utilize the underlying processing resources and / or hardware in the most effective manner. Such design choices would be within the remit of the skilled person to make.

[0105] In practice, the size or dimensions of a sparse matrix may not align optimally with the dimensions of vectors used by the algorithm used to obtain the compressed CSR form. Nevertheless, the inventors have established that the different algorithms disclosed herein may be combined. Therefore, a preferred compression algorithm operates initially on larger chunks of data, moving onto smaller chunks of data when larger chunks can no longer be read from the remaining values in the sparse matrix. Finally, the preferred algorithm can complete the compression process by operating on scalar values remaining in the sparse matrix.

[0106] Fig. 7 shows a flowchart illustrating the operation of such a combined algorithm 700. Generally, this combined algorithm 700 comprises the steps of the algorithms (600, 300, 200) disclosed inFig. 6, Fig. 3, and Fig. 2. As a first step, after the offset value is initialised at zero, the multi -vector algorithm 600 is performed. Preferably, the largest viable chunk size is used for a given sparse matrix: for example, 64 elements may be handled concurrently where possible. As mentioned, algorithm 600 also comprises a pre-fetch instruction which allows chunks (or multiple chunks) of data corresponding to subsequent loops to be pre-loaded into the cache to thereby avoid delays in the processor pipeline (i.e., to avoid delays in which the processor attempts to perform an instruction, but the relevant data is not yet available in the cache).

[0107] In some cases, algorithm 600 may vary its operational parameters midway through processing in order to adapt to the largest chunk size that can be handled at any given time. For example, algorithm 600 may operate as follows:

[0108] Iterate over a sparse matrix with chunks of 64 elements;

[0109] Iterate over the sparse matrix with chunks of 32 elements;

[0110] Iterate over the sparse matrix with chunks of 16 elements;

[0111] Iterate over the sparse matrix with chunks of 8 elements.

[0112] Generally, even when larger chunk sizes are used, the size of the vector handled by the algorithm may remain the same size, e.g., 4 elements long. In some cases, the size of the vector may be determined by the underlying hardware. For example, some computing architectures may be limited to handling a predetermined number of vector elements at a time. Preferably, the number of elements in each vector is 4, thereby allowing the same lookup table as illustrated in Fig. 4. Thus, when chunks of a larger size are used, e.g., a 32 element chunk, eight vectors of 4 elements each are used. This enables eight vectors to be unrolled in one iteration.

[0113] Once the multi-vector algorithm has been used to the extent possible (e.g., the number of values remaining in the sparse matrix is fewer than the size of the chunk being handled by the algorithm 600), the compression process moves onto vector algorithm 300 which operates on 4 elements at a time. Although not indicated in Fig. 3, the vector algorithm 300 may also make use of pre-fetched data in order to mitigate read-based delays. After iterating over one or more vectors (preferably 4-element vectors) within the sparse matrix (preferably all possible vectors remaining in the matrix), the scalar-based algorithm 200 illustrated in Fig. 2 may be used to complete the compression processing.

[0114] Although not necessarily indicated, generally, all algorithms disclosed herein may be configured to generate a complete CSR output containing: a value array 108, column index array 106, and row pointer array 104, as illustrated in Fig. 1. Examples

[0115] The following table provides results indicating the latency of packing a sparse matrix into the CSR form using a known CSR method and an efficient compression embodiment of the present disclosure. The right-most column indicates the increase in speed of a presently disclosed embodiment compared to a known CSR compression method, thus indicating the potential advantages of the method. The examples were performed for three different matrix dimensions and for each matrix dimension for three different levels of sparsity (where, e.g., a sparsity of 30 % indicates that 30 % of the matrix values are zero with the remaining values being non-zero).

[0116] In this disclosure, when the subject of a phase is described as being "configured to" or “arranged to”, followed by a term defining a condition or function, this is used to indicate that the subject of the phrase is in a state in which it has that condition, or is able to perform that function, without the subject being modified or further configured.

[0117] Some implementations may be described using the expressions “one / an embodiment” or “one / an implementation” or “one / an example”, along with their derivatives. These terms mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in some implementations” in various places in the specification are not necessarily all referring to the same embodiment. Moreover, unless otherwise noted the features described above are recognized to be usable together in any combination. Thus, any features discussed separately may be employed in combination with each other unless it is noted that the features are incompatible with each other.

[0118] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present disclosure may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the appended claims.

[0119] The foregoing description of example embodiments has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Many modifications and variations are possible in light of this disclosure. It is intended that the scope of the present disclosure be limited not by this detailed description, but rather by the claims appended hereto. Future filed applications claiming priority to this application may claim the disclosed subject matter in a different manner and may generally include any set of one or more limitations as variously disclosed or otherwise demonstrated herein.

Claims

CLAIMS1. A computer-implemented method for compressing a sparse matrix into a compressed format that identifies non-zero values of the sparse matrix, the method comprising: reading a first subset of one or more values from the sparse matrix, wherein the first subset comprises at least one zero; performing a logical bitwise operation on each value in the first subset to thereby identify the at least one zero; writing the one or more values of the first subset to an output array, the output array defining a portion of the compressed format; incrementing an offset counter in dependence on a number of identified non-zeros in the first subset; reading a second subset of one or more values from the sparse matrix, wherein the second subset contains at least one non-zero value; performing a logical bitwise operation on each value in the second subset to thereby identify the at least one non-zero value; writing the second subset to the output array at a write position defined by the offset counter, wherein writing the second subset comprises overwriting the at least one zero in the output array with at least one non-zero value from the second subset.

2. A method of claim 1, wherein the output array is stored in cache memory, and wherein the writing of the one or more values of the first subset to the output array and the writing of the second subset to the output array comprises writing values to the output array stored in the cache memory.

3. The method of claim 1 or 2, wherein the method is performed by one or more processors, and wherein performing each logical bitwise operation on each value comprises applying a pre-determined bitwise operation on the value to obtain an indication value that identifies whether the value is equal to zero without causing the one or more processors to branch during execution of the logical bitwise operation.

4. The method of any preceding claim, wherein the first subset and second subset each contain a single value, and wherein performing the logical bitwise operation on each value comprises generating a binary indication whose binary value indicates whether the value is equal to zero, wherein the method further comprises, in response to determining that the value in the first subset is equal to zero: incrementing the offset counter in dependence on a value of the binary indication, wherein the offset counter defines an offset position for a subsequent write operation to the output array; writing the non-zero value of the second subset to the output array at the offset position defined by the offset counter thereby causing the overwriting of the zero in the output array with the non-zero value from the second subset.

5. The method of claim 1 or 2, wherein the first subset and the second subset each comprise a vector of values read from the sparse matrix, wherein performing the logical bitwise operation comprises generating a mask vector identifying positions of non-zero values, the mask vector having the same dimension as the vector of values read from the sparse matrix.

6. The method of claim 5, further comprising: identifying a position and a number of non-zero values in the first subset in dependence on the mask vector generated for the first subset; prior to writing the first subset of values to the output array, re-ordering the first subset to obtain a re-ordered first subset in which non-zero values are leading values; writing the re-ordered first subset to the output array;incrementing an offset counter in dependence on the number of non-zero values in the first subset, wherein the offset counter defines an offset position for a subsequent write operation to the output array; writing the at least one non-zero value of the second subset to the output array at the offset position defined by the offset counter thereby causing overwriting of one or more zeroes in the output array with the at least one non-zero value of the second subset.

7. The method of claim 6, further comprising: identifying a position of non-zero values in the second subset in dependence on a mask vector generated for the second subset; re-ordering the second subset to obtain a re-ordered second subset in which the at least one non-zero values are leading values; writing at least the leading values of the re-ordered second subset to the output array at the offset position defined by the offset counter thereby causing the overwriting of one or more zeroes in the output array.

8. The method of claim 6 or 7, wherein re-ordering the first subset and / or the second subset comprises: determining a number uniquely defined by the mask vector generated for the first and / or second subset; using the uniquely defined number to determine a re-order vector defining a unique permutation of non-zero values in the first subset and / or the second subset; independence on the re-order vector, overwriting one or more leading values of the first subset and / orthe second subset using one or more non-zero values.

9. The method of claim 8, wherein determining the re-order vector defining a unique permutation of non-zero values comprising using a pre-defined lookup table comprising all unique permutations of non-zero values for a vector having the same dimension as the first subset and / orthe second subset.

10. The method of any of claims 5 to 9, the method comprising: reading at least the first and second subsets from the sparse matrix concurrently, and concurrently processing each vector defining at least the first and second subsets by one or more processors using single instruction multiple data, SIMD, processing.

11. The method of any of claims 5 to 10, the method further comprising: sending a prefetch instruction prior to processing the first subset of values, the prefetch instruction causing one or more processors to read at least the second subset; in response to the prefetch instruction, obtaining the vector of values defining the second subset prior to a point in time at which the one or more processors commence processing the second subset.

12. The method of claim 11, wherein obtaining the vector of values defining the second subset in response to the prefetch instruction comprises reading the vector of values into a cache coupled with at least one processor of the one or more processors.

13. The method of any preceding claim, wherein the sparse matrix encodes a set of activation values and / or weight values for use in a neural network, NN.

14. A computer program comprising a program code for performing the method according to any preceding claim.

15. An apparatus for compressing a sparse matrix into a compressed format that identifies non-zero values of the sparse matrix, the apparatus comprising one or more processors and a memory storing data in non-transient form defining program code executable by the one or more processors for compressing the sparse matrix, the apparatus being configured to: read a first subset of one or more values from the sparse matrix, wherein the first subset comprises at least one zero; perform a logical bitwise operation on each value in the first subset to thereby identify the at least one zero; write the one or more values of the first subset to an output array, the output array defining a portion of the compressed format; increment an offset counter in dependence on a number of identified non-zeros in the first subset; read a second subset of one or more values from the sparse matrix, wherein the second subset contains at least one non- zero value; perform a logical bitwise operation on each value in the second subset to thereby identify the at least one non-zero value; write the second subset to the output array at a write position defined by the offset counter, wherein writing the second subset comprises overwriting the at least one zero in the output array with at least one non-zero value from the second subset.

Images