Activating compression and / or decompression of data

Abstract

The present disclosure relates to the compression and / or decompression of activation data. A method is provided for compressing activation data of a neural network to be written to a storage device. The activation data is formed into a plurality of groups, and a state indicator indicates whether there are any data elements within each group that have non-zero values. A second state indicator indicates, for groups with non-zero values, whether a sub-group within the group contains data elements with non-zero values. A sub-group state indicator indicates, for each sub-group with non-zero values, which data elements within the sub-group have non-zero values. The non-zero values of data elements in the activation data are encoded and a compressed data set is formed that includes the first state indicator, any second state indicators, any sub-group state indicators, and the encoded non-zero values.

Description

BACKGROUND

[0001] TECHNICAL FIELD

[0002] The present invention relates to compression and / or decompression of activation data, and more particularly to compression and / or decompression of activation data from neural networks and the like.

[0003] DESCRIPTION OF RELATED ART

[0004] One way to perform lossless data compression is to convert values to Golomb Rice codes. To convert a numerical value to a Golomb Rice code, a parameter known as the divisor is selected. To create a Golomb Rice code, the numerical value is divided by the divisor to generate two parts. The first part is the quotient, which is the number of times the numerical value is evenly divided by the divisor. The second part is the remainder, which is the amount left over, if any, after the numerical value is evenly divided by the divisor.

[0005] An example of a Golomb Rice code is shown in Figure 1 . In the example shown in Figure 1 , values between 0 and 10 are shown as Golomb Rice codes. The first part of the Golomb Rice code, the quotient, is represented in unary format. In this format, the number is represented by a number of ‘1’s equal to the value of the quotient followed by a stop bit ‘0’. For example, the unary portion of the number 9 is ‘1110’, which is three ‘1’s followed by a stop bit ‘0’, because 9 is divided by 3 three times. The second part of the Golomb Rice code is a fixed length binary portion. Since the divisor in this example is ‘3’, the remainder can only be 0, 1, or 2. Therefore, this can be represented by a fixed length binary of two bits. The last two bits in each of the Golomb Rice codes represent the remainder in binary form. The remainder can be referred to as the ‘mantissa’ of the Golomb Rice code, because it appears after the stop bit of the unary portion of the Golomb Rice code.

[0006] Neural networks are also known in the art. In a neural network, an activation function is responsible for transforming the summed weighted inputs from a node into an activation of that node. Activation values from multiple nodes form an activation map. When hardware executes a neural network, each activation value can need to be saved to and loaded from storage possibly multiple times. A paper by Georgios Georgiadis of Samsung Semiconductor published by the Computer Vision Foundation titled 'Accelerating Convolutional Neural Networks via Activation Map Compression' describes activation map compression using an entropy encoding algorithm that is a variant of exponential Golomb encoding.

[0007] In view of the above, an improved method of compressing activation data is desirable in order to improve the processing efficiency of neural networks. SUMMARY

[0008] According to a first aspect, there is provided a method performed by a processing element for compressing activation data of a neural network to be written to a storage device, wherein the activation data is formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: forming a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; identifying whether there are any data elements within each group that have a non-zero value and forming a first status indicator for each group, the first status indicator indicating whether the group contains data elements having a non-zero value; forming a plurality of sub-groups within each group of data elements identified as containing at least one data element having a non-zero value; identifying whether there are any data elements within each sub-group that have a non-zero value and forming a second status indicator, the second status indicator indicating for each sub-group whether the sub-group contains data elements having a non-zero value; for each sub-group containing data elements having a non-zero value, forming a sub-group status indicator for the sub-group, the sub-group status indicator indicating which data elements within the sub-group have a non-zero value; encoding the non-zero values of the data elements in the data unit; and forming a compressed data set comprising the first status indicators, any second status indicators, any sub-group status indicators and the encoded non-zero values.

[0009] According to a second aspect, there is provided a method performed by a processing element for compressing activation data of a neural network to be written to a storage device, wherein the activation data is formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: forming a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; identifying whether there are any data element values having a zero value within each group and forming a first status indicator for each group, the first status indicator indicating whether the group contains data elements having a zero value; forming a plurality of sub-groups within each group of data elements identified as containing at least one data element having a zero value; identifying whether there are any data elements having a zero value within each sub-group and forming a second status indicator, the second status indicator indicating for each sub-group whether the sub-group contains data elements having a zero value; for each sub-group containing data elements having a zero value, forming a sub-group status indicator for the group, the sub-group status indicator indicating which data elements within the sub-group have a zero value; encoding non-zero values of the data elements in the data unit; and forming a compressed data set comprising the first status indicators, any second status indicators, any sub-group status indicators and the encoded non-zero values.

[0010] According to a third aspect, there is provided a data processing system comprising a processing element and a storage device storing a code portion which, when executed by the processing element, causes the data processing system to perform a method of compressing activation data of a neural network, wherein the activation data is formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: forming a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; identifying whether there are any data element values having a non-zero value within each group and forming a first status indicator for each group, the first status indicator indicating whether the group contains data elements having a non-zero value; forming a plurality of sub-groups within each group of data elements identified as containing at least one data element having a non-zero value; identifying whether there are any data elements having a non-zero value within each sub-group and forming a second status indicator, the second status indicator indicating for each sub-group whether the sub-group contains data elements having a non-zero value; for each sub-group containing data elements having a non-zero value, forming a sub-group status indicator for the sub-group, the sub-group status indicator indicating which data elements within the sub-group have a non-zero value; encoding non-zero values of the data elements in the data unit; and forming a compressed data set comprising the first status indicators, any second status indicators, any sub-group status indicators and the encoded non-zero values. BRIEF DESCRIPTION OF DRAWINGS

[0011] The technology will now be described by reference to the embodiments as illustrated in the drawings, in which:

[0012] Figure 1 is a table of Golomb-Rice codes;

[0013] Figure 2 components of a neural processing unit are shown;

[0014] Figure 3a is a diagram showing steps for compressing activation data;

[0015] Figure 3b is a table showing multiple compression schemes that can be used to compress activation data;

[0016] Figure 4 logic for a prediction scheme for delta encoding of activation data is shown;

[0017] Figure 5 working examples of delta encoding of activation data are shown;

[0018] Figure 6 working examples of mask encoding of activation data are shown;

[0019] Figure 7 working examples of inverse mask encoding of activation data are shown;

[0020] Figure 8 is a flow diagram showing steps in a method of processing activation data;

[0021] Figure 9 Golomb-Rice codes are shown divided into streams;

[0022] Figure 10 multiple chunk structures for use in encoding activation data for storage are shown;

[0023] Figure 11 is a flow diagram showing steps in a method for encoding activation data for storage;

[0024] Figure 12 is a flow diagram showing steps for selecting a chunk structure for use in decoding activation data;

[0025] Figure 13 is a flow diagram showing steps for selecting a chunk structure for use in decoding activation data;

[0026] Figure 14 is a flow diagram showing steps for selecting a chunk structure for use in decoding activation data;

[0027] Figure 15a is a flowchart showing steps of a method for decoding one-ary data;

[0028] Figure 15b working examples of decoding one-ary data are shown; and

[0029] Figure 16 processing sequences for recovering delta-encoded activation data are shown. DETAILED DESCRIPTION

[0030] The following description provides an overview of embodiments and associated advantages.

[0031] According to one embodiment, a method is provided for compressing activation data of a neural network to be written to a storage device by a processing element, the activation data formed of units of data elements, each unit including a plurality of data elements and each data element having a numerical value, the method comprising: forming a plurality of groups of data elements within a unit of activation data, each group including a plurality of data elements; identifying whether there are any data elements within each group that have a non-zero value and forming a first status indicator for each group, the first status indicator indicating whether the group contains data elements having a non-zero value; forming a plurality of subgroups within each group of data elements identified as containing at least one data element having a non-zero value; identifying whether there are any data elements within each subgroup that have a non-zero value and forming a second status indicator, the second status indicator indicating for each subgroup whether the subgroup contains data elements having a non-zero value; for each subgroup containing data elements having a non-zero value, forming a subgroup status indicator for the subgroup, the subgroup status indicator indicating which data elements within the subgroup have a non-zero value; encoding the non-zero values of the data elements in the data unit; and forming a set of compressed data including the first status indicators, any second status indicators, any subgroup status indicators, and the encoded non-zero values.

[0032] This embodiment can be used to improve compression of activation data that is sparse due to including a large number of zero values. For example, the method can be effective in compressing activation data from a neural network that utilizes a ReLU activation function.

[0033] The groups of data elements, subgroups of data elements, and data elements can form a quadtree structure. More specifically, in some embodiments, a data unit can include 64 data elements, the data unit can be divided into four groups of 16 elements each, and the groups can be subdivided into subgroups of 4 data elements each.

[0034] Within the compressed dataset, the first state indicators can be grouped into a first set of state indicators, and the second state indicators can be grouped into a second set of state indicators. This is referred to as breadth-first encoding in other implementations, as the first state indicators of a group are clustered together and the second state indicators of a subgroup are also clustered together. Such an arrangement can allow a resolver of the neural processing unit to resolve the compressed dataset more efficiently.

[0035] Encoding of non-zero values of data elements can include delta encoding of the non-zero values. For example, delta encoding can be performed by taking a difference between an actual value of each data element and a predicted value of that data element. A data unit can include a square number of data elements, and except for a top-left corner data element, a predicted value of each data element on a left-hand edge of the square of data elements can equal a value of a data element positioned one position above the predicted data element. Except for the top-left corner data element, a prediction for a data element on a top row of the square of data elements can equal a value of a data element positioned one position to the left of the predicted data element. A prediction for a data element within the square of data elements can be based on a linear combination of values of data elements positioned around the data element. In some implementations, a prediction for a data element within the square of data elements can be based on a linear combination of values of data elements positioned one position to the left of the data element, one position above the data element, and diagonally one position to the upper left corner of the data element.

[0036] Delta encoded non-zero values can be encoded as Golomb-Rice codes. In some implementations, before converting the delta encoded non-zero values to Golomb-Rice codes, the delta encoded non-zero values are converted to unsigned form, such that all of the delta encoded non-zero values are converted to integers greater than or equal to zero. Golomb-Rice encoding of the delta encoded non-zero values can have the effect of improved compression, as the non-zero values are smaller due to the delta encoding and Golomb-Rice codes for lower values are shorter.

[0037] According to a second embodiment, there is provided a method performed by a processing element for compressing activation data of a neural network to be written to a storage device, the activation data being formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: forming within each unit of activation data a plurality of groups of data elements, each group comprising a plurality of data elements; identifying whether there are any data element values within each group that have a zero value and forming a first status indicator for each group, the first status indicator indicating whether the group contains data elements having a zero value; forming within each group of data elements that is identified as containing at least one data element having a zero value a plurality of subgroups; identifying whether there are any data elements within each subgroup that have a zero value and forming a second status indicator, the second status indicator indicating for each subgroup whether the subgroup contains data elements having a zero value; for each subgroup that contains data elements having a zero value, forming a subgroup status indicator for the group, the subgroup status indicator indicating which data elements within the subgroup have a zero value; encoding non-zero values of the data elements in the data unit; and forming a compressed data set comprising the first status indicators, any second status indicators, any subgroup status indicators and the encoded non-zero values.

[0038] Some methods implemented according to the second embodiment can be more efficient when compressing slightly sparse activation data that includes some data elements having a zero value but does not include a large number of data elements having a zero value.

[0039] According to a third embodiment, there is provided a data processing system comprising a processing element and a storage device storing a code portion which, when executed by the processing element, causes the data processing system to perform a method of compressing activation data, wherein the activation data is formed in units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: forming a plurality of groups of data elements within the unit of activation data, each group comprising a plurality of data elements; identifying whether there are any data elements within each group having a non-zero value and forming a first status indicator for each group, the first status indicator indicating whether the group contains data elements having a non-zero value; forming a plurality of sub-groups within each group of data elements identified as containing at least one data element having a non-zero value; identifying whether there are any data elements within each sub-group having a non-zero value and forming a second status indicator, the second status indicator indicating for each sub-group whether the sub-group contains data elements having a non-zero value; for each sub-group containing data elements having a non-zero value, forming a sub-group status indicator for the sub-group, the sub-group status indicator indicating which data elements within the sub-group have a non-zero value; encoding the non-zero values of the data elements in the unit of data; and forming a compressed data set comprising the first status indicators, any second status indicators, any sub-group status indicators and the encoded non-zero values.

[0040] According to a fourth embodiment, there is provided a data processing system comprising a processing element and a storage device storing a code portion which, when executed by the processing element, causes the data processing system to perform a method of compressing activation data from a neural network, the activation data being formed in units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: forming a plurality of groups of data elements within the unit of activation data, each group comprising a plurality of data elements; identifying whether there are any data elements within each group having a zero value and forming a first status indicator for each group, the first status indicator indicating whether the group contains data elements having a zero value; forming a plurality of sub-groups within each group of data elements identified as containing at least one data element having a zero value; identifying whether there are any data elements within each sub-group having a zero value and forming a second status indicator, the second status indicator indicating for each sub-group whether the sub-group contains data elements having a zero value; for each sub-group containing data elements having a zero value, forming a sub-group status indicator for the group, the sub-group status indicator indicating which data elements within the sub-group have a zero value; encoding the non-zero values of the data elements in the unit of data; and forming a compressed data set comprising the first status indicators, any second status indicators, any sub-group status indicators and the encoded non-zero values.

[0041] According to a fifth embodiment, there is provided a method performed by a processing element for decompressing compressed activation data read from storage for a neural network, the activation data formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: obtaining, from the compressed activation data, a first status indicator for each of a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; if the first status indicator indicates that there are no data elements within the group having a non-zero value, assigning a zero value to each data element within the group; if the first status indicator indicates that each group of data elements contains at least one data element having a non-zero value, obtaining, from the compressed activation data, a second status indicator for each of a plurality of subgroups within the group; if the second status indicator indicates that there are no data elements within the subgroup having a non-zero value, assigning a zero value to each data element within the subgroup; for the case that the second status indicator indicates that the data elements within each subgroup have a non-zero value, obtaining a subgroup indicator for the subgroup from the compressed activation data and assigning a zero value to the data elements within the subgroup in accordance with the subgroup indicator; decoding an ordered set of data elements having a non-zero value within the compressed activation data; and forming decompressed activation data by assigning the decoded data elements having a non-zero value to locations that have not been assigned to data elements having a zero value.

[0042] The groups of data elements, the subgroups of data elements and the data elements within the compressed activation data can form a quadtree structure. The unit of data can comprise sixty-four data elements, the unit of data can be divided into four groups of sixteen elements each, and the groups can be subdivided into subgroups of four data elements each.

[0043] Within the compressed data set, the first status indicators can be grouped into a first set of status indicators, and the second status indicators can be grouped into a second set of status indicators. The first status indicators, the second status indicators and the subgroup status indicators can be arranged in one of row-major order or column-major order in the compressed data set.

[0044] The step of assigning the decoded data elements having a non-zero value to locations that have not been assigned to data elements having a zero value can comprise assigning the decoded data elements to locations that have not been assigned to data elements having a zero value in one of row-major order or column-major order.

[0045] According to a sixth embodiment, there is provided a method performed by a processing element for decompressing compressed activation data read from a storage device for a neural network, the activation data formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: obtaining, from the compressed activation data, a first status indicator for each of a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; if the first status indicator indicates that each group of data elements contains at least one data element having a zero value, obtaining, from the compressed activation data, a second status indicator for each of a plurality of subgroups within the group; for the case that the second status indicator indicates that each subgroup contains a data element having a zero value, obtaining, from the compressed activation data, a subgroup indicator for the subgroup and assigning, in dependence on the subgroup indicator, a zero value to the data elements within the subgroup; decoding an ordered set of data elements having non-zero values within the compressed activation data; and forming decompressed activation data by assigning, to a location that has not been assigned to a data element having a zero value, a decoded data element having a non-zero value.

[0046] According to a seventh embodiment, there is provided a data processing system comprising a processing element and a storage device storing a code portion which, when executed by the processing element, causes the data processing system to perform a method for decompressing compressed activation data, the activation data formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: obtaining, from the compressed activation data, a first status indicator for each of a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; if the first status indicator indicates that there is no data element having a non-zero value within the group, assigning a zero value to each data element within the group; if the first status indicator indicates that each group of data elements contains at least one data element having a non-zero value, obtaining, from the compressed activation data, a second status indicator for each of a plurality of subgroups within the group; if the second status indicator indicates that there is no data element having a non-zero value within the subgroup, assigning a zero value to each data element within the subgroup; for the case that the second status indicator indicates that the data elements within each subgroup have a non-zero value, obtaining, from the compressed activation data, a subgroup indicator for the subgroup and assigning, in dependence on the subgroup indicator, a zero value to the data elements within the subgroup; decoding an ordered set of data elements having non-zero values within the compressed activation data; and forming decompressed activation data by assigning, to a location that has not been assigned to a data element having a zero value, a decoded data element having a non-zero value.

[0047] According to an eighth embodiment, there is provided a data processing system comprising a processing element and a storage device storing a code portion which, when executed by the processing element, causes the data processing system to perform a method for decompressing compressed activation data, the activation data being formed from units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: obtaining, from the compressed activation data, a first status indicator for each of a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; if the first status indicator indicates that each group of data elements contains at least one data element having a zero value, obtaining, from the compressed activation data, a second status indicator for each of a plurality of subgroups within the group; for the case that the second status indicator indicates that the data elements within each subgroup have a zero value, obtaining, from the compressed activation data, a subgroup indicator for the subgroup and assigning, in dependence on the subgroup indicator, a zero value to the data elements within the subgroup; decoding an ordered set of data elements having non-zero values within the compressed activation data; and forming decompressed activation data by assigning the decoded data elements having non-zero values to the positions that have not been assigned to data elements having a zero value.

[0048] According to a ninth embodiment, there is provided a data processing system for compressing activation data, wherein the activation data is formed from units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the data processing system comprising a processing means configured to: form a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; identify whether there are any data elements within each group that have a non-zero value and form a first status indicator for each group, the first status indicator indicating whether the group contains data elements having a non-zero value; form a plurality of subgroups within each group of data elements that are identified as containing at least one data element having a non-zero value; identify whether there are any data elements within each subgroup that have a non-zero value and form a second status indicator, the second status indicator indicating, for each subgroup, whether the subgroup contains data elements having a non-zero value; for each subgroup that contains data elements having a non-zero value, form a subgroup status indicator for the subgroup, the subgroup status indicator indicating which data elements within the subgroup have a non-zero value; encode the non-zero values of the data elements in the data unit; and form a compressed data set comprising the first status indicators, any second status indicators, any subgroup status indicators and the encoded non-zero values.

[0049] According to a tenth embodiment, there is provided a data processing system for compressing activation data from a neural network, the activation data formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the data processing system comprising processing means configured to: form a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; identify whether there are any data elements within each group that have a zero value and form a first status indicator for each group indicating whether the group contains data elements having a zero value; form a plurality of subgroups within each group of data elements identified as containing at least one data element having a zero value; identify whether there are any data elements within each subgroup that have a zero value and form a second status indicator indicating for each subgroup whether the subgroup contains data elements having a zero value; for each subgroup containing data elements having a zero value, form a subgroup status indicator for the group indicating which data elements within the subgroup have a zero value; encode non-zero values for data elements in the data unit; and form a compressed data set comprising the first status indicators, any second status indicators, any subgroup status indicators, and the encoded non-zero values.

[0050] According to an eleventh embodiment, there is provided a data processing system for decompressing compressed activation data, the activation data formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the data processing system comprising processing means configured to: obtain from the compressed activation data a first status indicator for each of a plurality of groups of data elements within a unit of activation data, each group comprising a plurality of data elements; if the first status indicator indicates that there are no data elements within the group having a non-zero value, assign a zero value to each data element within the group; if the first status indicator indicates that each group of data elements contains at least one data element having a non-zero value, obtain from the compressed activation data a second status indicator for each of a plurality of subgroups within the group; if the second status indicator indicates that there are no data elements within the subgroup having a non-zero value, assign a zero value to each data element within the subgroup; for the case where the second status indicator indicates that data elements within each subgroup have a non-zero value, obtain from the compressed activation data a subgroup indicator for the subgroup and assign a zero value to data elements within the subgroup according to the subgroup indicator; decode an ordered set of data elements having non-zero values within the compressed activation data; and form decompressed activation data by assigning to a location that has not been assigned to a data element having a zero value a decoded data element having a non-zero value.

[0051] According to a twelfth embodiment, there is provided a data processing system for decompressing compressed activation data, the activation data formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the data processing system comprising processing means configured to: obtain, from the compressed activation data, a first status indicator for each of a plurality of groups of data elements within an activation data unit, each group comprising a plurality of data elements; if the first status indicator indicates that each group of data elements contains at least one data element having a zero value, obtain, from the compressed activation data, a second status indicator for each of a plurality of subgroups within the group; for the case where the second status indicator indicates that the data elements within each subgroup have a zero value, obtain, from the compressed activation data, a subgroup indicator for the subgroup and assign, in dependence on the subgroup indicator, a zero value to the data elements within the subgroup; decode an ordered set of data elements having non-zero values within the compressed activation data; and form decompressed activation data by assigning, to a location that has not been assigned to a data element having a zero value, a decoded data element having a non-zero value.

[0052] A particular embodiment will now be described with reference to the accompanying drawings.

[0053] Figure 2 Some, but not all, components 2 of a neural processing unit (NPU) are shown, which is a special-purpose chip for performing computations, and in particular computations relating to neural networks, in connection with artificial intelligence applications. In other words, the NPU allows hardware acceleration of certain computations relating to neural networks. The components 2 are components for writing activation values to DRAM (not shown) external to the NPU.

[0054] When performing computations relating to neural networks, computations can be performed for each layer of the neural network. These computations generate output, referred to as activation data, which can be large and needs to be stored before further computations can be performed using the data. Due to constraints on data transfer from external memory to the processor, storing and retrieving activation data from memory can be a relatively slow process. It is therefore desirable to compress the data from the activation layer using Golomb Rice codes in order to improve processor performance. Some other methods of compressing activation data can involve lossy compression. However, such ‘lossy’ compression can affect the accuracy of the neural network, which can not be acceptable for certain applications. The following compression scheme is a lossless scheme and is therefore expected to be acceptable for all applications.

[0055] The component 2 is configured to process activation data for storage. The activation data is received and grouped into units in the form of data blocks. A data block is defined as a group of 8x8 elements, where a data element is an 8-bit uncompressed activation data value. The activation data includes a zero point, which is a global parameter of the activation data indicating which of the unsigned values 0 to 255 is to be considered zero. The zero point has a value of 8 bits length. The selection of the value of the zero point can lead to an efficient compression of the activation data, but since this is not relevant for the implementation discussed herein, it will not be discussed further. Throughout this specification, the terms 'zero' and 'zero point' are used interchangeably. The processing element in the form of an encoder 20 is configured to compress the received activation data by converting the activation data into Golomb Rice codes. Then, further steps are performed, which will be described below, to make the compressed activation data easier to decode.

[0056] As indicated above, the embodiments described below make use of 8-bit activation data that is unsigned, i.e. only has positive values. In other implementations, the activation data can be signed (taking both positive and negative values), and the activation can have a different value range, such as 16-bit values. To handle signed data, a certain offset is added before compression and removed after decompression. For 8-bit signed numbers, the value 128 can be added during the signed to unsigned conversion (i.e. flipping the sign bit). Subsequently, the unsigned to signed conversion can be performed after decompression by flipping the 7th bit of the 8-bit value.

[0057] Figure 3a The steps for compressing the activation data are shown. The compression of the data is performed block-wise. Thus, a block 31 is shown as input. During the compression of the activation data, a scheme selection 32 is performed. Then, the selected scheme 33 is encoded as 4 bits in the compressed activation data to be sent for storage in the DRAM. Figure 3bThe available schemes are shown in Table 1. The 'raw' scheme is the scheme that encodes the activation data simply without applying any compression. In case of errors, the 'raw' scheme can be used as a fallback scheme. If all activation data values within an activation data tile have a zero value, the 'zero' scheme is used. In this case, the tile can be compressed efficiently by simply indicating the selection of the zero scheme, which indicates that all values within the activation data tile are zero. Schemes 2 to 6 are used for delta Golomb-Rice encoding. This process will be explained in more detail below. The difference between delta GRC 1, delta GRC 2, delta GRC 3, delta GRC 4 and delta GRC 5 lies in the value of the divisor used to calculate the Golomb-Rice code. The Golomb Rice divisor varies with the number of the scheme, so for delta GRC 1, delta GRC 2, delta GRC 3, delta GRC 4 and delta GRC 5, the divisor is two, four, eight, sixteen and thirty-two respectively. Schemes 7 to 11 are used for masked delta Golomb-Rice encoding and schemes 12 to 15 are used for unmasked delta Golomb-Rice encoding. These encoding schemes will also be explained in more detail below. Again, the variants within the same type of scheme use different Golomb-Rice code divisors.

[0058] Delta Golomb-Rice encoding (schemes 2 to 6) will first be explained. Golomb-Rice codes are variable length codes that have a smaller size in case of encoding smaller values. Therefore, it is desirable to reduce the numerical value of the data to be encoded, if possible. Delta Golomb-Rice encoding makes a prediction of the values in the activation data tile and then determines the difference value from this prediction. Then, only the difference value is encoded, which tends to result in encoding smaller values and thus yields better encoding efficiency.

[0059] Figure 4 It is shown how the prediction for a data tile is made. An 8x8 data tile 31 has values elem[i][j], where i is the vertical coordinate and j is the horizontal coordinate. The data element in the upper left corner of the tile is elem[0][0]. The prediction for each entry in the tile is pred[i][j], where i is the vertical coordinate and j is the horizontal coordinate. The difference value delta[i][j] is the difference between elem[i][j] and pred[i][j].

[0060] delta[i][j] = elem[i][j] - pred[i][j]

[0061] To calculate pred[i][j], the following scheme is used. As Figure 4As shown, the data element located one position above is used to predict the element to the left of the block. This applies to all left edge elements except the top left corner element.

[0062] pred[i][j] = elem[i - 1][j], where j = 0 and i ≠ 0

[0063] The element of pred[i][j] on the top row is predicted using the data element located one position to the left. This applies to all top elements except the top left corner element.

[0064] pred[i][j] = elem[i][j - 1], where i = 0 and j ≠ 0

[0065] For the remaining elements of pred[i][j] except the top left corner element, a weighted combination of the data element located one position to the left, the data element located diagonally across from the top left corner, and the data element located above the predicted element is used to make the prediction.

[0066] pred[i][j] = 3 * elem[i][j - 1] + 2 * elem[i - 1][j - 1] + 3 * elem[i - 1][j] » 3

[0067] The above formula indicates that the left and top data element values are multiplied by three and the top left corner data element is multiplied by two in the linear combination. The last feature (» 3) indicates that the calculation is done in binary and represents a bit shift of 3 places. This has the effect of dividing by 8 while maintaining the integer value of pred[i][j].

[0068] The top left most data value is not predicted and is stored in the original format. This data value forms Figure 3a the 8-bit initval 34 shown in the top data grid.

[0069] Reference will now be made to Figure 5 provide a working example of delta encoding. As noted above, delta encoding is performed on an 8 x 8 block of activation data. However, to keep the description simple, an explanation of delta encoding on a 4 x 4 grid of data will be given. Figure 5 The top data grid shown in the top data grid is the original data used for delta encoding. This data forms the elem[i][j] as described above.

[0070] In the middle data grid, the predicted values pred[i][j] are shown. Since the top row of the elem[i][j] data is all zero values, the pred[i][j] values on the top row are also all zero values. Looking at the left column, there is only one non-zero value in the elem data. pred[3][0] is equal to elem[2][0], which has a value of 1.

[0071] Looking at the values far left and top edge, the calculation of the three non-zero values in pred[i][j] will be explained. The value of pred[3][2] is 4 and is calculated as follows:

[0072] pred[3][2] = 3*0 + 2*0 + 3*11 » 3

[0073] = 33 » 3

[0074] = 100001 » 3

[0075] = 100

[0076] = 4

[0077] The value of pred[2][3] is 5 and is calculated as follows:

[0078] pred[2][3] = 3*11 + 2*0 + 3*3 » 3

[0079] = 40 » 3

[0080] = 101000 » 3

[0081] = 101

[0082] = 5

[0083] The value of pred[3][3] is 19 and is calculated as follows:

[0084] pred[3][3] = 3*23 + 2*11 + 3*22 » 3

[0085] = 157 » 3

[0086] = 10011101 » 3

[0087] = 10011

[0088] = 19

[0089] Figure 5 The bottom data grid in is delta[i][j] and this is simply the difference between elem[i][j] and pred[i][j]. The value of delta[i][j] is less than the value of elem[i][j] which helps with efficient Golomb-Rice encoding.

[0090] In the delta Golomb-Rice coding scheme (schemes 2 to 6), the data values delta[i][j] are converted to Golomb-Rice codes. However, to do this, it is preferable that all the values to be encoded are positive as this avoids the need to use a sign bit. Therefore, a signed to unsigned conversion is performed as follows:

[0091] udelta[i][j] = delta[i][j] < 0? -2 * delta[i][j] - 1 : 2 * delta[i][j]

[0092] The logic here is that if delta[i][j] is zero, then udelta[i][j] is zero. Otherwise, positive values of delta[i][j] are doubled and negative values of delta[i][j] are multiplied by -2 and subtracted by 1 from them. This maps positive delta[i][j] values to even numbers on the positive axis and negative delta[i][j] values to odd numbers on the positive axis. The values of udelta[i][j] are then Golomb-Rice encoded.

[0093] As mentioned previously, Figure 3b The difference between schemes 2 to 6 in is the value of the divisor used to generate the Golomb-Rice code. Once the values of udelta[i][j] have been converted to Golomb-Rice codes, the compressed data is formed by the scheme value (4 bits), the initial value (initval) corresponding to the top left data element (elem[0][0]), and the Golomb-Rice codes corresponding to udelta[i][j].

[0094] The masked delta Golomb-Rice encoding schemes (schemes 7 to 11) will now be described. In addition to the delta encoding step, these schemes include a masked encoding step. Referring to Figure 3a , the masked delta Golomb-Rice encoding includes a delta encoding 35, a signed to unsigned conversion 36, a conversion to Golomb-Rice codes 37, and an additional masked encoding step 38.

[0095] Figure 6 The masked process is illustrated. Figure 6 The top of shows a grid of data elements where each square cell represents an 8x8 activated data block. Many neural networks produce activation data that includes many zero values. For example, a neural network can use a ReLU activation function that tends to produce a sparse activation map. The zero values are in Figure 6 The grid in is shown with unfilled cells for zero values and filled for activation data values that have any non-zero value. From Figure 6 As can be seen from the sparse example in, there are many zero values. The masked process provides a way to encode the location of the zero values as will now be explained.

[0096] The first mask is generated by considering the 8x8 grid as a collection of groups of data elements in the form of the following four 4x4 grids: the top-left 4x4 grid, the top-right 4x4 grid, the bottom-left 4x4 grid, and the bottom-right 4x4 grid. Each 4x4 grid is checked to see if there are any non-zero values. In Figure 5 In the example shown, there are non-zero values in each of the top-left, top-right, and bottom-left grids. However, the bottom-right 4x4 grid does not include any non-zero values. Thus, a 4x4 mask

[1110] is formed. The first status indicator in the form of a bit value of ‘1’ indicates a 4x4 grid that will be further subdivided because that grid contains non-zero values, and a first status indicator in the form of a bit value of ‘0’ indicates a 4x4 grid in which all data elements have zero values.

[0097] For each 4x4 grid that includes non-zero values, a subgroup in the form of a 2x2 grid is formed: top-left, top-right, bottom-left, and bottom-right. Each of these sub-grids is checked to see if they contain non-zero values. This process is performed in row-major order on the 4x4 grids that are found to have non-zero values. For the top-left 4x4 grid, each 2x2 grid includes non-zero value elements. Thus, the 2x2 mask for this grid is

[1111] . A second status indicator in the form of a bit value of ‘1’ indicates a 2x2 grid that is further subdivided because that grid contains at least one non-zero value, and a second status indicator in the form of a bit value of ‘0’ indicates that all data elements in a 2x2 grid have zero values. The top-right 4x4 grid includes a single 2x2 grid in which the bottom-right corner has no non-zero elements. Thus, the 2x2 mask for this 4x4 grid is

[1110] . Looking at the bottom-left 4x4 grid, only the top-left 2x2 grid contains non-zero values. Thus, the 2x2 mask for this 4x4 grid is

[1000] . This gives us Figure 6 the 2x2 mask [1111 1100 1000] shown. Note that the line breaks in the 2x2 mask shown below the Figure 6 grid shown are shown to facilitate readability.

[0098] For those 2x2 grids that include non-zero values, a subgroup indicator in the form of a 1x1 mask is created that shows the location of zero values and non-zero values within each 2x2 grid. This is done in row-major order for each 4x4 grid, so the top-left 2x2 grid in the top-left 4x4 grid is encoded as

[0001] , the top-right 2x2 grid is encoded as

[0001] , the bottom-left 2x2 grid is encoded as

[0101] , and the bottom-right 2x2 grid is encoded as

[1101] . A ‘1’ designates a data element with a value other than zero, and a ‘0’ indicates a value equal to zero. Figure 6The line breaks in the 1x1 mask in the middle show the 1x1 masks in the different 4x4 grids (top left, top right and bottom left). The overall mask encoding is the concatenation of the 4x4 mask, the 2x2 mask and the 1x1 mask.

[0099] Once the zero values have been encoded in the masks, as described above, there is still a need to encode the non-zero values so that their values and their positioning are included in the compressed activation data. This is done in a similar way as described in the reference Figure 4 encoding described in the explanation. However, the delta values are only calculated for the non-zero values, not for all data elements within the zone. For each non-zero value, pred[i][j] is calculated and the difference value delta[i][j] is calculated. The value delta[i][j] is made unsigned by converting to udelta[i][j] as described above, and then converted to a Golomb-Rice code using the divisor specified for the selected scheme (Mask Delta GRC 1 to Mask Delta GRC 5).

[0100] The compressed activation data for the Mask Delta GRC schemes includes a scheme identifier (4 bits), an initial value for the delta encoding (initval, 8 bits) and a mask encoding that encodes the positioning of the zero values. The mask encoding is a 4x4 mask (4 bits), a 2x2 mask (bit count = 4*4 of the bit count of the 4x4 grid) and a 1x1 mask (bit count = 4*2 of the bit count of the 2x2 grid). A feature of masks in this form is that all of the content is 4-bit aligned, which makes it subsequently easier to parse by an NPU to decompress the activation data.

[0101] The inverse mask delta Golomb-Rice encoding schemes (schemes 12 to 15) will now be described. These schemes are closely related to the mask delta Golomb-Rice encoding schemes. In the mask delta Golomb-Rice encoding schemes described above, the positioning of the non-zero values is indicated in the 4x4 and 2x2 masks. However, if the non-zero values are more numerous than the zero values, it can be more efficient to encode the positioning of the zero values within the zone rather than the positions of the non-zero values. Figure 7 Figure 6 The same data grid shown in Figure 1 will be used to describe the inverse mask delta Golomb-Rice encoding. This explanation will also show that the mask delta Golomb-Rice encoding is more efficient than the inverse mask delta Golomb-Rice encoding when the zero values are more numerous than the non-zero values.

[0102] Looking at Figure 7 , the same process will be performed as for Figure 6 ​The same process is described, with the difference being that it is determined whether there are any zero values in each 4x4 grid, rather than whether there are any non-zero values in each 4x4 grid. As before, consider the top-left 4x4 grid, the top-right 4x4 grid, the bottom-left 4x4 grid, and the bottom-right 4x4 grid. Each 4x4 grid is checked to see if there are any zero values. In Figure 7 In the example shown, there are zero values in all of the grids. Thus, a 4x4 mask

[1111] is formed. The '1' bit value indicates that there is a zero value within the 4x4 grid and that grid will be subdivided, and the '0' value indicates that the 4x4 grid is formed only of non-zero values.

[0103] For each 4x4 grid that includes zero values, 2x2 grids are formed: top-left, top-right, bottom-left, and bottom-right. Each of these 2x2 grids is checked to see if they contain zero values. This process is performed in row-major order on the 4x4 grids that contain zero values. The '1' bit value indicates that there is a zero value within the 2x2 grid, and the '0' value indicates that the 2x2 grid is formed only of non-zero values. For the top-left 4x4 grid, each 2x2 grid includes zero value elements. Thus, the 2x2 mask for this grid is

[1111] . The top-right 4x4 grid includes a single 2x2 grid in which there is no zero value element in the top-left corner. Thus, the 2x2 mask for this 4x4 grid is

[0111] . Looking at the bottom-left 4x4 grid, all of the 2x2 grids have zero value data elements. Thus, the 2x2 mask for this 4x4 grid is

[1111] . Also, the bottom-right 4x4 grid includes zero value elements in each 2x2 grid. This gives a mask of

[1111] for the bottom-right 4x4 grid. This gives a 2x2 mask of Figure 7 [1111011111111111] is shown. Note that the line breaks in the 2x2 mask Figure 7 The line breaks in the 2x2 mask shown below the grids in

[0104] For those 2x2 grids that include zero value data elements, a 1x1 mask is created that shows the location of zero values and non-zero values within each 2x2 grid. This is done in row-major order for each 4x4 grid, starting with the top-left 4x4 grid. In the top-left 4x4 grid, the top-left 2x2 grid is encoded as

[1110] , the top-right 2x2 grid is encoded as

[1110] , the bottom-left 2x2 grid is encoded as

[1010] , and the bottom-right 2x2 grid is encoded as

[0010] . Figure 7 The line breaks in the 1x1 mask in Figure 7 The '1' value indicates a zero value, and the '0' value indicates a non-zero value. The overall mask encoding is the concatenation of the 4x4 mask, the 2x2 mask, and the 1x1 mask.

[0105] As with the masked delta Golomb-Rice encoding, the non-zero values are delta encoded in the same way as described in connection with Figure 4 However, the delta values are only calculated for the non-zero values, not for all data elements within the block. For each non-zero value, pred[i][j] is calculated and the difference value delta[i][j] is calculated. The value delta[i][j] is made unsigned by converting to udelta[i][j] as described above in connection with the delta Golomb-Rice encoding, and then converted to a Golomb-Rice code using the divisor specified for the selected scheme (inverse masked delta GRC 1 to inverse masked delta GRC 5).

[0106] The compressed activation data consists of a scheme identifier (4 bits), an initial value for the delta encoding (8 bits), and the above described inverse mask.

[0107] As explained in connection with Figure 2 The selection between the different encoding schemes is made based on which scheme can most efficiently encode the activation data block. The decision can be made by a deterministic process (i.e. calculating the length of the bit stream that would be created using each scheme). The bit counts are:

[0108] Delta encoded bit count: Golomb-Rice bit count + scheme bits + initval

[0109] Masked delta bit count: mask bit count + Golomb-Rice bit count (non-zero elements) + scheme bits + initval

[0110] Inverse masked delta bit count: inverse mask bit count + Golomb-Rice bit count (non-zero elements) + scheme bits + initval

[0111] The compression of the activation data using one of the delta Golomb-Rice encoding, the masked delta Golomb-Rice encoding, and the inverse masked delta Golomb-Rice encoding Figure 3b schemes 2 to 15 shown above generates Golomb-Rice codes and other data. The other data (which is either the scheme bits and initval or the scheme bits, initval, and mask data) will be referred to as SIM data in the following.

[0112] In general, delta Golomb-Rice encoding is efficient for slowly changing activation data, masked delta Golomb-Rice encoding is efficient for sparse data including many zero values, and unmasked delta Golomb-Rice encoding is efficient for data with some sparsity. For example, if there are more than 32 non-zero values in an activation data tile, then unmasked delta Golomb-Rice encoding can be efficient, otherwise masked delta Golomb Rice encoding will likely be more efficient.

[0113] When decoding variable length codes such as Golomb Rice codes, it is difficult to parse at high rates. This is because there is a sequence dependency between Golomb Rice codes such that the length of the previous Golomb Rice code needs to be known in order to identify and decode the next Golomb Rice code. Thus, when parsing directly using a single parser, a typical hardware implementation for decoding Golomb Rice codes can achieve a rate of 1 or 2 Golomb Rice codes per clock cycle. For this reason, the Golomb-Rice codes and SIM data are further processed before being stored in DRAM, as will now be described.

[0114] Figure 8 is a flowchart showing the steps performed by the encoder 20 on the compressed activation data. In step S80, the Golomb Rice codes are obtained for each activation data tile, in this case by conversion by the encoder 20. Then, in step S81, the encoder 20 separates the Golomb Rice codes into a stream of unary values and a stream of remainder values, and stores them in Figure 2 the RAM 21 shown.

[0115] Figure 9 Three data streams are shown. The source data stream 90 is a stream of Golomb Rice codes. The source data stream 90 includes a series of Golomb Rice codes indicated by the values GR1 to GR5. Each Golomb Rice code has a variable length unary part and a fixed length binary part of the type described in the Related Art Description. Although Figure 9 three bit fixed length binary parts are shown in the Related Art Description, the length of the binary part is not important and other lengths can be used. The encoder 20 separates the Golomb Rice codes into two parts to generate Figure 9 the other two streams 91 and 92 shown. The first stream 91 is a stream of unary values and the second stream 92 is a stream of remainder values, each binary value having a fixed length.

[0116] In step S82,Figure 2 The splicing processor 22 shown splices the first stream, the second stream and the SIM data stream together to form a processed stream. This is done on a cell by cell basis, where each cell represents 32 blocks of uncompressed data (2,048 elements) stored in a 2112 byte slot. The slot is larger than the cell in order to allow for certain overheads and rounding to an integer number of 64 bytes.

[0117] Each cell is formed by the splicing processor 22 into a number of chunks. The first chunk of a cell always includes a header. The binary data from the stream 90 is always spliced by the splicing processor 22 into the chunks of the processed stream in 32 bit portions.

[0118] The cells are formed by the splicing processor 22 using a set of stream control rules, as will now be described with reference to Figure 10 to Figure 11 the description. Figure 10 Different structures of the chunks that the splicing processor 22 can use to form a cell are shown, and Figure 11 the steps performed by the splicing processor 22 in forming a cell are shown.

[0119] As mentioned above, the first chunk of a cell must include a header that provides information about the length of the cell and the lengths of the binary sub-streams included within the cell. The length of the remainder value within the cell is not included in the header, but can be derived from the length of the cell and the lengths of the binary sub-streams.

[0120] Figure 10 The available chunk formats are shown, which have been designed for parsing efficiency to allow each chunk to be consumed in a single clock cycle of a decoder that decodes Golomb-Rice codes. The chunk structures shown are divided into three categories: the first two chunk structures 100 and 101 are for use when selecting the first chunk in a cell, the next three chunk structures 102 to 104 are for use when transmitting the first chunk in a block, and the bottom five chunk structures 105 to 109 are extended chunks for use when otherwise emitting binary and remainder data. The top two chunk structures 100 and 101 include a 32 bit length header portion, which is required at the start of a cell. The chunk structures 102 to 104 that form the first chunk encoding a new block within a cell include SIM data at the front of the chunk, which can be followed by remainder data and / or binary data.

[0121] In order to select the appropriate chunk structure to use, the encoder 20 uses a set of stream control rules. Once the chunk structure of a chunk has been identified, the chunk structure can be filled with the appropriate data and emitted. The stream control rules used by the splicing processor 22 are as follows. When selecting the first chunk structure for a cell, if less than 64 bits of SIM data are transmitted in conjunction with the first block of the cell, then the Figure 10The first chunk structure 100 shown, there is at least 32 bits of binary data available for the cell, as at the start of the header, and the combined size of the SIM data and the remainder data of the first chunk of the cell is at least 64 bits. Otherwise, the second first chunk structure 101 is used to form the first chunk of the cell. In Figure 10 The chunk structure portion labelled 'SIM...REM' is initially filled with SIM data until there is no SIM data remaining, and then filled with remainder data to fill the allocated bits within the chunk.

[0122] For chunks in the cell that are issued by the encoder 20 after the first chunk has been issued, which still contain compressed activation data relating to the first chunk in the cell, the extended chunk 105 to 109 is used. If there is still binary data of the first chunk to be sent, and there is still more than 128 bits of binary data to be included in the cell, then the chunk structure 105 is used. If there is still 96 bits of binary data to be encoded for the cell, then the chunk structure 106 is used, if there is still 64 bits of binary data to be encoded for the cell, then the chunk structure 107 is used, and if there is still 32 bits of binary data to be encoded for the cell, then the chunk structure 108 is used. In the case where all of the binary data of the chunk has been encoded, then the chunk structure 109 is used to issue the remainder data of the chunk, including the remainder data of the first chunk of the cell. It should be noted that the binary data and remainder data included in the above chunk structures is selected irrespective of the element grouping of the following chunk and the following activation data, such that binary data from different chunks and / or elements can be included in the same chunk.

[0123] With all SIM data, unary data, and remainder data of the first block already included in the block, the first block of the next block is selected from block structures 102 to 104. The value 'CR' representing the carried remainder is monitored by the encoder. The carried remainder is the number of remainder bits that the block has already transmitted in the block structure before the first block of this block. Similarly, the value 'CU' representing the carried unary data is monitored by the encoder. The carried unary data is the number of unary bits that the block has already transmitted in the block structure before the first block of this block. When selecting the first block structure for a new block, block structure 102 is selected if: the carried remainder is less than a threshold of the buffer threshold representing the remainder value; the carried unary data is less than a threshold of the buffer threshold representing the unary value; the new block still has at least 32 bits of unary data to transmit; and the amount of SIM and remainder data to be transmitted in the new block is greater than 96 bits. In this case, block structure 102 can be used because both unary and remainder data can be transmitted with the SIM data. If block structure 102 is not used, and the remainder carried is less than the buffer threshold of the remainder value, then block structure 103 is used to transmit SIM data, followed by the remainder data. Otherwise, block structure 104 is used, and SIM data is transmitted only within the block.

[0124] Figure 11 The above method for forming compressed activation data for a cell is shown in Figure S110. In step S110, from... Figure 10 The block structures 100 and 101 shown select the first block structure. This selection depends on the selection criteria described above. After selecting the block structure, the stitching processor 22 creates a header section. The stitching processor 22 evaluates the length of the unary substream to be included in the cell based on 32 uncompressed data blocks and adds information indicating that length to the header section. The stitching processor 22 evaluates the length of the SIM data substream to be included in the cell based on 32 uncompressed data blocks and adds information indicating that length to the header section. The length of all data to be included in the cell is also evaluated and added to the header section. Data from the first unary data stream 41 (if needed) and data from the second remainder data stream are added to the block according to the selected block structure chosen by the stitching processor 22.

[0125] In step S111, an appropriate block structure is selected from block structures 102 to 109 according to the flow control rules described above to form the next block of the processed data stream. After selecting the block structure, a block is formed by filling the relevant parts of the block structure with data from the first unary data stream 91, data from the second remainder data stream 92, and SIM data.

[0126] In step S112, the stitching processor 22 determines whether there is more data to be formed into a block to complete the cell. If there is more data to be formed into a block, the method proceeds to S111 to form the next block. If there is no more data to process, the method proceeds to S113, where the method ends for that cell. This method is performed for multiple cells until the compressed activation data has been encoded.

[0127] The above method assumes that a complete 128-bit block can be formed from 32 blocks in a cell, and that the unary portion can be concatenated into the 32-bit portion. In reality, these conditions may not be met. In such cases, stop bits '0' are used to pad the first unary portion stream and the second remainder portion stream until they reach the desired size. Since the length of the data added to the cell is stored in the header, it is possible to identify the length of the data within the cell and determine where to begin padding when decoding the processed data stream.

[0128] Then, the compressed activation data, which has been encoded into cells by the splicing processor 22, is stored in DRAM.

[0129] Next, we will refer to Figure 12 to Figure 14 This describes a method for decoding stored compressed activation data using a decoder. In this case, the decoder is part of the NPU, allowing activation data to be read from DRAM for further computation. The decoder stores... Figure 10 A copy of the block structure shown is used by encoder 20 to store compressed activation data in DRAM. Figure 12 In step S120, the decoder receives the first block of the cell containing the activation data for decoding from RAM. The decoder reads the header and identifies the length of the unary data in the cell, the length of the SIM data in the cell, and the total cell length. The decoder also determines the length of the remainder in the cell by subtracting the lengths of the other components in the cell from the total cell length. In step S121, the decoder identifies whether the length of the unary data specified in the header is 32 bits or greater, whether the SIM data in the cell is less than 64 bits, and whether the SIM data plus the remainder data in the cell is greater than 64 bits. If the conditions in S121 are met, the decoder uses block structure 100 for decoding. Otherwise, the decoder uses block structure 101 for decoding. If there is still data for the first block based on the received SIM data, then according to... Figure 13The next chunk is selected using the illustrated process. When decoding the data, the decoder maintains a parameter U_remaining which is initially set to the value of the length of the alphabet in the cell when the header of the cell is examined and is updated each time an alphabet data is retrieved from a chunk to record the amount of alphabet data remaining in the cell. Thus, if the chunk structure 100 is used for the first chunk, then after 32 bits of alphabet data is extracted from the first chunk, the parameter U_remaining will decrease by 32.

[0130] At this stage, the decoder has received the SIM of the first zone and can determine how many alphabet portions (alphabet symbols) and remainder values it will receive for the first zone. If the decoder determines that the first zone still has data to receive, then Figure 13 The method used by the decoder to determine the extended chunk structure of the next chunk is illustrated. In step S130, the next chunk of active data is received. Based on the number of decoded alphabet portions (alphabet symbols), it is determined whether the first zone still has alphabet data to issue. If the first zone still has alphabet to issue, then the method proceeds to step S131. In step S131, the parameter U_remaining is checked to determine whether the amount of alphabet data to be extracted from the cell is greater than or equal to 128 bits. If the amount of alphabet data to be extracted is greater than or equal to 128 bits, then the decoder determines that the chunk structure 105 is being used. In step S132, the decoder extracts data from the chunk according to the chunk structure 105 and updates the parameter U_remaining to account for the amount of alphabet data extracted.

[0131] If the amount of alphabet data to be extracted is less than 128 bits, then the method proceeds to step S133. In step S133, the parameter U_remaining is checked to determine whether the amount of alphabet data to be extracted from the cell is equal to 96 bits. If the amount of alphabet data to be extracted is equal to 96 bits, then the decoder determines that the chunk structure 106 is being used. In step S134, the decoder extracts data from the chunk according to the chunk structure 106 and updates the parameter U_remaining to account for the amount of alphabet data extracted.

[0132] If the amount of alphabet data to be extracted is less than 96 bits, then the method proceeds to step S135. In step S135, the parameter U_remaining is checked to determine whether the amount of alphabet data to be extracted from the cell is equal to 64 bits. If the amount of alphabet data to be extracted is equal to 64 bits, then the decoder determines that the chunk structure 107 is being used. In step S136, the decoder extracts data from the chunk according to the chunk structure 107 and updates the parameter U_remaining to account for the amount of alphabet data extracted.

[0133] If the binary data to be extracted is less than 64 bits, the method proceeds to step S137. In step S137, the parameter U_remaining is checked to determine if the amount of binary data to be extracted from the cell is equal to 32 bits. If the amount of binary data to be extracted is equal to 32 bits, the decoder determines that the chunk structure 108 is being used. In step S136, the decoder extracts data from the chunk according to the chunk structure 108 and updates the parameter U_remaining to account for the amount of binary data extracted.

[0134] If in step S130, the decoder determines that there is no binary remainder to issue based on the number of decoded binary portions (binary symbols), but there is a remainder data to issue for the first block, the decoder determines that the chunk structure 109 is being used. In step S139, the decoder extracts data from the chunk according to the chunk structure 109.

[0135] When the decoder determines that all values for a block have been received based on the number of decoded binary portions (binary symbols) and the received remainder data, the next chunk structure is determined based on the logic shown. In step S140, it is determined that the first chunk in a new block is being sent. As with the encoder, the decoder monitors the value 'CR' representing the carried remainder. The carried remainder is the number of remainder bits of the next block that have been received in the chunk structure prior to the first chunk of the next block. Similarly, the value 'CU' representing the carried binary is monitored by the decoder. The carried binary is the number of binary bits of the next block that have been received in the chunk structure prior to the first chunk of the next block. Figure 14

[0136] In step S141, it is determined whether the carried remainder is less than the buffer value for the carried remainder. The buffer value for the carried remainder is the same as the buffer value for the carried remainder used by the encoder. It is also determined whether the carried binary is less than the buffer value for the carried binary. Again, the buffer value for the carried binary is the same as the buffer value for the carried binary used by the encoder. The decoder also determines whether there is more than 32 bits of binary data remaining in the cell and whether the amount of SIM plus remainder data in the cell is greater than 64 bits. If all of these conditions are met, the decoder decides that the chunk structure 102 is the structure for the next chunk.

[0137] If the conditions in step S141 are not met, the method proceeds to step S143. In S143, if the carried remainder is less than the buffer value for the carried remainder, the chunk structure 103 will be used to decode the next chunk of data. Otherwise, the chunk structure 104 is used to decode the next chunk of data.

[0138] Based on the above, the decoder determines the next chunk structure based on the logic shown in FIG. 7. Figure 12 to Figure 14 ​The described process, the decoder can efficiently recreate the first Radix data stream 41, the second Residue data stream 42 and the SIM data from the processed data stream stored in DRAM by the encoder 20. In decoding the 32 data blocks from a cell, when decoding the first group of blocks in a cell, the logic shown in Figure 12 is used and thereafter, when decoding the first group of blocks or the extended group of blocks of the next cell, the logic shown in Figure 13 and Figure 14 is used. By using the flow control rules shown in Figure 12 to Figure 14 the decoder can decide the type of data that will be found at any point in the incoming stream without the need to indicate the bit cost of the type of data within the incoming compressed data stream.

[0139] After extracting the Radix and Residue data from the processed data stream, the decoder needs to decode the Golomb-Rice codes to recreate the non-zero values. The second Residue data stream 42 is an array of fixed length binary values and is decoded directly using techniques known in the art. Therefore, this process is not discussed further herein.

[0140] The decoding of the first Radix data stream 91 is next described in relation to Figure 15a In step S150, the 8-bit block of Radix data is converted to binary form which indicates the location or locations of the stop bit within the binary block. This is done by using a look-up table. In step S151, since the Radix code can span 8-bit blocks, four 8-bit blocks are combined into a 32-bit block and then four 32-bit blocks are combined into a 128-bit block. The 128-bit block is likewise a list of stop bit location. To retrieve the Radix value, in step S152, the difference between each adjacent stop bit location value is taken which gives the value of the Radix code.

[0141] The method of Figure 15a can be achieved by using a look-up table to analyse the 8-bit block of Radix data into an intermediate form. This is shown in Figure 15b where the top row 154 identifies the bit positions within each byte shown below it. The first bit in each byte is bit '0' and the last bit is bit '7'. The second row 155 shows the byte of Radix data. It will be recalled that the stop bit in the Radix data is '0'. In the intermediate form shown in the third row 156, each byte is expanded into a list of up to eight 3-bit codes which indicate the location of the stop bit within the byte. Radix-4 combinations of four 8-bit segments to 32-bit segments are performed and the Figure 15bshown in the fourth row 97 and the fifth row 98. In the fourth row 157, a pair of identification bits are added as the most significant bits (MSB) to the code. For the first byte, the value '00' is added to the 3-bit code, for the third byte, the value '10' is added to the 3-bit code, etc. In the fifth row 158, the 5-bit codes are concatenated to form a list of stop bit position within the 32-bit word. The subsequent Radix-4 combination of the four 32-bit segments to the 128-bit segment uses a similar process to generate a list of 7-bit codes indicating the location of the unary stop bit. As in step S152, the subtraction of the adjacent values yields the length of the unary data and thus the value of the unary data.

[0142] The recreation of the activation data is performed block by block and varies according to the scheme used to encode the activation data. The scheme can be read from the scheme bit in the SIM data of the block. The scheme identifier also identifies the divisor used to create the Golomb-Rice code, allowing the Golomb-Rice code to be decoded once the length of the unary and binary remainders have been identified. The value recovered from the Golomb-Rice code is converted back to a signed data value using the following logic:

[0143] delta[i][j] = udelta[i][j] = odd value? -0.5 * (udelta[i][j] + 1) : 0.5 * udelta[i][j]

[0144] The logic is that if udelta[i][j] has an odd value, it corresponds to a negative delta[i][j] value. Therefore, 1 is added to the udelta value and it is multiplied by -0.5. If udelta[i][j] has an even value, it corresponds to a positive delta[i][j] value. The positive delta[i][j] value is recovered as 0.5 times the udelta[i][j] value.

[0145] In the case where the activation data has been incrementally encoded (schemes 2 to 6), the original data can be recovered as follows. Referring again to Figure 4 , the top left value elem[0][0] is known because it is stored in the SIM data as initval. Based on this value, elem[1][0] in the left-hand column can be determined by determining that pred[1][0] is equal to initval and using the following equation:

[0146] elem[i][j] = delta[i][j] + pred[i][j]

[0147] Once elem[l][0] is determined, it is possible to compute elem[2][0] following the left-hand edge, and so on. Similarly, elem[0][l] on the top edge can be determined based on initval in the compressed activation data and the value delta[0][l]. Thus, the values of elem[i][j] for the top row can be recovered. The remaining values of elem[i][j] diagonal away from the top-left corner can be determined.

[0148] Figure 16 The order in which the values elem[i][j] are recovered is shown in FIG. 6. The values along the left-hand edge are computed first and are shown with the numeral 1 in FIG. 6. Next, the values along the top edge are computed and can also be computed by starting with elem[0][l] and working along the edge. This is shown with the numeral '2' in FIG. 6. Once the elem[i][j] values for the top edge and the left edge of the block are known, it is possible to determine the values of elem[i][j] diagonal away from the top-left corner because the values of elem[i][j] necessary to determine pred[i][j] become known. These computations are shown in order by the numerals '3' through '15' in FIG. 6. In this way, the activation data is recovered according to the delta Golomb Rice encoding scheme (schemes 2 through 6). Figure 16 Figure 16 Figure 16

[0149] The process for recovering the mask and anti-mask activation data is similar to the process described above for delta-encoded activation data. However, because the zero values are not delta-encoded, they are recovered first. In the case of the anti-mask encoding scheme (schemes 12 through 15), the mask indicates the location of the zero values. Thus, it is straightforward to identify from the 4x4 mask which 4x4 cells include zero values, which 2x2 cells within the 2x2 cells include zero values, and the actual positioning of the zero values from the 1x1 mask. For the mask scheme (schemes 7 through 11), the positioning of the non-zero values is identified from the 4x4, 2x2, and 1x1 masks, and then the mask is inverted to identify the location of the zero values. Once the zero values of elem[i][j] are known, the process of recovering the delta-encoded values can be performed as described above in connection with the description of the delta Golomb Rice encoding scheme. Figure 16

[0150] As described above, the first particular implementation combines the integer and remainder portions of the Golomb Rice codes within a cell. Each cell can include an integer and remainder portion. Mixing the integer and remainder portions of the activation data within a cell tends to spread the integer and remainder portions across the processed data stream retrieved from the DRAM. This allows the size of the parsing buffer at the decoder for storing the integer data prior to decoding to be reduced, thereby reducing the hardware requirements.

[0151] ​​​​The above embodiments should be understood to be illustrative. Additional embodiments are contemplated. For example, the above embodiments describe a 4x4 grid of data elements. Other embodiments contemplate a different grid size. For example, a 2x2 grid of data elements can be used. In this case, the 2x2 grid is encoded first, followed by the 1x1 grid. This is referred to as breadth first traversal, as all 2x2 grids are evaluated first, followed by the 1x1 grid. An alternative encoding method is possible, which would be referred to as depth first traversal. In depth first traversal, each 2x2 block is considered at each depth level (1x1) before moving to the next 2x2 grid. For example, for the top left 2x2 grid shown, the 2x2 encoding would be 1, as the grid includes a data element with a non-zero value. The next bit in the encoding would be for the 1x1 grid and would be [0], as the top left 1x1 grid does not include a data element with a non-zero value. The next bit in the encoding would be for the bottom left 1x1 grid, which would be encoded as [1]. Thus, the top left grid would be encoded as

[011] . After this, the top right 2x2 grid would be encoded at all depth levels, the bottom left grid would be encoded at all depth levels, and the bottom right grid would be encoded at all depth levels. Figure 6 and Figure 7 The mask encoding described above encodes the 4x4 grid values first, then the 2x2 grid values, and then the 1x1 grid values. This is referred to as breadth first traversal, as all 4x4 grids are evaluated first, followed by all related 2x2 grids, and then the related 1x1 values. An alternative encoding method is possible, which would be referred to as depth first traversal. In depth first traversal, each 4x4 block is considered at each depth level (2x2 and 1x1) before moving to the next 4x4 grid. For example, for the top left 4x4 grid shown, the 4x4 encoding would be 1, as the grid includes a data element with a non-zero value. The next bit in the encoding would be for the 2x2 grid and would be

[1111] , as each 2x2 grid within the top left grid includes a data element with a non-zero value. The next bit in the encoding would be for the 1x1 grid, which would be encoded as [0001000101011101]. Thus, the top left grid would be encoded as [111110001000101011101]. After this, the top right 4x4 grid would be encoded at all depth levels, the bottom left grid would be encoded at all depth levels, and the bottom right grid would be encoded at all depth levels. Figure 6

[0152] The above depth first encoding compared to the breadth first encoding described in the first embodiment shows that breadth first encoding can be preferable for the NPU, as it has well aligned 4-bit fields, which allows for a more efficient parser design.

[0153] The above embodiments describe an implementation in which there are 32 blocks in a cell. This is merely an example, and different implementations can include a different number of blocks per cell. Similarly, the above embodiments include data elements with 8-bit values. Depending on the implementation, the data elements can have a different value range. Figure 10 The chunk structure shown illustrates chunks that can take 128-bit SIM, remainder, or unary data. In other implementations, the chunk structure can vary to have different sizes and / or have different structure configurations.

[0154] The above embodiments describe data blocks that consist of an array of 8x8 elements. In other implementations, the block size can be 4x4. In such an implementation, the first mask described above and generated by treating the 8x8 grid as a set of data element groups in the form of four 4x4 grids is unnecessary when performing mask encoding. After this, the 4x4 blocks are masked by 4-bit 2x2 masks and 16-bit 1x1 masks.

[0155] ​In further embodiments, the size of the tiles can vary depending on the type of layer of the neural network being processed. For example, the mask encoding described above can be used for some layers of the neural network, but a 2x2 tile size can be used for fully connected layers with a batch size of four.

Claims

1. A method performed by a processing element for compressing activation data of a neural network to be written to a storage device, wherein the activation data is formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: forming within a unit of activation data a plurality of groups of data elements, each group comprising a plurality of data elements; identifying whether there are any data elements within each group that have a non-zero value and forming a first status indicator for each group, the first status indicator indicating whether the group contains data elements having non-zero values; forming within each group of data elements that is identified as containing at least one data element having a non-zero value a plurality of subgroups; identifying whether there are any data elements within each subgroup that have a non-zero value and forming a second status indicator, the second status indicator indicating for each subgroup whether the subgroup contains data elements having non-zero values; for each subgroup that contains data elements having non-zero values, forming a subgroup status indicator for the subgroup, the subgroup status indicator indicating which data elements within the subgroup have non-zero values; encoding the non-zero values of the data elements in the data unit; and forming a set of compressed data comprising the first status indicators, any second status indicators, any subgroup status indicators, and the encoded non-zero values.

2. The method of claim 1, wherein the groups of data elements, the subgroups of data elements, and the data elements form a quadtree structure.

3. The method of claim 1, wherein within the set of compressed data, the first status indicators are grouped into a set of first status indicators, the second status indicators are grouped into a set of second status indicators.

4. The method of claim 3, wherein the first status indicators, the second status indicators, and the subgroup status indicators are arranged in the set of compressed data in one of row-major or column-major order.

5. The method of claim 1, wherein the encoding of the non-zero values of data elements comprises delta encoding the non-zero values, wherein the delta encoding is performed by taking the difference between an actual value of each data element and a predicted value of the data element.

6. The method of claim 5, wherein the data unit comprises a square number of data elements, and except for the upper left corner data element, the predicted value of each data element on the left hand edge of a square of data elements is equal to the value of the data element positioned one position above the predicted data element.

7. The method of claim 6, wherein the prediction for a data element within the square of data elements is based on a linear combination of the values of the data elements positioned one position to the left of the data element, one position above the data element, and diagonally one position to the upper left of the data element.

8. The method of claim 5, wherein the delta-encoded non-zero values are encoded as Golomb-Rice codes, and prior to converting the delta-encoded non-zero values to Golomb-Rice codes, the delta-encoded non-zero values are converted to unsigned form such that all of the delta-encoded non-zero values are converted to integers greater than or equal to zero.

9. A method performed by a processing element for decompressing compressed activation data read from storage for a neural network, the activation data formed in units of data elements, each unit including a plurality of data elements and each data element having a numerical value, the method comprising: obtaining from the compressed activation data a first status indicator for each of a plurality of groups of data elements within an activation data unit, each group including a plurality of data elements; if the first status indicator indicates that there are no data elements within the group having a non-zero value, assigning a zero value to each data element within the group; if the first status indicator indicates that each group of data elements contains at least one data element having a non-zero value, obtaining from the compressed activation data a second status indicator for each of a plurality of subgroups within the group; if the second status indicator indicates that there are no data elements within the subgroup having a non-zero value, assigning a zero value to each data element within the subgroup; for the case where the second status indicator indicates that data elements within each subgroup have a non-zero value, obtaining from the compressed activation data a subgroup indicator for the subgroup and assigning to data elements within the subgroup zero values in accordance with the subgroup indicator; decoding an ordered set of data elements within the compressed activation data having non-zero values; and forming decompressed activation data by assigning to locations not already assigned to data elements having zero values decoded data elements having non-zero values.

10. A data processing system comprising a processing element and storage, the storage storing a portion of code which when executed by the processing element causes the data processing system to perform a method of compressing activation data for a neural network, wherein the activation data is formed in units of data elements, each unit including a plurality of data elements and each data element having a numerical value, the method comprising: forming a plurality of groups of data elements within an activation data unit, each group including a plurality of data elements; identifying whether there are any data elements within each group having a non-zero value and forming a first status indicator for each group, the first status indicator indicating whether the group contains data elements having non-zero values; forming a plurality of subgroups within each group of data elements identified as containing at least one data element having a non-zero value; identifying whether there are any data elements within each subgroup having a non-zero value and forming a second status indicator, the second status indicator indicating for each subgroup whether the subgroup contains data elements having non-zero values; for each subgroup containing data elements having non-zero values, forming a subgroup status indicator for the subgroup, the subgroup status indicator indicating which data elements within the subgroup have non-zero values; encoding the non-zero values of data elements in the data units; and forming a compressed data set comprising the first state indicators, any second state indicators, any subgroup state indicators, and the encoded non-zero values.

11. A data processing system comprising a processing element and a storage device storing a code portion which, when executed by the processing element, causes the data processing system to perform a method for decompressing compressed activation data, the activation data being formed of units of data elements, each unit comprising a plurality of data elements and each data element having a numerical value, the method comprising: obtaining from the compressed activation data a first state indicator for each of a plurality of groups of data elements within an activation data unit, each group comprising a plurality of data elements; if the first state indicator indicates that there are no data elements within the group having a non-zero value, assigning a zero value to each data element within the group; if the first state indicator indicates that each group of data elements contains at least one data element having a non-zero value, obtaining from the compressed activation data a second state indicator for each of a plurality of subgroups within the group; if the second state indicator indicates that there are no data elements within the subgroup having a non-zero value, assigning a zero value to each data element within the subgroup; for the case that the second state indicator indicates that the data elements within each subgroup have a non-zero value, obtaining from the compressed activation data a subgroup indicator for the subgroup and assigning a zero value to the data elements within the subgroup in accordance with the subgroup indicator; decoding an ordered set of data elements within the compressed activation data having non-zero values; and forming decompressed activation data by assigning to the locations that have not been assigned to data elements having a zero value, the decoded data elements having a non-zero value.

Images