SYSTEM AND METHOD FOR COMPRESSING AND DECOMPRESSING OFF-CHIP DATA FOR MACHINE LEARNING NETWORKS - Patent application

Abstract

A system and method for compressing and decompressing a data stream for use by a machine learning network is provided, the method including: encoding each value of the data stream, determining a mapping to one of a plurality of non-overlapping ranges, where each value is encoded as a symbol representing the range and a corresponding offset; arithmetically encoding the symbols using a probability count; storing a compressed data stream including the arithmetically encoded symbols and corresponding offsets; decoding the compressed data stream with arithmetically decoding using the probability counts, where the arithmetically decoded symbols use offset bits to arrive at a decoded data stream; and communicating the decoded data stream for use by the machine learning network.

Description

[Technical field]

[0001] The following relates generally to data compression techniques, and more specifically to systems and methods for compressing and decompressing off-chip data for machine learning networks. [Background technology]

[0002] Data compression in memory hierarchies is attractive for deep learning (DL) workloads and accelerators where memory accesses account for a large proportion of overall energy consumption. First, data compression at any level of the hierarchy can potentially increase its effective capacity by encoding each value using fewer bits. Second, it can improve effective latency and energy efficiency by reducing slower and more energy-intensive accesses to higher hierarchies, including accesses to off-chip memory. Third, compression can reduce the number of bits read and written per value, improving effective bandwidth and energy efficiency. Finally, compression naturally complements blocks for data flow and data reuse, which are frontline techniques for increasing the energy efficiency of deep learning memory hierarchies. Summary of the Invention

[0003] In one aspect, a method is provided for compressing and decompressing a data stream for use by a machine learning network, the data stream including weight data or activation data for use by the machine learning network, the method including receiving a data stream to be encoded, determining a mapping of each value of the data stream to one of a plurality of non-overlapping ranges, each value being encoded as a symbol representing a range and a corresponding offset, determining, arithmetically encoding the symbols using probability counts, storing a compressed data stream including the arithmetically encoded symbols and corresponding offsets, decoding the compressed data stream with arithmetic decoding using the probability counts, the arithmetically decoded symbols being used with offset bits to arrive at a decoded data stream, decoding, and communicating the decoded data stream for use by the machine learning network.

[0004] In this particular case of the method, the probability counts are represented by ranges of values ​​that represent the approximate frequency of occurrence of the symbols in the input samples.

[0005] In another version of this method, the probability counts are initialized with a uniform value distribution.

[0006] Yet another version of this method uses probability counts of the input samples to determine the non-overlapping ranges.

[0007] In yet another version of this method, encoding each value in the data stream includes encoding multiple values ​​simultaneously using pipelining, and decoding the compressed data stream includes decoding multiple values ​​simultaneously using pipelining.

[0008] In yet another version of this method, the data stream includes a tensor, and the method further includes splitting the tensor into a number of subtensors, each subtensor being encoded and decoded in a respective pipeline.

[0009] In yet another version of this method, the method further includes encoding the corresponding offset using two levels of encoding.

[0010] In yet another version of this method, each of the non-overlapping ranges is defined by a respective minimum value, a maximum value, and an offset length.

[0011] In yet another version of this method, the non-overlapping ranges are ordered, with the maximum value of each range being the minimum value minus one of the next ranges in the sequence.

[0012] In yet another version of this method, the symbol includes the minimum value of the mapped range and the corresponding offset includes the minimum value of the mapped range subtracted from the value from the data stream.

[0013] In yet another version of this method, encoding each value of the data stream includes determining a probability count of the activation data using profiling.

[0014] In another aspect, a system is provided for compressing and decompressing a data stream for use by a machine learning network, the data stream including weight data or activation data for use by the machine learning network, the system including: one or more encoders receiving a data stream to be encoded and encoding each value of the data stream, the encoding including determining a mapping to one of a plurality of non-overlapping ranges, each value being encoded as a symbol representing a range and a corresponding offset; and arithmetically encoding the symbols using a probability count; a memory storage storing a compressed data stream including the arithmetically encoded symbols and corresponding offsets; and one or more decoders decoding the compressed data stream with arithmetic decoding using the probability counts and communicating the decoded data stream for use by the machine learning network, the decoder using the arithmetically decoded symbols and offset bits to arrive at the decoded data stream.

[0015] In the particular case of this system, the probability counts are represented by ranges of values ​​that represent the approximate frequency of occurrence of the symbols in the input samples.

[0016] In another version of this system, the probability counts are initialized with a uniform value distribution.

[0017] Yet another version of this system uses probability counts of the input samples to determine the non-overlapping ranges.

[0018] In yet another version of this system, multiple values ​​in the data stream are simultaneously encoded by each of multiple encoders, and multiple values ​​in the compressed data stream are simultaneously decoded by each of multiple decoders.

[0019] In yet another version of this system, the data stream includes a tensor and the method further includes splitting the tensor into a number of subtensors, each subtensor being encoded by a separate encoder and decoded by a separate decoder.

[0020] In yet another version of this system, the encoder further encodes the corresponding offset using two levels of encoding.

[0021] In yet another aspect of this system, each of the non-overlapping ranges is defined by a respective minimum value, a maximum value, and an offset length.

[0022] In yet another version of this system, the symbol includes the minimum value of the mapped range and the corresponding offset includes the minimum value of the mapped range subtracted from the value from the data stream.

[0023] In yet another version of this system, the non-overlapping ranges are ordered, with the maximum value of each range being the minimum value minus one of the next ranges in the sequence.

[0024] In yet another version of this system, the encoder encodes each value in the data stream using profiling to determine a probability count of the activation data.

[0025] These and other aspects are contemplated and described herein. It will be understood that the foregoing Summary of the Invention illustrates exemplary aspects of systems and methods to aid those skilled in the art in understanding the following detailed description.

[0026] A better understanding of the embodiments may be obtained with reference to the drawings. [Brief description of the drawings]

[0027] [Figure 1]FIG. 1 illustrates a block diagram of a system for compressing and decompressing data streams used by a machine learning network, according to an embodiment. [Diagram 2] 1 shows a flowchart diagram of a method for compressing and decompressing data streams used by a machine learning network, according to an embodiment. [Diagram 3] 1 shows a block diagram of an exemplary systolic array accelerator. [Figure 4] 2 shows a block diagram of another exemplary systolic array accelerator using the system of FIG. 1. [Diagram 5] 1 is a graph showing an example cumulative distribution of occurrence frequencies for values ​​of two layers of a machine learning network. [Figure 6] 2 shows a block diagram of an interface to an exemplary embodiment of an encoder according to the system of FIG. 1; [Figure 7] 2 shows a block diagram of an exemplary embodiment of the internal workings of an encoder according to the system of FIG. 1; [Figure 8] 2 shows a block diagram of an exemplary structure of a SYMBOL lookup block according to the system of FIG. 1; [Figure 9] 2 shows a block diagram of an exemplary structure of a PCNT table block according to the system of FIG. 1. [Figure 10] 2 shows a block diagram of an exemplary structure of a Hi / Lo / Code Generation block according to the system of FIG. 1; [Figure 11] 2 shows a block diagram of an interface to an exemplary embodiment of a decoder according to the system of FIG. 1; [Figure 12] 2 shows a block diagram of an exemplary implementation of the internal workings of a decoder according to the system of FIG. 1; [Figure 13] 13 is a graph showing an example of the energy consumed by off-chip memory transfers for the weights of a DNN model normalized against a baseline that does not implement a compression technique. [Figure 14]13 is a graph showing an example of the energy consumed by off-chip memory transfers for a DNN model run-time activation, normalized against a baseline that does not implement a compression technique. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] For simplicity and clarity of description, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or similar elements. Additionally, numerous specific details are set forth to provide a thorough understanding of the embodiments described herein. However, it will be understood by those skilled in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Additionally, the description is not intended to limit the scope of the embodiments described herein.

[0029] Various terms used throughout this specification may be construed and understood as follows, unless the context dictates otherwise: "or" as used throughout is considered inclusive and states "and / or." Singular articles and pronouns used throughout include their plurals and vice versa. Similarly, gender pronouns include their counterparts, so pronouns should not be understood as limiting what is described herein to the use, implementation, performance, etc., of a single gender. Further definitions of terms may be set forth herein, which may also apply to the instances before and after these terms, as will be understood upon reading this specification.

[0030] Any of the instruction executing modules, units, components, servers, computers, terminals, or devices illustrated herein may include or access computer readable media, such as storage media, computer storage media, or data storage devices (removable and / or non-removable), e.g., magnetic disks, optical disks, or tapes. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that may be used to store desired information that may be accessed by an application, module, or both. Any such computer storage media may be part of the device or may be accessible or connectable to the device. Additionally, unless the context clearly indicates otherwise, any processor or controller described herein may be implemented as a single processor or multiple processors. The multiple processors may be collocated or distributed, and any processing functions referred to herein may be performed by a single or multiple processors even though a single processor is illustrated. Any methods, applications, or modules described herein may be implemented using computer readable / executable instructions that may be stored or carried by such computer readable media and executed by a single or multiple processors.

[0031] Since off-chip access is generally orders of magnitude more expensive than on-chip access (e.g., older technology nodes accessing 32b from DRAM cost 640pj vs. 5pj from 8KB of SRAM), compression methods for neural networks generally focus primarily on off-chip access. The success of these compressions has increased the relative energy cost of on-chip access. For example, some approaches quantize the parameters (weights) into a 16-entry dictionary of 16-bit values, which are then encoded using a Huffman code. These approaches require an additional training step and only target the modal weights. Other approaches target zeros, while others also target 0 or 1 prefixes. These approaches either do not compress activations, but impose additional constraints on the values ​​of the weights, or target specific expected value patterns. Advantageously, embodiments of the present disclosure exploit any non-uniform behavior in the value distribution of the weights and values. Arithmetic coding (AC) is built on a lossless data encoding / decoding approach that can achieve near-optimal coding efficiency in some cases. Using AC for machine learning workloads presents significant challenges because: (1) AC implementations tend to be costly due to the need for expensive operations such as division, and (2) AC is inherently sequential, which conflicts with the needs of the broad range of target workloads, which are high-bandwidth.

[0032] The embodiments of the present disclosure provide a lossless off-chip memory compression approach that may be directly compatible with any accelerator, including, for example, SIMD units such as those used in graphics processors with Tensor cores, systolic arrays such as those in the TPU family, and lattice-like processing units with or without sparsity support such as SCNN.

[0033] Referring to FIG. 1, a diagram of a system 100 for compressing and decompressing off-chip data of a machine learning network is shown, according to an embodiment. The system 100 is composed of dedicated hardware circuits of multiple encoders 102 and decoders 104, each of which communicates with one or more memory units 106 (e.g., registers) comprising a temporary or non-transitory computer-readable medium. In further cases, the system 100 can be implemented with a processor, microprocessor, microcontroller, other integrated circuit, or the like. The system 100 may include other components, as applicable or appropriate, such as a local bus that enables the dedicated hardware circuits of the encoders 102 and decoders 104 to receive inputs (e.g., from a shared memory 140), communicate with the memory units 106, and output encoding / decoding (e.g., to a buffer and / or processing unit 150). In some cases, the system 100 may include a pre-processing module 108 in communication with the encoders 102 and decoders 104 and the inputs.

[0034] The system 100 performs a chip memory compression method for fixed-point quantized models. The method reduces data width by taking advantage of non-uniform value distribution in deep learning applications. In this manner, the method can be used to increase effective memory capacity, reduce off-chip traffic, and / or achieve desired performance / energy goals while using smaller off-chip memory. Referring to FIG. 2, a flow chart of a method 200 for compressing and decompressing off-chip data for machine learning networks is shown therein.

[0035] In block 202, a data stream is received from a buffer of the processing unit 150 or from the processing unit itself. The processing unit causes the execution of the machine learning network to occur. The values ​​of the data stream may include weights and / or activation related data of the machine learning network, for example in tensor format.

[0036] FIG. 3 shows an example of a systolic array accelerator, and FIG. 4 shows an example of a systolic array accelerator with the system 100 located between the on-chip and off-chip memory hierarchies and the processing units, where it decodes and encodes values ​​read from or written to the off-chip. The system 100 can work with any desired data flow, transparently packing data in the memory so that it can be input to the corresponding units at relatively little cost. The system 100 may be able to maintain high concurrency and wide data width accesses using several encoder and decoder units. The off-chip memory hierarchy generally still sees a regular flow of wide accesses, although there are fewer accesses. The off-chip accesses generally remain the same width as they would be if the system 100 were not introduced. Since the on-chip memory hierarchy and the processing units are usually directed to the original values, the system 100 can be plug-in compatible and substantially transparent, except for the timing differences when accessing the off-chip memory system.

[0037] To achieve high compression ratios, the encoder 102 exploits the naturally occurring data value distribution of machine learning workloads. Rather than targeting specific cases such as zero, or a prefix of 0s or 1s, as targeted by other approaches, the encoder 102 instead uses the fact that the distribution is far from uniform. For example, as described herein, values ​​found in binary formats often tend to be heavily clustered near two extremes, upper and lower values, such as all 0 values ​​or all 1 values. This does not mean that the values ​​are necessarily all 0s or all 1s, it simply means that the values ​​tend to be close to these two patterns (when sorted numerically). This approach may share some similarities with arithmetic codes, which can seamlessly exploit any non-uniformity in value distributions, but implementations of arithmetic codes are generally expensive and typically low bandwidth. Arithmetic codes generally use expensive operations such as division, and the implementations often process one input value during encoding and generate a few bits at a time during decompression. However, such limitations are not suitable for the usage context of the system 100. On the one hand, system 100 deploys a lightweight approach where the energy efficiency benefits of compressing data far outweigh the costs of performing the compression / decompression. Furthermore, in contrast to arithmetic coding, system 100 can handle machine learning workloads that are typically highly data-parallel, and thus favor multiple extensive data accesses that translate into high-bandwidth memory system needs. In this way, system 100 can achieve the benefits possible with arithmetic coding while enabling a low-cost, highly concurrent, high-bandwidth implementation.

[0038] As shown in the example of Figure 4, in some cases, system 100 may introduce multiple decoder and encoder units that transform data as it travels to and from off-chip. In the example of Figure 4, the rest of the accelerator, including the memory hierarchy, may still see the same sequence of incoming data and may still write data as it would otherwise. In this case, system 100 is a lossless approach since it only modifies the data content as it is transferred from / to off-chip.

[0039] In block 206, each encoder 102 encodes an incoming value v received from the data stream as a (symbol, offset) pair, where v=symbol+offset. Symbol is a bit prefix of the original value v. For example, assuming 8-bit data, it is b7...b s0...0. The encoder 102 encodes the input data stream containing tensors into two streams, one containing the symbol sequence and the other containing the corresponding offset. An arithmetic code can be used on the symbol stream. The arithmetic code can use a statically derived probability count and symbol assignment table (called a probability count table). In block 208, the encoder can store the probability count table in the memory 106. Each probability count represents a range of values ​​instead of an individual value. This contributes to achieving a low-cost implementation with little loss of compression efficiency. A heuristic search algorithm can be used to build the probability count table. In the case of neural networks, the heuristic search algorithm can operate at the level of layers, although other groupings are possible. The heuristic search algorithm receives a sample input and uses the distribution of values ​​to divide the input value range into several subranges, each characterized by a starting value that also corresponds to a symbol prefix plus an offset. For each subrange, the encoder 102 assigns a probability count that represents the approximate frequency with which a value in the subrange appears in the input sample.

[0040] In some cases, for values ​​generated at run time, such as activations, the encoder 103 may use profiling to build a probability count table. In some cases, the system 100 is unaware of the probability counts for activations, as they are calculated at run time. Running a tabulation algorithm can be prohibitively expensive and must be fixed before running the network. For profiling, the neural network can be run on a particular input (e.g., a sample image), and all activations for this input are recorded. A probability count table is determined for this input and used for subsequent runs. This profiling assumes that the probability counts determined for the sample image are a representative approximation of those that occur for other input images. This is because while individual values ​​can vary significantly with different input images, when viewed as a group, the distribution remains similar.

[0041] System 100 can seamlessly decode input values ​​as they are read from off-chip memory and encode output values ​​as they are written to off-chip memory using efficient, typically low-cost hardware encoders 102 and decoders 104. While different groupings are possible, for clarity of explanation, this disclosure assumes that processing proceeds layer-by-layer.

[0042] At block 210, the encoder 102 outputs the encoded symbols and offsets to a memory storage, such as the shared memory 140. At block 212, the decoder receives the encoded symbols and offsets from the shared memory as a data stream that is decoded back into weight and / or activation related data, e.g., back into tensor form.

[0043] In block 214, the contents of the probability count table are loaded from the memory 106 into the decoder 104. In block 216, the decoder 104 uses the probability count table to decode the input data. The input data is encoded into two streams, one for the symbol and one for the corresponding offset. In most cases, both streams are read sequentially, with each decoder 104 generating one value per step. In certain cases, to maintain the bandwidth typically required for machine learning workloads, multiple decoders 104 can operate in parallel. To this end, the input data stream can be split into separate substreams, each of which is encoded and decoded independently. Each decoder 104 can be time-multiplexed across multiple streams. Furthermore, several such decoders 104 can operate in parallel. At the output of each layer, one or more encoders 102, each of which may be time-multiplexed across several substreams, encode values ​​into a symbol stream and an offset stream before outputting (e.g., before writing to memory).

[0044] In block 218, the decoder 104 outputs the decoded data stream to a buffer of the processing unit 150, or to the processing unit itself, for use in executing the machine learning network.

[0045] The encoder 102 and decoder 104 may possibly use fixed-point logic and respective tables, e.g., containing 16 rows of 10-bit and 11-bit values, respectively. These tables may be shared between multiple streams. In this example, encoding each stream requires three 16-bit and two 8-bit register memory units 106 to maintain state. Decoding each stream requires three 16-bit and one 8-bit register memory units 106. The encoders and decoders may be pipelined, allowing the system 100 to time-multiplex the encoders and decoders, but only need to replicate these state registers for each stream. This allows the system 100 to meet the needs of a typical workload using fewer encoders and decoders. Additionally, pipelining may reduce the amount of work per stage, and therefore the amount of work per cycle, reducing latency constraints on the logic, thereby enabling even greater energy efficiency.

[0046] Advantageously, system 100 can increase effective off-chip capacity and bandwidth without requiring changes to neural network models. In this manner, neural network architects can deploy the present embodiment at design time to reduce the amount of off-chip memory and therefore the cost required to meet desired performance goals. Advantageously, the computing system as a whole requires less frequent off-chip, and rewards optimization of established models such as quantization and pruning, without requiring it for every model. System 100 may benefit from quantization approaches that reduce bit widths, and generally rewards further evolution to even shorter bit widths. In particular, system 100 provides a substantial advantage over any neural network approach that results in a more biased value distribution.

[0047] As described herein, the inventors implemented the compressor and decompressor in Verilog and performed experimental examples of the embodiment to show its performance and energy efficiency at a 65 nm technology node. As can be seen, the system 100 was able to reduce the data footprint (weights and activations) by 45%, 30%, and 69%. Adapting and compressing the model using a more aggressive quantization method reduces the data footprint by 83% and 47% for BiLSTM, pruned GoogleNet, MobileNet, NCF, and Q8BERT. For example, for the quantized version of ResNet18, we reduce the data footprint by 54% and 36% for Intel's INQ and IBM's PACT, respectively. For clarity, this disclosure assumes that the input values ​​are 8-bit integers. However, values ​​of any suitable length, longer or shorter, can be used.

[0048] Figure 5 shows an example cumulative distribution of the frequency of occurrence for values ​​in two layers. The first distribution shown is layer 10 from Q8BERT, and the second is layer 1 from BILSTM. The activation and weight distributions are shown separately. The distribution is far from uniform, with about half of the values ​​tending to be close to zero and the other half or so tending to be close to 255. Traditional memory hierarchies do not take full advantage of this property, typically storing all imap or fmap elements using a data width that is long enough to accommodate the possible values. Some compression approaches take advantage of such a distribution, for example by storing a prefix of 0 (for groups close to zero) and 1 (for groups close to 255). However, if the length of the values ​​is explicitly encoded, the values ​​are first grouped and a common length is used for all values ​​in the group. This reduces the coding efficiency for three reasons: (1) the bit length field is a fixed overhead, since it is necessary to specify a length that can be one of 1 to 8; and (2) the bit length used for each group is dictated by the value with the highest magnitude. (3) A bit vector per group with one bit per value identifies those that are zero. Ideally, the system 100 wants to encode values ​​using as few bits as possible. Arithmetic coding advantageously has the length of the coding representation used scale by the frequency of each value in the original stream. It can therefore outperform other frequency-dependent coding approaches by being able to allocate even a small number of bits to each input value. This is not possible with other frequency-based approaches that use bit strings to represent each value and scale the length of each string by the frequency of the value.

[0049] Off-chip compression of deep learning workloads presents a variety of challenges and opportunities. Indexing is one of these challenges, as compression methods for general-purpose systems typically need to support random, fine-grained access. This requires the ability to quickly and finely find compressed values ​​in memory. This typically forces general-purpose compression approaches to use small containers. To support random access, some compression approaches reduce the amount of data transferred, but do not reduce the size of the containers used for storage. For example, they encode data within a cache line, so fewer bits must be read or written. However, the entire cache line is still reserved. Thus, they only improve bandwidth and not effective capacity. Other approaches use a level of indirection to identify where the data is currently located in memory, further increasing capacity. However, these require a careful balance between flexible placement and metadata overhead. Some approaches use coarse-grained containers when accessing off-chip. However, this comes at a heavy price whenever the system does not access the entire container.

[0050] Concurrency and payload are another challenge. While the memory hierarchy of a general-purpose system needs to support a few processing cores, deep learning is typically suited to large-scale data parallel execution. To maintain performance, this requires a highly concurrent and wide payload memory hierarchy. If the data is uncompressed, this is easily achieved by using several wide on-chip memory banks. Individual data values ​​can be ordered in their memory such that they are directly aligned with the inputs of the corresponding functional units, avoiding lateral data movement. However, when the data is compressed, this alignment is usually broken. Since wiring delays and energy are major considerations at various technology nodes, care must be taken to avoid data movement over long distances, if possible. To gain the benefits of high compression, while avoiding expensive on-chip crossbars that support lateral movement of wide data payloads, the system 100 can target off-chip memory compression, where the cost of compression / decompression can be easily justified given the high cost of off-chip access. Advantageously, this allows the system 100 to use several limited decompressors / compressors. This is possible because (1) machine learning workloads are massively data parallel and therefore typically latency tolerant, and (2) they typically exhibit regular, mostly a priori known access patterns. Thus, the inputs and outputs can be split into multiple contiguous substreams that can then be decoded and encoded in parallel to meet the bandwidth requirements imposed by the workload on the off-chip memory system.

[0051] Another challenge concerns content. In general, compression approaches for general-purpose systems exploit value behaviors found in “typical” programs, such as complete or partial value redundancy. For example, memory pointers tend to share prefixes (e.g., pointers to stack- or heap-allocated structures). Programs often use aggregate data structures that tend to exhibit partially repeated value patterns (e.g., flag fields). Compression approaches must handle a variety of data types, such as integers and floating-point numbers, or characters from various character sets (e.g., UTF-16). In addition, programs typically manage data types with various power-of-two data widths, such as 8 bits, 16 bits, 32 bits, or even larger. Finally, programmers often use a “default” integer or floating-point data type (currently 32 or 64 bits). Compression approaches must handle and exploit these characteristics. The majority of values ​​in deep learning workloads are for fmaps and imaps, which are large arrays of short fixed-point values. 8 bits, for example, or even 4 bits are possible (16 bits is still required for some models, e.g. for certain segmentation models, where even a small loss of accuracy translates into very noticeable artifacts).

[0052] An embodiment of the system 100 utilizes arithmetic coding. Arithmetic coding converts an input sequence of symbols (or values) into a value between 0 and 1. This value uniquely identifies this particular sequence of input values ​​and can therefore be decoded to recreate them. The bit length of the coded values ​​depends not only on the sequence of values ​​but also on their frequency.

[0053] As shown in Algorithm 1 provided below, arithmetic coding is a process of coding the symbols {vs ,..., v0}∈V, the input symbol S=s N ...si...0 sequence, one range for each symbol in S and V j ,plow j) table. It outputs a code value, a fractional number in [0,1), that uniquely represents the input sequence S. Internally, the coding uses two state variables, high and low. The encoding starts by setting the current range bounds to [0.0,1.0) in line 1. For each symbol s i is read in line 3, which is used to index into the range table in line 4. Line 5 calculates the length of the current range, range. The new bounds are offset from the current lower bound by adding the bounds scaled by the range symbol bounds. [Table 1]

[0054] To illustrate, consider an example where four values ​​A-D are to be encoded with probabilities of occurrence of 0.4, 0.1, 0.3, and 0.2, respectively. Arithmetic coding may involve assigning the range [0,0.4) to A, the range [0.4,0.5) to B, and the ranges [0.5,0.8) and [0.8,1.0) to C and D, respectively. In this way, the single value A can be represented by any number in the range [0,0.4), and B can be represented by any value in the range [0.4,0.5). This may seem inefficient, but only because it involves a single value. Instead, consider the following sequence: ABA. Arithmetic coding proceeds by initially maintaining the range [low,high), which is [0.0,1.0). First, encode A. This restricts the range to [0.0,0.4). To encode B, further restrict the range to indicate that B has occurred. This is done by adjusting the high and low marks to represent the subrange [0.4,0.5) of the current range [0.0,0.4). The new range is now [0.16,0.20). Any value within this range can be used to represent the string AB. To encode the next A, the coder now further restricts the current range so that its subrange corresponds to A's [0.0,0.4). Hence the range is now [0.16,0.176). Note that before A, the range width was 0.04=0.20-0.16, and A's high mark was 0, so a new high mark 0.20+0.04*0.4 is made from the new low (it remains 0.16, since A is assigned a range starting at 0). To encode the last B, the existing range must now be further restricted to its [0.8,1.0) subrange, or [0.2002,0.2025). Any value within this range can be used to represent the ABAC encoding sequence. This approach, while effective, has characteristics that make it undesirable for hardware implementation: (1) it requires infinite precision arithmetic, and (2) it uses a range table with one entry for each potential input value. Thus, it is generally unacceptably expensive for our purposes.

[0055] Approximations of infinite precision arithmetic may be possible. However, they are expensive and require execution times that may be proportional to the current precision. Even if the implementation costs are not prohibitive, as would be expected from encoding / decoding thousands of elements, the execution time and energy make this approach impractical. The system 100 advantageously utilizes an implementation of arithmetic coding that uses finite precision. In one embodiment, the system 100 implements a single-step encoding / decoding where all updates to the state (upper, lower, and code) are performed in a single step, and arithmetic coding is only used on the variable portion of the input stream by using probability counts to decide which prefix to encode. The bit length of this prefix generally depends on the expected frequency of occurrence of the value. For example, we use 16-bit wide upper, lower, and code registers, and 10-bit range bound table values.

[0056] Using a table with one entry for each potential input value is orders of magnitude more expensive in terms of energy and area. In most cases, the encoder 102 and decoder 104 process one symbol (value) per step, so it is desirable to minimize the area and energy costs so that units can be replicated to achieve high bandwidth data feeds. For an 8-bit model, the boundary table alone would require at least one table of 256 x 10b x 25Kb of storage. Even if area were not an issue, the energy required to access such a table would reduce and potentially eliminate any benefits gained from having the data stream off-chip. Instead of allocating a table entry per symbol, an embodiment of the system 100 could instead use a limited number of entries.

[0057] The encoder 102 divides the input value space into a number of non-overlapping ranges, for example 16 ranges. Each range has its minimum value v min , its maximum value v max, and an offset length OL in bits. All values ​​v in the range are encoded as (symbol, offset), where symbol = vmin and offset=v-symbol, which is an unsigned integer with OL bits. min is always of the form x...x0...0, i.e., a bit prefix of all values ​​with a range. In an exemplary implementation for values ​​with 8 bits or less, system 100 uses an 8-bit v min and v max , and a 3-bit OL, a range of 16 is used. In an implementation using 16-bit values, 16 ranges can be used, R min and OL are expanded to 16 bits and 4 bits, respectively.

[0058] Depending on the input sequence of values, the encoder 102 compresses it into two sequences. The encoder 102 processes each value one by one. Depending on the range to be mapped, it splits the values ​​into (symbol, offset) pairs. The encoder 102 then compresses the symbols (v min ) arithmetically. The encoded symbols comprise the first output stream and the corresponding offset comprises the second output stream. The encoding is complete when the last symbol is encoded. Along with the two encoded streams, the encoder 102 may also store metadata including, for example, (1) the number of encoded symbols (which can be used to terminate decoding) and (2) range and probability tables used in the arithmetic encoding.

[0059] For arithmetic coding, each value range has a probability count range (low i , high i) are also assigned. In an exemplary implementation, a 10-bit probability count is used, with a full range of (0x0, 0x3ff) assigned to the entire symbol. As described herein, heuristic algorithms can be used to determine a range of values ​​that reduces the overall footprint, including the encoding symbols and offset stream.

[0060] Table 1 shows an example of a probability count table used for the layer weights of a BILSTM. The fields "IDX" and "p" describe the row index and symbol probability, respectively, and are shown for clarity. It is not necessary to store them. Row 0 captures four values ​​in the range [0x00, 0x03], all associated with a probability count range [0x000, 0x1EB). If the maximum probability count is 210-1=0x3FF, this corresponds to a probability "p" of 0.4795. Any value in this range is mapped to symbol 0, which is the symbol probability that v is selected during decoding. min = 0×00. To recover the original number, OL=2b is remapped to v=v min+offset. Rows 3-12 are all assigned to the range of zero-length probability counts ([0x23A,0x23A] (where low=high). These are values ​​that do not appear at all in the input weight tensor. This is permissible since the weights are statically known. Row 13 captures all values ​​in the range [0xD0,0xF3] that map to symbol 13. Note that 6 bits are required for the offset because 0xF0-0xD0=0x23. Since 0x23<26-1, this means that not all offset values ​​will be used in this range. No special processing is required to ensure that this is the case. If the table is implemented with all the fields shown, entries with probability 0 can be omitted and the order of the rows can be freely changed. For example, if the search through the table is pipelined at the row level, the rows may be ordered according to their respective probabilities (the “p” column). This allows system 100 to power gate any stage / row following a match. In some cases, min [i]=V max We can assume that they are ordered as [i-1]+1 for i>0, so we need to store only one of the two per row. Similarly, we need to store only the high count per row. [Table 2] TIFF2025504420000004.tif130165

[0061] In the above approach, an incoming value can be compared to entries / rows in a hardware table with explicit lower and upper range values ​​to find a table entry / row whose range (defined by lower and upper) contains that value. With this arrangement, there are typically no constraints on how the rows can be ordered, so long as each row is unique and does not overlap with other rows. In this way, ranges of rows can be stored in the table in any order. In other cases, where entries are stored in ascending order and the lower mark of a row is the upper mark of the previous row +1, the system typically does not need to store either the lower or upper value, as they are implied. However, in such cases, there are typically cases where there are no gaps in the range.

[0062] The decoder 104 accepts two sequences as input, one containing compressed symbols and the other containing the corresponding offsets. At output, the decoder 104 generates the corresponding original value. To do so, it uses arithmetic decoding on the symbol sequence to obtain the prefix value corresponding to the next symbol. The decoder 104 then uses the symbol table to extract the appropriate number of offset bits and adds it to the prefix value. The process continues until all symbols are decoded. The decoder 104 generates a single value for each step. Using pipelining and replication, the system 100 can achieve the desired bandwidth goal. In a particular case, the upper and lower values ​​and bit sets received from the symbol stream are used to determine which symbol is represented by each bit set from the symbol stream, using the lower and upper fields of the probability count table. The symbols are represented by V min and OL (the number of offset bits). Then, we use V min You can get the original value back by adding (or ORing) it to

[0063] 6 shows the interface to an exemplary implementation of the encoder 102. The following ports are available for initialization: HIin and LOin are used to initialize the decoder ranges which are stored in two internal registers of 16 bits each. The enable signal indicates that a value should be loaded into the corresponding register. Figure 6 shows the ports with 16+1, meaning that these ports contain 16-bit data values ​​and 1-bit enable signal. Alternatively, an initialization signal can be given which forces the registers to 0xFFFF and 0x0000 respectively. SYMT is used to initialize the value range and offset fields of the symbol table. In the implementation, we assume that the entire range is mapped, that the rows are ordered by value, and that each range begins immediately after the previous range ends. Therefore, this write port is written to R max It contains 8b+3b=11b for, OL for one row, a 4-bit index to the target row, and 1-bit enable. PCTN is used to initialize probability count entries. They are ordered to match the symbol table. The port uses 10 bits for the probability count (one per row), a 4 bit row index, and a 1 bit enable. ·The 8-bit IN port, together with its 1-bit enable, is used to transmit the input value that needs to be decoded in this step (enabled, if present). A 1-bit done signal indicates that there are no more values ​​to encode. In some cases, SYMin and PCNTin may be combined into a single port that shares the enable and index portions.

[0064] In the above exemplary implementation, in all steps, the encoder 102 receives the value to encode via the IN port. The port carries an 8-bit value and a 1-bit enable. This can be used to support the case where there is currently no value to encode. For example, the value may be from a processing element that has not yet produced it. In the above exemplary implementation, the encoder 102 processes the symbols internally and produces the following outputs: Contains an 8-bit value containing OFSout and the corresponding offset. The offset is displayed in the least significant bit. The offset length is given by the 3-bit OFSr. An additional 1-bit signal in OFSr indicates whether an offset is generated in this step. · The 16-bit CODEout contains the number of bits to be written into the coded symbol stream. The bits are taken from the most significant bit of this output. The number of bits taken is given by the 4-bit CODEc, which also contains an additional 1-bit enable signal indicating if all the bits are there. The 5-bit OUTu, together with its 1-bit enable, indicates whether the output should contain (enable) additional bits, and how many of them (the count is the 5-bit part). These bits are inserted between the most significant bits of CODEout to get the inverse value of that bit. For example, if CODEOUT is 0xF0000, using 4 bits, and the enable of CODEu is zero, then 0xF should be written into the coded symbol stream. However, if CODEuenable is 1, and CODEu5 bits indicate that 2 bits should be used, then the coded stream should be written with 0x27 (inserting two 0s after the most significant bit 1 of CODEout, outputting a total of 6 bits).

[0065] FIG. 7 shows an example implementation of the internal mechanisms of the encoder 102. In this example, there are three blocks: "SYMBOL Lookup", "PCNT Table", and "High / Low / Code Generation". Two 16-bit registers, HI and LO, maintain the current range. They are initialized to 0xFFFF and 0x0000, respectively. Encoding the input value IN starts with the "SYMBOL Lookup" block, which identifies the symbol to which this value is mapped. This block produces two outputs. One is an output to the offset stream, which includes the ports OUT_out and OFS_r mentioned above. The second output is SYMi, which is a 1-HOT encoded 16-bit vector indicating the table row (symbol index), and the value mapped to SYMi is input to the "PCNT Table" block, which identifies which range of probability counts this symbol is mapped to. The output from the "PCNT Table" block is then used by the "High / Low / Code Generation" block to update the current code range and generate any encoded stream output bits. The 5-bit UBC register counts the number of underflow bits. Its value is input to the block and updated by the unit using the UBCn signal.

[0066] Figure 8 shows the structure of the "SYMBOL lookup" block. The 8-bit IN value is compared to the maximum value of all ranges (base[i]=R max -1). The rows are ordered so that the smaller basis value appears first, so the matching row is the last row in the order whose basis is greater than the input value. This is used to generate a SYMi vector. The SYMi vector is then used to route the corresponding basis and offset length fields (ob in the diagram) to a component that generates the output offset. The offset is calculated as the difference between the 8-bit IN value and the matching basis value, and trimmed to ob bits via a "mask" block.

[0067] Figure 9 shows an example of a PCNT table block. The SYMi signal is used to select the range boundary counts cHI and cLO that correspond to the current symbol. For each symbol index, the unit holds a 10-bit probability count that represents the maximum value of the range (inclusive). cHI is the value of this register, and cLO is the value of the register of the previous row, or 0 if this is row 0. The values ​​of cHI and cLO must be scaled according to the current range. The current range is given by range=(HI-LO+1), where HI and LO are 16-bit registers that hold the current range. The scaling is performed by multiplying the range by cHI and cLO. As range is a 16-bit number and cHI and cLO are 10-bit numbers, the multiplication derives a 26-bit number. Since the maximum probability count is 210-1, the 10 least significant bits can be discarded, as this amounts to dividing by the maximum probability count (a necessary step in arithmetic coding, in general, since it converts counts to probabilities). The multiplier can therefore omit any part that serves to generate only one of those 10 least significant bits. The resulting scaled 16-bit range bounds sHI and sLO are then input to the "High / Low / Code Generation" block.

[0068] The Upper / Lower / Code Generation block, an example of which is shown in FIG. 10, begins by calculating new scaled ranges tHI and tLO given new values. The sHI and sLO signals are used to calculate the values ​​of tHI and tLO by adding to the current LO respectively, where the sHI and sLO values ​​correspond to the appropriately scaled range, but with an offset starting from 0. This offset is adjusted to LO to place this range in the appropriate position, taking into account all symbols seen so far. Two 16-bit adders perform the necessary additions. For the upper bound, a 1 is added since it is inclusive at this point. The rest of the unit performs two functions in sequence: (1) determining whether tHI and tLO have a common prefix that can be output, and (2) detecting whether tHI and tLO are in danger of underflow. For common prefix detection, the XOR block can generate a bit vector that identifies any bit positions that differ between tHI and tLO. The LD1 block then identifies the length of any of the common prefixes, if any, to detect in leading positions whether there is a difference. Those bits can be written into the encoded symbol stream and, since they can no longer change, can be shifted out, which updates tHI' and tLO'.

[0069] For underflow detection and handling, the encoder 102 uses finite precision arithmetic with arbitrary precision arithmetic. This is possible by effectively maintaining a 16-bit window within the bounds of an arbitrary bit length. The HI and LO registers contain these 16-bit windows. In practice, the bounds can be the HI value plus an "infinite" number of 1's suffixed, and the LO value plus an infinite number of 0's suffixed. The 16-bit window can slide to less significant bits by shifting out prefix bits that are sure not to change anymore. To understand why this works, observe that the bounds of HI and LO shrink as the arithmetic coding encodes one symbol after another. HI always gets smaller, while LO always gets larger. However, since each symbol encoded is assigned a non-zero probability, there should always be HI>LO. HI and LO grow longer and longer common prefixes (most significant bits that do not change anymore). These are the bits that can be safely "discarded" by shifting them out of the HI and LO registers while the encoder writes to the coding stream.

[0070] However, depending on the probability range of the new symbol and the current range, having a 16-bit window may not be enough to scale the range appropriately so that HI remains greater than LO. In this case, HI contains values ​​of the form 100 and low contains values ​​of the form 011. This may require being able to perform more than 16-bit operations to accommodate the case where HI and LO are approaching approximately 0.5 but can end up both above and below 0.5. This occurs when the range adjustment made by the input symbol is much smaller such that it does not affect any of the 16 bits of the current window (it must affect the lower bits further), and can be referred to as an underflow bit. The encoder 102 handles these bits by entering a state that records the number of underflow bits required to allow the window to slide. Finally, once another prefix is ​​found, the encoder 102 can determine if the range is above or below 0.5 and insert them into the output stream.

[0071] Thus, encoder 102 handles the underflow bit by identifying any prefixes of tHI' and tLO' and starting from the second most significant bit, where tLO is all 0's and tHI is all 1's. This subprefix is ​​shifted away from tHI' and tLO'. The result is an updated version of tHI" and tLO". To find this subprefix length, encoder 102 uses a leading ones detector for tHI' (ignoring the most significant bit) and a leading zeros detector for tLO' (again ignoring the most significant bit). The subprefix is ​​the most significant position of the two. This is implemented in the 01PREFIX block. Internally, this block uses a two-input AND gate for each bit position, with one input connected directly to tLO' and the other connected to tHI' after inversion (NOT). Those 15 AND outputs drive the leading zeros detector. The leading zeros are where the subprefix, if any, ends. This subprefix is ​​stripped from tLO' and tHI' to produce tLO" and tHI". This subprefix length is added to a UBC register which counts the number of underflow bits detected so far.

[0072] In generating the final HI and LO, after the common prefix and underflow sub-prefix are discarded, the encoder 102 adjusts the final HI and LO values. The encoder 102 inserts a suffix 1 into HI to compensate for the fact that it shifted out some most significant bits. Note that HI is meant to slide over the number as a suffix of infinite 1s. Additionally, the encoder 102 sets the most significant bit of HI to 1 if it enters underflow bit mode. The final output is the value of nHI and nLO, which are loaded into the HI and LO registers, respectively.

[0073] Other ways to avoid underflow in the HI and LO registers are possible: for example, the range may be expanded whenever it falls below half the maximum range.

[0074] 11 shows the interface to an exemplary implementation of the decoder 104. In this example, the decoder 104 has the same initialization ports as the encoder 102 (instead of a "SYMBOL lookup" unit, the decoder has a "SYMBOL generation" unit, but internally these two units have the same symbol table). During decoding, it accepts two input streams, CODEIN and OFSin, to read bits from the encoding symbol and offset streams, respectively. The ports operate as follows: CODEIN is a 16-bit input that the decoder requests to receive a stream of encoded symbols. At each step / cycle, the decoder reads the number of bits and signals which unit provides those numbers via the 5-bit CODEr output signal. This specifies a 4-bit count plus a 1-bit enable signal, indicating whether any bits are requested or not. The encoder 102 requests that bits be read starting from the most significant bit of CODEin. OFSin is similar to an 8-bit input that the decoder 104 uses to read the offset bits. The number of bits used depends on the output OFSr, which is 3 bits plus 1 bit enable. The offset bits are read from the most significant bit position of OFSin.

[0075] FIG. 12 shows an exemplary internal mechanism of the decoder 104. In this example, there are four blocks: "SYMBOL Generation", "PCNT Table", "High / Low / Code Adjustment", and "OFS Adjustment". Two 16-bit registers HI and LO maintain the current range. They are initialized to 0xFFFF and 0x0000, respectively. The 16-bit CODE register is used to read in the encoded symbol stream, and the 8-bit OFS register is used to read in the offset stream. In this example, the decoding of a symbol occurs by processing the current range boundary values ​​HI and LO in the "PCNT Table". As a result, it identifies the symbol row index of SYMi and two adjusted range boundary values ​​adjHI and adjLO. SYMi is used by the "SYMBOL Generation" block to generate the original value in a process that consumes the corresponding offset from the offset stream. The decoded value is displayed on the OUT port. The "High / Low / Code" block generates new range boundary values ​​nHI and nLO, adjusts the CODE value, and potentially consumes bits from the encoded symbol stream.

[0076] For the PCNT table unit, the objective is to determine which symbol range the current CODE value falls within. The unit first adjusts the CODE value to be an offset starting from 0. This can be done via a subtractor that calculates CODEadj=(CODE-LO+1), thus removing the current lower boundary from CODE. Before the unit can determine which range this value falls within, it needs to scale the probability according to the current range width. The subtractor is therefore tasked with calculating the length of the current range as range=(HI-LO+1). The probability counts that define the upper boundary of each row are multiplied by the range. This is a 10-bit by 16-bit multiplication, and from here only the upper 16 bits are needed, since the values ​​are normalized to the total probability count of 210-1. The index of the first row where CODEadj is not less than or equal to the adjusted count corresponds to the symbol that needs to be decoded. The PCNT table unit produces two sets of outputs: (1) the first contains the adjusted range boundaries adjHI and adjLO, and (2) the second is the 1-HOT coding vector SYMi that identifies the row index. adjHI and adjLO are input to the "Adjust HI / LO / CODE" block, and the SYMi output is input to the "Generate SYMBOL" block.

[0077] The SYMBOL generator generates the decoded value. It uses SYMi to access the symbol table and min In addition to the offset, we get an ob field which identifies the number of bits the offset is. We use the ob field to extract those bits from the OFS register. These bits are shifted out and additional bits are shifted in from the external offset stream using OFS on the read port. The final value is v min plus an offset of zero extended to 8 bits.

[0078] The High / Low / Code Adjustment block is tasked with adjusting the range boundary registers HI and LO and consuming bits from the encoded symbol stream. The process starts by first adjusting the range boundaries of adjHI and adjLO to offset them based on the current LO boundary. Two 16-bit adders perform this adjustment, producing tHI=(adjHI+LO+1) and tLO=(adjLO+LO). The XOR and LD1 blocks identify any common prefix bits between tHI and tLO, which are discarded (shifted out) to produce tHI' and tLO'. This prefix length is given by the cpl signal. The 01PREFIX block looks for the longest prefix (ignoring the most significant bit) that contains only 0 and 1 bits in tHI' and tLO', respectively. If such a prefix exists, there is a risk that the range is too small and needs to be adjusted. This sub-prefix length is given by p01l. The LD1 and LD0 blocks can be implemented by combining them via an AND gate (after inversion / NOT, one input is connected to tLO and the other input is connected to tHI) for each bit position followed by a leading zero detector. These bits are discarded to generate tHI” and tLO”. The mask block inserts l1l+p01l bits starting from the LSb into tHI” (note that HI is assumed to be a value with a sufficient number of “infinite” number 1s). Since the cpl+p01l bits were shifted out of HI and LO, we need to shift this much out of CODE and we need to shift this much bits into CODE from the external encoded symbol stream. This is performed using a shift block that communicates with the external source through the CODEr and CODEin ports. The hMSb block sets the most significant bit of the final nHI if the adjustment range is too small. In this case, the final CODE, CODEn, needs to be inverted, which is performed by the cMSb block.

[0079] In some cases, processing neural network places can present significant bandwidth demands on memory. While some embodiments described herein can be used to process a single value per cycle, it may be necessary to process multiple values ​​per unit time to meet the bandwidth demands of heavy workloads. Below, we provide two exemplary approaches that can be used to process multiple values: pipelining and replication.

[0080] In some cases, encoding and decoding can be pipelined to increase the operating frequency and reduce the number of units required to achieve a desired throughput, thus improving area cost and energy efficiency. In block 204, the pre-processing module 108 can split the input tensor into several sub-tensors, each of which can be encoded and decoded as an independent stream. Some state elements of the encoder and decoder units can be replicated to support pipeline processing of multiple such streams. In some cases, all streams can use the same probability count table, but in general, it is not necessary to replicate the PCNT (HiCnt[i]) and symbol generation and lookup tables (base[i] and ob[i] fields). However, separate sets of OFS registers, code registers, upper registers, and lower registers can be used for each stream. Splitting the encoder 102 and decoder 104 into pipeline stages can be accomplished in multiple ways. First, the PCNT lookup can be split into several stages, with each stage considering only the range of the table entries. For example, the table lookup can be split into two stages, with the first stage considering the first half of the entry and the second stage considering the second half of the entry. If a match is obtained in the first stage, the second stage can be optionally power gated to reduce energy. In some cases, the partitioning may require temporary registers to hold the results produced in each stage. Adjusting the upper / lower / code can be another stage of the decoder, and generating the upper / lower / code can be another stage of the encoder. Similarly, offset generation and offset extraction can be other stages.

[0081] Pipelining increases throughput because it reduces the amount of processing that occurs per cycle, thus enabling higher operating frequencies. Throughput can be further increased by replicating the encoder 102 and decoder 104. This replication can include splitting the input tensor into several sub-tensors whose streams are encoded and decoded independently. Such a splitting can be desirable for several reasons, for example it is often used to exploit locality in the on-chip memory hierarchy. The number of units required depends on the target application and the number of processing cores.

[0082] In some cases, the offset stream may be distorted, with shorter offsets being more common. In such cases, the offsets may be distributed unevenly, with different bit patterns appearing with different probabilities. Therefore, a second-level encoder may be used for these purposes. In some cases, the first-level encoder may be able to capture some of the unevenness with a sufficient number of probability count table entries, but with reduced efficiency.

[0083] In some cases, the offset stream can also be encoded by the encoder 102. In a particular case, the encoding obtained with the offset stream can comprise three streams: (a) a symbol stream generated for the value, (b) a symbol stream generated for the offset, and (c) an offset stream that is added to each symbol generated by (b) to recover the original offset. In this case, a two-level encoder can be used, where the output of the first-level offset generator becomes the input to the second-level encoder. In this way, the first-level encoder writes only the code stream. The second-level encoder observes the offset stream generated by the first-level encoder. All offsets can be zero-extended to the maximum possible bit length and then encoded using the same approach as the values. The output from the second-level encoder can be two streams: one containing symbols corresponding to the original offset and the other containing the new offset.

[0084] In some cases, the encoder 102 can use other encoding approaches to further improve effectiveness. For example, dictionary-based methods such as GOBO encode most of the values ​​as indices into a dictionary of representative values. Such indices are typically 3-bit to 5-bit integers. The distribution of these indices is skewed, and rather than storing the indices as is, the above approach can be used to reduce the footprint of the index array. Example experiments performed by the inventors with a modified version of GOBO resulted in a reduction in the index footprint to 79% of the original size.

[0085] To generate the contents of the probability count tables, such as those shown in Table 1, one set of probability tables can be generated per layer of the machine learning model, i.e., one for each input tensor, where a tensor is an array of values. For each layer of the machine learning model, two separate tables can be generated: one for the activations and one for the weights. In some cases, profiling can be used for both the weights and the activations, such that profiling is performed for one or a few input samples to generate the tables. For tensors that are statically known and therefore do not depend on the input, such as weights, the tables generated are identical to the tables that would have been generated if the tensor had been called at run time. For tensors that are input-dependent, the tables generated were not identical to the tables that would have been generated if the tensor had been called at run time, given different inputs. Although individual activation values ​​may change with the input, the overall distribution of the activation tensors generally does not change significantly. Therefore, if a profiling approach is used, it is sufficient to apply it for a small subset of the potential inputs, such as more than 10 inputs in the experimental example.

[0086] In some cases, table generation can initialize the table with values ​​corresponding to a uniform value distribution and then estimate the resulting footprint, which is the number of bits required to store the input tensor. The table entries can then be iteratively adjusted to find a better assignment. The adjustment can consider all possible configurations allowed by the search algorithm, but only if this results in a reduction in the footprint above a predefined threshold. In one example, the predefined threshold can be set to 1%. An example of this table generation approach is provided in Listing 1. Without loss of generality, this example assumes that the inputs are 8-bit values, but inputs of any bit length can be used. In this example, the input tensor is examined and a histogram is created with 28 buckets, where each bucket h(i) represents the number of times the corresponding value i appears in the tensor. The findPT() routine is called, which takes the histogram as input. The probability count table PT is initialized with default settings, where the range of possible values ​​(e.g., [0,28-1]) is evenly divided among the table entries (line 38 in Listing 1). The number of entries in PT can be selected as a design parameter. In general, the more entries, the more accurately the input data distribution can be represented and, depending on the input distribution, the better the expected compression ratio. In the extreme case, for 8-bit input values, a table of 28 entries would be needed to represent the frequency of each value. Maintaining a table with 28 entries for 8-bit values ​​may be prohibitively expensive in terms of hardware cost and access latency in some cases. Therefore, in a practical implementation, a 16-entry table can be used. In such a table, each entry i has a set of possible values ​​[v min , vmax An example of the default settings for a PT with 16 entries is shown in Table 2. [Table 3] TIFF2025504420000006.tif69165 [Table 4]

[0087] In general, the initial configurations above are suitable for uniformly distributed values, which may not be typical for DNN models. The opportunity for compression stems from the fact that in DNNs, the probability distribution of values ​​is usually highly skewed, with most of the values ​​being small and clustered around one point in the range of possible values; only a few scattered around the two extremes. The table generation advantageously determines a suitable set of 16 subranges that best represent the input value distribution. Thus, the table generation is an iterative approach that starts from an initial default setting and proceeds to try alternative configurations. Each alternative configuration is generated by expanding or contracting the subranges. The goal is to minimize the footprint. To this end, the number of bits required to represent the input is estimated, given the current table configuration.

[0088] Before trying other configurations of the probability count table, findPT() is used to estimate the footprint, as shown in line 41 of Listing 1, given the current probability count table PT. Search() is then called to search through the candidate configurations and return the best PT found and its corresponding footprint. Line 43 of Listing 1 determines whether further better configurations are to be searched for by testing the ratio of the newly found best footprint (new size) to the previous best footprint (size). As long as this ratio is less than a threshold (0.99 was used), another iteration is performed and additional PT configurations are considered.

[0089] The process used to generate and consider candidate PT configurations is implemented by search() in Listing 1. The inputs are the histogram values, the current best PT and its footprint (minimum size), plus two additional parameters "depth" and "around". The output is the best PT found and its corresponding footprint. If search() does not find a better PT, it returns the PT and the footprint it was called on. The implementation of the search uses recursion whose depth is controlled by the "depth" parameter. When findPT() calls search(), it can set the depth to 1. When search() calls itself, it increases the depth. As long as the depth is less than the maximum allowed value DEPTHMAX, search() can call itself. In this example model, a maximum depth of 2 was sufficient. The parameter "around" identifies which PT entry index to search to try for adjustment. If around is negative, it can search to adjust all entries. The only case in which this occurs is when findPT() calls search(). Otherwise, around is the index itself, and search() attempts to refine only those entries whose distance from around is at most either below or above a threshold. In this example, the threshold is set to 1.

[0090] The search() process first makes a copy of the input PT and stores it in tryPT (Listing 1, line 11). Then the search() process iterates over table indexes 1 to N (Listing 1, line 12). If entry i can be considered taking into account the value of the around parameter (Listing 1, line 13), the corresponding tryPT entry is adjusted. min is decremented if possible. This effectively increases the range covered by entry i. To check if a decrement is possible, the search() process first checks v of the previous tryPT entry.min Check the value (lines 16-17 in Listing 1). Candidate v min v of the previous entry min Decrement is possible only if candidate v is greater than v (line 18 in Listing 1). min is decremented (Listing 1, line 19). If the search has not reached the maximum depth, it recursively calls search() with around set to i (Listing 1, lines 20-21). Otherwise, search() estimates the footprint of tryPT and, if it turns out to be smaller than the current best one, updates the bestPT and footprint (Listing 1, lines 22-24). Entry i is then restored to its original value (Listing 1, line 25). The process then continues in a similar manner, searching instead for v of entry i. min Consider the table construction where i is incremented (lines 26-35 in Listing 1). This effectively shrinks the range covered by entry i. Finally, search() returns the best PT and footprint it found.

[0091] At each step of that iterative search, table generation can involve estimating the number of bits required to represent the input, given the set of subranges currently being considered (a call to encoding size()). This can be done as follows: Let each value be the probability P(x1), ..., P(x n ) possible values ​​x1, ..., x n Considering X as a discrete random variable with σ, the entropy of X is: [ka] During the ceremony, [ka] According to Shannon's entropy theory, the probability P(x i ) occurs at the value x i The minimum number of bits required for a symbol to represent is defined as follows: [ka] A profile can be used to find a mapping that minimizes the collective footprint in bits of the profiled data. In the specific example of 8-bit values ​​and a 16-entry table, the default initial mapping can use equal-sized subranges of length 28 / 16=16. Thus, as shown in Table 2, the offset length OL for all entries in the initial mapping is 4 bits. For entry i, the Count column contains the range [v min and v max ] is the sum of the counts of values ​​that fall within [ka] A value x within the range of entry i is encoded as (symbol, offset), where offset is of length OL i bit and the symbol S i The bit length of is proportional to its probability according to Shannon's entropy theory. [ka] Thus, for a particular DNN layer, given a particular table configuration, the entire footprint of profiled data can be defined as: [ka]

[0092] v min After the values ​​are determined, they can be used to map probability counts. The range of probability counts [0...2m] is divided proportionally to the frequency of values ​​within the corresponding range, where m can be a design parameter (in one example, m=10).

[0093] As explained, profiling can be used to generate probability count tables used for activations. When using profiling, it is possible that some values ​​will not appear. However, just because certain values ​​did not appear when processing a limited set of inputs does not guarantee that these values ​​will not appear when processing other inputs. The probability count tables can be adjusted if these values ​​are not covered by any of the ranges of the table. When this occurs, table entries with zero range (vmin=vmax) may be generated. Therefore, as a post-processing step for tensors that are not static, the v of such entries can be adjusted. max By increasing (or v min ) and "steal" one probability count from any of the other entries that have a probability count greater than two.

[0094] In example experiments conducted by the inventors, the effectiveness of the system 100 was determined against a set of DNN models spanning a wide range of applications, such as image classification, object detection, segmentation, and natural language processing. The models evaluated are shown in Table 3. All models were quantized to 8 bits. The only exception was ResNet18-PACT, where values ​​were quantized to 4 bits, except for the first layer, which required 8 bits of precision. [Table 5]

[0095] Table 4 shows the overall footprint reduction for several models using this embodiment. Depending on the quantization approach, the compression ratio varies significantly, from as little as 0.83 to 0.30. Regardless of the quantization approach used, the system 100 can provide benefits without the need for custom construction. [Table 6]

[0096] Table 5 provides a detailed view of the compression possible with system 100 for one of the machine learning models (BILSTM). The compression ratio varies significantly from layer to layer and generally tends to be higher for activations. System 100 exploits a highly skewed distribution of activation tensors that tend to be biased towards zero. [Table 7] TIFF2025504420000017.tif119165

[0097] Experimental examples verify that system 100 can achieve substantial improvements in (1) compression ratios for both model weights and run-time activations, and (2) energy savings due to compressed off-chip traffic to and from DRAM. Experimental energy results assume a total of 16 gigabytes of dual-channel DDR4-3200 off-chip memory. In the experimental examples, 32 compression engines and 32 decompression engines were modeled.

[0098] Example experimental results showed that system 100 required a total area of ​​1.14 mm2 and consumed a total average power of 179.2 mW, which constitutes a 4.7% overhead over the power consumed by a dual channel DDR4-3200 memory system saturated at 90% of the peak bandwidth.

[0099] For the weights and run-time activations of a DNN model, Figs. 13 and 14 respectively show the energy consumed by off-chip memory transfers normalized against a baseline that does not implement compression techniques. The illustrated estimates assume that each layer's weights and input activations are read off-chip only once. As shown in Figs. 13 and 14, the energy savings vary between models according to the achievable compression ratio. For example, for pruned models such as AlexNet-Eyeriss and GoogLeNet-Eyeriss, when most of the model's weights are zero, the system 100 takes advantage of a highly skewed value distribution to save 91% and 72% of the off-chip access energy, respectively. For models with less skewed value distributions, i.e., nearly uniform distributions, such as NCF APack, the system 100 achieves a lower compression ratio (1.2x) and therefore saves 13% of the off-chip energy consumption. However, the same model may have a more skewed distribution for its run-time activations. For example, the compression ratio for NCF activations is 22×, leading to an energy savings of 53%. On average across all models, system 100 reduces the energy consumption of model weights and runtime activations by 61% and 57%, respectively.

[0100] As demonstrated in experimental examples, system 100 is plug-in compatible with many accelerators and provides an easy-to-implement and effective off-chip compression approach for machine learning models. Experimental examples show that it reduces off-chip traffic and significantly reduces the energy consumed by expensive off-chip memory transfers. Furthermore, system 100 can facilitate memory-bound models by reducing stalls associated with off-chip data accesses.

[0101] Advantageously, the present embodiment can use heuristics to partition the value space into prefix groups to maximize compression ratio. In this way, the present embodiment can exploit the parallelism of memory accesses in machine learning workloads to replicate several encoder / decoder units to operate in parallel. Combined with the ability to pipeline these units so that they can be time-multiplexed across several data streams, the present embodiment can accommodate the data bandwidth demands of the target workload. In some cases, the system 100 can be placed just before an off-chip memory controller, and the system 100 can function transparently to the rest of the on-chip system. Thus, the remaining on-chip memory and computational units can see the original data stream. As a result, the present embodiment can be used in many machine learning accelerators, such as vector-like or tensor core units, systolic arrays, and units that process sparse tensors. In further cases, the encoding and decoding can be performed by other aspects of the computing system, such as running on a central processing unit or other processor.

[0102] While the foregoing has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the appended claims.

Claims

1. 1. A method for compressing and decompressing a data stream for use by a machine learning network, the data stream including weight data or activation data for use by the machine learning network, comprising: receiving the data stream to be encoded; encoding each value of the data stream, determining a mapping to one of a plurality of non-overlapping ranges, each value being encoded as a symbol representing the range and a corresponding offset; arithmetically encoding the symbols using probability counts; storing a compressed data stream including the arithmetically coded symbols and the corresponding offsets; decoding the compressed data stream with arithmetic decoding using the probability counts, wherein the arithmetic decoded symbols are used with the offset bits to arrive at a decoded data stream; and communicating the decoded data stream for use by the machine learning network.

2. The method of claim 1 , wherein the probability count is represented by a range of values ​​that represents an approximate frequency of occurrence of the symbol in input samples.

3. The method of claim 2 , wherein the probability counts are initialized with a uniform value distribution.

4. The method of claim 2 , wherein the probability counts of the input samples are used to determine the non-overlapping ranges.

5. 2. The method of claim 1 , wherein encoding each value in the data stream comprises encoding multiple values ​​simultaneously using pipelining, and wherein decoding the compressed data stream comprises decoding multiple values ​​simultaneously using pipelining.

6. 6. The method of claim 5, wherein the data stream includes a tensor, the method further comprising splitting the tensor into a number of subtensors, each subtensor being encoded and decoded in a respective pipeline.

7. The method of claim 1 , further comprising encoding the corresponding offsets using two levels of encoding.

8. The method of claim 1 , wherein each of the non-overlapping ranges is defined by a respective minimum value, a maximum value, and an offset length.

9. 9. The method of claim 8, wherein the non-overlapping ranges are ordered and the maximum value of each range is the minimum value minus one of the next ranges in the order.

10. 2. The method of claim 1, wherein the symbol comprises a minimum value of the mapped range and the corresponding offset comprises the minimum value of the mapped range subtracted from the value from the data stream.

11. The method of claim 1 , wherein encoding each value of the data stream comprises using profiling to determine the probability count of activation data.

12. 1. A system for compressing and decompressing a data stream for use by a machine learning network, the data stream including weight data or activation data for use by the machine learning network, comprising: one or more encoders for receiving the data stream to be encoded and encoding each value of the data stream, the encoding comprising: determining a mapping to one of a plurality of non-overlapping ranges, each value being encoded as a symbol representing the range and a corresponding offset; and arithmetically encoding the symbols using probability counts. a memory storage for storing a compressed data stream including the arithmetically coded symbols and the corresponding offsets; and one or more decoders that decode the compressed data stream with arithmetic decoding using the probability counts and communicate the decoded data stream for use by the machine learning network, the decoders using the arithmetic decoded symbols and the offset bits to arrive at a decoded data stream.

13. The system of claim 12 , wherein the probability count is represented by a range of values ​​that represents an approximate frequency of occurrence of the symbol in input samples.

14. The system of claim 13 , wherein the probability counts are initialized with a uniform value distribution.

15. The system of claim 13 , further comprising: determining the non-overlapping ranges using probability counts of the input samples.

16. 13. The system of claim 12, wherein multiple values ​​in the data stream are encoded simultaneously by each of a plurality of the encoders, and multiple values ​​in the compressed data stream are decoded simultaneously by each of a plurality of the decoders.

17. 17. The system of claim 16, wherein the data stream includes a tensor, the method further comprising splitting the tensor into a number of subtensors, each subtensor being encoded by a separate encoder and decoded by a separate decoder.

18. The system of claim 12 , wherein the encoder further encodes the corresponding offset using two levels of encoding.

19. The system of claim 12 , wherein each of the non-overlapping ranges is defined by a respective minimum value, a maximum value, and an offset length.

20. 20. The system of claim 19, wherein the non-overlapping ranges are ordered, the maximum value of each range being the minimum value minus one of the next ranges in the order.

21. 13. The system of claim 12, wherein the symbol comprises a minimum value of the mapped range and the corresponding offset comprises the minimum value of the mapped range subtracted from the value from the data stream.

22. The system of claim 12 , wherein the encoder uses profiling to determine the probability counts of activation data.

Images