Block-based encoding and decoding for random tensor access

Abstract

Methods and systems, including computer-readable media, are described for encoding data values using encoder circuitry for an integrated circuit. A system receives an instruction comprising an opcode that specifies a compression operation and identifies, based on the opcode, a set of uncompressed data blocks corresponding to an uncompressed superblock of outputs generated by a host processor. For the set of uncompressed data blocks: compressing, by the encoder circuitry, each uncompressed data block in parallel using a plurality of encoders at the encoder circuitry. Encoder circuitry of the system generates compressed data blocks in response to compressing each uncompressed data block in parallel and generates a compressed superblock by selectively including one or more of the compressed data blocks.

Description

BLOCK-BASED ENCODING AND DECODING FOR RANDOM TENSOR ACCESSBACKGROUND

[0001] This specification generally relates to data decoding and decompression.

[0002] Data compression can include techniques for reducing the number of bits of input data while still preserving at least some of the same information represented by the input data. Compression can be lossless, where no information is lost during compression, or lossy, where some information is lost during compression. Data decompression can include techniques for reversing data compression — receiving input compressed data and generating the data in its original uncompressed fonn. Entropy encoding refers to a lossless data compression technique where values, or “codewords,” of a fixed length are replaced with shorter, symbols of varying length. The mapping between codewords and the symbols can be determined based on a probability distribution of the rate of occurrence of the codewords. For example, more frequently occurring codewords can be replaced with shorter symbols, and less frequently occurring codewords can be replaced with longer symbols.

[0003] Edge devices that implement machine-learning models (“ML”) often store data associated with those models in a memory' device that is separate and spaced apart from a processor (e.g., an ML hardware accelerator) that implements the models. The memory' device can be dynamic random-access memory (DRAM), such as a Double Data Rate (DDR) synchronous DRAM (SDRAM) device. As ML models grow in size, the bandwidth required for data transfers between the memory' device and the processor also increases. As a result, memory' transfer bandwidth and throughput often become a limiting factor to efficient, low- latency execution of large ML models on ML accelerators at the edge. In some cases, these bandwidth and throughput limitations result in underutilization of available compute units / tiles of the processor or hardware accelerator, leading to further inefficiencies.SUMMARY

[0004] This specification describes an architecture for respective hardware encoding and decoding circuitry and corresponding processes that use the encoding / decoding circuitry to efficiently implement data compression or decompression operations. The encoding / decoding operations are executed in support of high-bandwidth data transfers based on block interleaved coding techniques. The encoding / decoding circuitry' is implemented on a hardware integrated circuit, such as a System-on-Chip (“SoC”) that communicates with an integrated memory device.

[0005] The respective encoder and decoder circuits can be adj acent (or local) to a specialpurpose (or host) processor of the SoC. The encoder is configured to encode / compress data (codewords) prior to transferring that compressed data to the memory device, whereas the decoder is configured to process encoded (or compressed) data received from the integrated memory device. In some examples, the decoder generates a stream of decoded data that is passed to the ML hardware accelerator for processing to execute an ML inference workload.

[0006] The block-based encoder / decoder techniques can be implemented on an SoC to improve memory bandwidth while concurrently enabling encoded / compressed data to be stored and accessed in a manner that efficiently supports random tensor access.

[0007] One aspect of the subj ect matter described in this specification can be embodied in a computer-implemented method performed using encoder circuitry for an integrated circuit. The method includes receiving an instruction comprising an opcode that specifies a compression operation and identifying, based on the opcode, a set of uncompressed data blocks corresponding to an uncompressed superblock of outputs generated by a host processor on the integrated circuit.

[0008] For the set of uncompressed data blocks, the method further includes compressing, by the encoder circuitry, each uncompressed data block in parallel using multiple encoders at the encoder circuitry. The method further includes: i) generating, by the encoder circuitry, multiple compressed data blocks in response to compressing each uncompressed data block in parallel: and ii) generating, using the encoder circuitry, a compressed superblock by selectively including one or more of the compressed data blocks.

[0009] These and other implementations can each optionally include one or more of the following features. For example, in some implementations, the method further includes generating, based on the compressed superblock, a stream of compressed data for routing to a memory device that exchanges data communications with the host processor. In some implementations, generating the compressed superblock includes: i) determining whether a compressed size for the set of uncompressed data blocks exceeds a threshold size; ii) generating, based on a selective encoding technique, the multiple compressed data blocks only when the compressed size is less than the threshold size; and hi) generating the compressed superblock in response to compressing the set of uncompressed data blocks based on the selective encoding technique.

[0010] The method can include generating a metadata tag that indicates a compression state of at least one of the multiple superblocks. The threshold size can be a current uncompressed size of the set of uncompressed data blocks. The method can includedynamically determining the threshold size based on a machine-learning workload assigned to a host processor in data communication with the encoder circuitry. The superblock can represent a minimum granularity for compressing a set of uncompressed data blocks.

[0011] In some implementations, the opcode: i) is a field in the instruction; and ii) represents a compression enable parameter of the instruction. The method can further include determining, based on the opcode, that compression by the encoder circuitry is enabled; and providing a descriptor as a control input to a compressor of the encoder circuitry in response to determining that compression is enabled. In some implementations, i) each data block in the set of uncompressed data blocks includes output activations of a neural network layer; and ii) at least one data block in the set of uncompressed data blocks is obtained from an output activation first-in first-out (FIFO) registers of the encoder circuitry .

[0012] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0013] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0014] The techniques described in this specification provide a block-based encoding (compression) and decoding (decompression) scheme that utilizes a combination of hardw are integrated circuits and related control logic. The control logic can be a software and / or firmware-based approach that operates on individual symbols to generate encoded, interleaved block and super block data structures. This overall approach allows for arbitrary tensor slicing with minimal / low resource overhead, making it an effective data compression / decompression solution that supports high-throughput random tensor access for ML and other applications.

[0015] In more detail, the disclosed techniques include example architectures for high- bandwidth encoding and decoding circuitry that efficiently compresses and decompresses machine-learning (ML) data values to enable high-throughput data transfers between a memory' device and a host processor on the SoC. Relative to data transfers involving uncompressed values, the Huffman coding techniques are leveraged such that encoding anddecoding operations are performed with improved efficiency over prior approaches for on- device data transfer operations.

[0016] For example, individual encoders and decoders of the combined encoding / decoding circuitry can operate in parallel to implement block interleaved operations based on the Huffman coding technique. The compression operations of the encoding circuitry complement data decompression operations performed by corresponding decoder circuitry. These compression and decompression operations are enabled by an encoder / decoder pipeline that reduces the memory space required at the memory device as well as bandwidth requirements for transferring data between the memory- device and a special-purpose processor of the SoC.

[0017] The Huffman compression is used to reduce the overall size of data transfers between the SoC and the memory device. This allows for transferring larger quantities of data, which can improve the compute utilization and overall energy consumption at the hardw are integrated circuit. Further, the disclosed techniques provide a block-based encoder / decoder framework that combines hardware and software approaches to enable randomized tensor accesses and low resource overhead for implementing data transfers between devices of a system. The encoder / decoder framew ork allows for efficient spill and fill operations as well as compression and decompression of parameter / weight values in support of ML workloads.

[0018] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below'. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Fig. 1 is a block diagram of an example system for implementing high-throughput decoding operations.

[0020] Fig. 2 is a block diagram of an example computing device that includes a systemon-chip (“SoC”) and a memory' device.

[0021] Fig. 3A shows an example high-level architecture that includes hardware elements for a compression / encoding operation.

[0022] Fig. 3B show s an example high-level architecture that includes hardw are elements for a decompression / decoding operation.

[0023] Fig. 4 shows an example memory' layout of uncompressed superblocks, compressed superblocks, and metadata.

[0024] Fig. 5 shows an example memory layout of a superblock, compressed superblock, and metadata of a superblock.

[0025] Fig. 6 shows an example superblock partitioning associated with tensor spill operation.

[0026] Fig. 7 shows an example of compressed block padding.

[0027] Fig. 8 shows an example of partitioning a descriptor into multiple superblocks.

[0028] Fig. 9 is a block diagram that indicates an example calculation of block offset bytes and size bytes within a superblock.

[0029] Fig. 10 is a block diagram of an example compressor architecture.

[0030] Fig. 1 1 is a block diagram of an example decompressor architecture.

[0031] Fig. 12 is an example process for block-based encoding that generates compressed data blocks for storing in memory.

[0032] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0033] Fig. 1 is a block diagram of an example system 100 for implementing high- throughput encoding and decoding operations. Decoding throughput refers to the rate at which compressed, e.g., encoded, data can be decompressed, e.g., decoded. For example, decoding throughput can be measured in tenns of bits per second, data blocks / symbols per clock cycle, or both. Generally, a higher decoding throughput means that the time required to decode a predetermined amount of encoded data can be reduced — or, put another way — a greater amount of encoded data can be decoded within a fixed length of time.

[0034] The system 100 includes a decompression engine 110. a compression engine 120, a memory device 130, and a processor 140. Each of the decompression engine 1 10 and the compression engine 120 is coupled to the memory' device 130 and the processor 140. Each of the decompression engine 110 and the compression engine 120 can be implemented according to any combination of software, firmware, and / or hardware.

[0035] The memory device 130 can include multiple memory dies. For example, the memory device 130 can include N memory die, where N is an integer greater than 1. The memory' device 130 can be a dynamic random-access memory' (DRAM) or Double Data Rate (DDR) synchronous DRAM (SDRAM). In some implementations, memory device 130 isconfigured to perform or support various types of Processing-in-Memory (“PiM”) operations, Computing-in-Memory (“CiM”) operations, and memory -near-computing operations ("MnC operations”). The memory device 130 performs or supports these operations using its multiple PiM compute elements, which are described below with reference to Fig. 2.

[0036] To achieve high-throughput decoding, the decompression engine 110 includes multiple decoders 110a- 11 On that can each operate on a respective one of the multiple encoded data streams 135a-n. Each encoded data stream 135 that is received by a corresponding decoder 1 10a-l l On includes multiple encoded data blocks. In some examples, each of the multiple encoded data blocks have the same size, whereas, in other examples, some (or all) of the multiple encoded data blocks are sized differently.

[0037] The decoders 110a- 11 On are configured to operate individually or in parallel with each other to generate a respective one of the multiple decoded data streams 145a-n in response to processing a respective one of the multiple encoded data streams 135a-n. The decoded data streams 145a-n can represent streams of data values and / or operands that are processed at the processor 140 to generate a corresponding output.

[0038] In some implementations, the processor 140 is a ML hardware accelerator configured to implement a neural network and the data values of the decoded data streams 145a-n include inputs to be processed through a neural network layer, a corresponding set of weights for the layer, or both. For example, the inputs and weights represent operands for a ML or neural network computation that is performed at the processor 140 to generate a corresponding ML output.

[0039] In the example of Fig. 1, the results of the computation (e.g., the ML output) are represented as data streams for encoding 155a-n, which are then encoded by compression engine 120 to generate encoded data streams 165a-n. In this example, the results of the computation are encoded / compressed and routed to be stored at memory device 130 using a high-bandwidth / throughput data transfer operation.

[0040] Like the decompression engine 110, the compression engine 120 also includes multiple encoders 120a-120n that can each operate on a respective one of the multiple data streams for encoding 155a-n. The encoders 120a-120n are configured to operate individually or in parallel with each other to generate a respective one of the multiple encoded data streams 165a-n based on encoding / compression operations performed on a respective one of the multiple data streams for encoding 155a-n. Each encoded data stream 165 that is generated by a corresponding encoder 120a-120n includes multiple encoded data blocks. Insome examples, each of the multiple encoded data blocks have the same size, whereas, in other examples, some (or all) of the multiple encoded data blocks are sized differently.

[0041] The system 100 can be implemented by any combination of hardware, software, and / or firmware. For example, the system 100 can be part of a system-on-a-chip (SoC) that includes multiple components and devices, where each device can be implemented as an integrated circuit(s) configured for installation on a printed circuit board (PCB). The processor 140 is referred to alternatively as an SoC processor 140. The system 100 can be included in a computing device and is configured to interact with other hardware components of the device, including memory, processors, network components, and / or peripherals. In some implementations, the memory device 130 and / or processor 140 can be coupled within system 100 through a physical interface that includes a slot(s) or socket(s).

[0042] Fig. 2 shows an example electronic device 200 that includes a system-on-chip 240 ("SoC 240”) and the memory device 130. The SoC processor 140 can be any processor or processing device of the SoC processor 240 included in electronic device 200.

[0043] In the example of Fig. 2, the SoC 240 includes a central processing unit 204 (“CPU 204”), a memory controller 205, a memory 206, a resource manager 208, and an IP / circuit block 210. In other examples, the SoC 240 can include fewer or more components, e.g., can include no CPU or IP / circuit block. In some implementations, the electronic device 200 can include multiple SoCs 240 and / or SoC processors 140 and descriptions about the SoC 240 can apply equally to each of the multiple SoCs and / or processors that may be included at electronic device 200.

[0044] The CPU 204 can be a general-purpose CPU (e.g., a single or multi-core CPU). The CPU 204 generates one or more indicators, such as an app-launch indicator or a function call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory and graphics processing resources to render graphical content of the game. The CPU 204 also generates one or more application values, such as pixel values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0045] The memory 206 is a system memory, shared memory, or both. In the example of Fig. 2, memory 206 is depicted external to circuit block 210. However, memory 206 can include portions of memory that are: i) specific to circuit block 210, ii) external to circuit block 210, or iii) both. The memory 206 can be random access memory of the SoC 240, suchas static random-access memory (SRAM), dynamic random-access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0046] In some implementations, aspects of memory 206 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit block 210. Memory 206 can also include various other ty pes of memory, such as high bandwidth memory (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0047] The system 100 includes a disk memory 260, which can be used to store weights that represent a parameter space for a neural network layer of a neural network model, a set of expert neural models, or both. In some cases, the parameter space for a given expert model is comprised of floating point or quantized parameters that are encoded in a certain format. The disk memory 260 can store the weights corresponding to multiple gigabytes of data. For example, the disk memory 260 can store a complete set of weights corresponding to a multipurpose generative inference neural network model trained to output generative text, audio, and / or image content.

[0048] In some implementations, the disk memory 260 is a non-volatile memory device that retains data through power cycles of the system 100. In some examples, the disk memory' 260 is a solid-state drive (SDD) or a hard disk drive (HDD) associated with one of the edge devices 130. The model parameters stored in disk memory' 260 can be stored in a compressed or uncompressed format. If the parameters are stored in an uncompressed format, then the compression engine 120 can be used to compress the parameters prior to storing the parameters in the memory device 130. In general, the encoder / decoder circuitry 255 can be used to compress or decompress data to be stored at, or routed from, disk memory' 260.

[0049] The resource manager 208 is implemented in hardware and software. Aspects of the resource manager 208 can be also implemented as firmware of an SoC processor 140 or firmware of a device of the SoC 240, such as a ML processor, the memory' controller 205, or the CPU 204. The resource manager 208 includes control logic implemented in hardware, software, or both. For example, the resource manager 208 can include resources such as flipflops, registers, buffers, etc. that are implemented in hardware and control logic (e.g., programmed code) that is implemented in software. As indicated in the example of Fig. 2, in some implementations, the resource manager 208 is a decoding / encoding resource manager that generates control signals for executing decompression and compression operations of system 100.

[0050] The circuit block 210 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 210 can include an image signal processor (ISP) 212, a host processing unit (HPU) 214, a digital signal processor (DSP) 216, and a graphics processing unit (GPU) 218. The circuit block 210 is referred to alternatively as an IP block 210, where the IP block can include one or more proprietary hardware elements. For example, each of the ISP 212, HPU 214, DSP 216, and GPU 218 can be a respective proprietary IP block (or IP device) of a particular entity or device manufacturer.

[0051] The HPU 214 can be a special-purpose processor, such as ML hardware accelerator, neural processor unit, neural network (or ML) processor, or another applicationspecific processor. In some implementations, the HPU 214 is a neural network tensor processor (e.g., a tensor processing unit (TPU)) with an integrated circuit architecture that is optimized for power-efficient execution of ML computations using tensor constructs such as multi-dimensional matrices / tensors.

[0052] One or more aspects of the resource manager 208 can be implemented as a software routine (or module) of the CPU 204, which uses one or more hardware resources of the CPU 204, such as registers, buffers, etc. The CPU 204 can be configured as an instruction and vector data processing engine that processes data obtained from memory device 130 and / or from a system memory of the SoC 202, such as memory 206. In some implementations, each processor, e.g.. ISP 212, DSP 216. HPU 214, GPU 218. of the SoC 240 includes multiple cores.

[0053] The CPU 204 and / or the resource manager 208 can generate control signals to control and / or manage the operations of one or more other components of the electronic device 200, such as decompression engine 110 and compression engine 120. The control signals are routed at the electronic device 200 using an example bus 222 (“SoC bus 222”) of the SoC 240. The control signals can include commands, requests, data, instructions, or combination of these.

[0054] In some implementations, the resource manager 208 cooperates with the CPU 204 and memory controller 205 to dynamically control and manage one or more PiM operations. CiM operations, or both. The operations involve accessing and processing data stored locally at the memory device 130. The PiM and CiM operations are executed locally at the memory device 130 based on control signaling generated at the SoC 240. The control signals are received at the memory’ device 130 and processed locally to trigger or initiate PiM / CiM operations using PiM / CiM compute elements of the memory device 130. The operations canbe executed in support of heterogeneous computations that involve two or more processing units of the IP block 210, the CPU 204, or both.

[0055] The memory device 130 can include an example decompression engine 110 that is used to decode or decompress encoded data values that are received at the memorv device 130 via an example encoded data stream 250. The encoded data stream 250 can include and / or correspond to some (or all) of the encoded data streams 135a-n.

[0056] The SoC 240 cooperates with the memory device 130 to perform computations using data (e.g., encoded data) obtained from memory banks of the memory device 130. The computations can be for operations or workloads that involve one or more of the processors at IP block 210. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 210. multiple IP blocks 210, or both. In some implementations, the heterogeneous operation includes a host device, e.g., HPU 214, and the memory device 130 cooperating to execute an example ML workload. In at least one example, the memory' device 130 may be external to the SoC 240, whereas in another example the memory device 130 may be internal to the SoC 240.

[0057] In the example of Fig. 2, the SoC 240 is an integrated circuit of an example user / client device 230, consumer electronic device, or mobile device, where each of these devices can include items such as a smartphone 230a, tablet 230b, laptop 230c, smartwatch or wearable device 230d. The device 230 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some implementations, the SoC 240 and memory device 130 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0058] In the example of Fig. 1, the decompression engine 110 and compression engine 120 is shown as being collocated with processing resources of the IP block 210. However, in other examples, the decompression engine 110 and compression engine 120 can be placed or positioned at other locations of the SoC 240. The decompression engine 110 and the compression engine 120 can be implemented as a set of integrated circuits, or discrete hardware modules, connected to an SoC processor 140 and memory' device 130. For example, the decompression engine 110 and compression engine 120 can be coupled to the memory device 130 through one or more memory channels, communication buses, or circuit interconnects in electronic device 200 or client device 230.

[0059] The system 100 can communicate data to and from the memory device 130 and the SoC processor 140, through the decompression engine 110 and the compression engine 120. The memory device 130 can store data that includes compressed / encoded data values ofencoded data streams 165a-n. The compressed data can be data that has been processed and compressed by the compression engine 120 or another component of the system 100. such as a pre-processing engine.

[0060] Multiple encoded data streams 135a-n can be sent from the memory device 130 to the SoC 240 for decoding at the decompression engine 110, and subsequent processing by the SoC processor 140. As indicated above, the processor 140 can be any processor or processing device of the IP block 210 or the SoC 240. The decompression engine 110 receives and processes the multiple encoded data streams 135a-n to generate, using techniques that will be described further below, uncompressed data in the form of multiple decoded data streams 145a-n. A host processor 140 of the SoC 240 receives the multiple decoded data streams 145a-n and performs one or more operations on the multiple decoded data streams 145a-n.

[0061] For example, the SoC processor 140 can perform logical or arithmetic operations on the multiple decoded data streams 145a-n or data derived (generated) from the multiple decoded data streams 145a-n, and generate output data. Optionally, the output data, which is similarly in the form of multiple data streams 155a-n, can be sent back to the compression engine 120. The compression engine 120 can compress the multiple data streams 155a-n to generate multiple encoded data streams 165a-n. The memory device 130 can receive and store the multiple encoded data streams 165a-n or data derived (generated) from the multiple encoded data streams 165a-n, which in turn can be sent back for subsequent processing by way of the decompression engine 110.

[0062] The data sent by the memory device 130 can be data for processing any of a variety of different workloads. A workload refers to a computerized task performed by system 100 on a portion of data. In some implementations, the workload that the system 100 is configured to perform can be to train or execute a neural network. Thus, for example, the data sent by the memory device 130 can include data defining values associated with the neural network, such as model parameters (e.g., weights and, optionally, bias values for each of multiple layers) of the neural network. As another example, the data can include inputs to the neural network, which will be processed in accordance with the model parameters to generate neural network outputs.

[0063] The memory device 130 can store encoded data blocks / symbols that correspond to model parameters that have been compressed (or encoded) by the compression engine 120. In these implementations, the data values that are compressed represent model parameters for a trained neural network, which generally remain unchanged when the trained neural network processes inputs to compute an inference.

[0064] During execution of the neural network, the memory' device 130 can pass the compressed model parameters that are represented in the form of multiple encoded data streams 135a-n to the decompression engine 1 10, and the decompression engine 110 can generate uncompressed model parameters that are represented in the form of multiple decoded data streams 145a-n to provide to the SoC processor 140, which then uses as input for performing some operations related to the neural network.

[0065] Each encoded data stream, in turn, includes multiple encoded data blocks. Each of the multiple encoded data blocks has the same size. In some implementations, each of the multiple encoded data blocks can be thirty -two bytes (32 B) in size. Likewise, each decoded data stream, in turn, includes multiple decoded data blocks. Each of the multiple decoded data blocks has the same size. In some implementations, each of the multiple decoded data blocks can be thirty-two bytes (32 B) in size.

[0066] The data can be stored and sent by the memory device 130 according to a variety^ of different formats. In examples in which the workload for the system 100 is to execute or train a neural network, data can be stored and sent as tensors. A tensor is a multi-dimensional array. For example, a zero-dimensional tensor is a scalar value, a one-dimensional tensor is a vector, and a two-dimensional tensor is a matrix.

[0067] Elements of a tensor can be, for example, values for different model parameters for a given layer of a neural network, or input or output between layers of a neural network or of the neural network. Elements for a tensor, or the data in general, can be in one or more of a variety of different formats, including floating point formats, such as 8, 16, 32, or 64-bit floats, and fixed-precision formats, such as 4, 8, 16, 32, and 64-bit integers. While tensors are provided as one example of a format in which data is transferred, in general any of a variety7of different data structures, fonnats, or representations of data, can be used.

[0068] In these implementations, the SoC processor 140 can perform one or more operations related to executing the neural network on the uncompressed model parameters that are represented in the form of multiple decoded data streams 145a-n, for example matrix multiplication or computing the output to an activation function, and generate, as the output of the one or more operations performed by the neural network, network outputs or intermediate outputs that are represented in the form of multiple data streams 155a-n. An intermediate output of a neural network refers to an output generated by one or more intermediate layers of the neural network, i.e., layers following the input layer but preceding the output layer.

[0069] Optionally, the multiple data streams 155a-n can be sent back to the compression engine 120, which is configured to compress the multiple data streams 155a-n to generate multiple encoded data streams 165a-n. The memory device 130 can receive and store the multiple encoded data streams 165a-n, which in turn can be sent back for subsequent processing, e.g., for backpropagation computation (in cases where the workload is to train a neural network) or for processing by a subsequent layer of the neural network (in cases where the workload is to execute a neural network), by way of the decompression engine 110.

[0070] The decompression engine 1 10 and compression engine 120 can implement any of a variety of different techniques for compression and decompression, respectively. For example, the compression engine 120 can be configured to perform an entropy encoding to compress incoming data from the SoC processor 140. Examples of entropy encoding that can be used include Huffman encoding, arithmetic encoding, and asymmetric numeral systems (ANS) encoding, although any of a variety' of different entropy encoding techniques can be applied. The decompression engine 110 can be configured to perform an entropy decoding (e.g., Huffman decoding, arithmetic decoding, and ANS decoding) that corresponds to the entropy encoding used in the compression engine 120 to decompress incoming data from the memory device 130.

[0071] The compression engine 120 can take advantage of the distribution of repeated values in the data executed as part of a workload. In implementations where the workload is to train or execute a neural network, the system 100 can be configured to generate entropy encodings based on the probability’ distribution of values of the model parameters of the neural network, and can further generate code tables for compressing neural network model parameters on a per-layer basis.

[0072] Fig. 1 shows that the compression engine 120 implements multiple encoders 120a- n for parallel execution of an entropy encoding for input data from the memory device 130. Likewise, the decompression engine 1 10 implements multiple decoders HOa-n for parallel execution of an entropy decoding for input data from the SoC processor 140.

[0073] More specifically, each of the multiple encoders 120a-n can be configured to execute, in parallel with others of the multiple encoders 120a-n. an entropy encoding of a corresponding one of the multiple data streams 155a-n to generate a corresponding one of the multiple encoded data streams 165a-n. For example, a first encoder 120a included in the compression engine 120 can execute an entropy encoding of a first data stream 155a received from the SoC processor 140 to generate a first encoded data stream 165a.

[0074] Each of the multiple decoders 1 lOa-n can be configured to execute, in parallel with others of the multiple decoders 1 lOa-n. an entropy decoding of a corresponding one of the multiple encoded data streams 135a-n to generate a corresponding one of the multiple decoded data streams 145a-n. For example, a first decoder 110a included in the decompression engine 110 can execute an entropy decoding of a first encoded data stream 135a received from the memory device 130 to generate a first decoded data stream 145a.

[0075] Encoding and decoding data in parallel can improve the overall operational efficiency of the system 100, e.g., the encoding (or decoding) power consumption, encoding (or decoding) throughput, or both of the system 100, at least because a greater amount of data can be encoded within a fixed length of time (or, analogously, a greater amount of encoded data can be decoded within a fixed length of time). Encoding and decoding data in parallel allows more data to be transferred between the memory device 130 or the SoC processor 140. As an example, if the compression engine 120 compresses data with a compression ratio of 1.5, then a transfer bandwidth saturated at 1 gigabyte / second could transfer up to 1.5 gigabytes per second of data, by first compressing the data. Storing data in a compressed format in the memory device 130 can also improve the total storage capacity of the memory device 130.

[0076] For many workloads, such as training or executing a neural network, however, the throughput bottleneck is not at the memory' device 130 or the SoC processor 140. but in the mismatch between the (relative lower) decoding bandwidth and the (relative higher) transfer bandwidth between the memory device 130 or the SoC processor 140. In the case of Huffman encoding, for example, due to the prefix code nature, Huffman decoding is ty pically a low- bandwidth operation (e.g., one byte per clock cycle). This can negatively impact the performance of system 100 when Huffman coding is used for some workloads which rely on high-throughput transfers (e.g., thirty-two bytes per clock cycle) to achieve the target performance.

[0077] Whilst this mismatch could be alleviated by having multiple decoders, e.g., 32 decoders in the example above, to execute in parallel, there remains yet another challenge in this parallel decoding scheme: although each decoder could potentially output data at a constant rate (e.g., one decoded data block in the size of 32 B per clock cycle), the amount of encoded data required to output the block depends on the compression ratio of the encoded data stream. This can further cause throughput issues, since in order to ensure that the system 100 meets the throughput requirements, each decoder will ideally have the encoded data that it needs to generate a given decoded data block.

[0078] For example, the decompression engine 110 fetches encoded data blocks from the encoded data streams in a round robin ordering. In some instances, an encoded data stream 135a can have a higher compression ratio, while another encoded data stream 135b has a lower compression ratio. If this occurs, then system 100 may experience a performance impact based on differences in the amount of time (or latency) required to fetch data compressed via a high compression ratio relative to data compressed via a low compression ratio.

[0079] For example, decoder 110a can operate on encoded data stream 135a and require an additional encoded data block, however, due to the round robin ordering, the decompression engine 110 will proceed to fetch an encoded data block from encoded data stream 135b for decoder 110b at a given clock cycle. If decoder 110a requires additional time or latency to fetch the high compression ratio data, then decoder 110a may not have a sufficient quantity of encoded data blocks for entropy decoding. This can cause delays that impact (or degrade) performance of block interleaved decoding operations at system 100. For example, some delays can cause up to 31 clock cycles to elapse before the additional encoded data block is fetched from encoded data stream 135a for decoder 110a. As indicated above, this can limit or degrade the decoding bandwidth of system 100.

[0080] Aspects of the disclosed techniques can be used to address these performance challenges. More specifically, the techniques provide an innovative decoding architecture that enables the space and bandwidth savings of compression mechanisms like Huffman encoding. For example, the techniques provide a high-throughput encoding & decoding architecture that can implement respective application-specific compression & decompression engines to achieve certain data throughput requirements.

[0081] Additionally, the encoding & decoding architecture allows for more efficient execution of large ML workloads, relative to existing approaches that do not compress and decompress the data for these workloads. The ML workloads can be for generative Artificial Intelligence (“GenAI”) applications, such as large language models (“LLMs”) or generative computer vision models. Thus, the techniques address the aforementioned challenges, as discussed further below.

[0082] Fig. 3A shows an example high-level architecture 300A that includes hardware elements of system 100 that are used for compression / encoding operations, whereas Fig. 3B shows an example high-level architecture 300B that includes hardware elements of system 100 that are used for decompression / decoding operations. Architecture 300A is referred toalternatively as compression hardware architecture 300 A, whereas architecture 300B is referred to alternatively as decompression hardware architecture 300B.

[0083] The compression hardware architecture 300A includes a compressor hardware 320 that cooperates with a flexible Direct Memory Access (DMA) unit 310 to execute compression operations, whereas the decompression hardware architecture 300B includes a decompressor hardware 340 that cooperates with the flexible DMA unit 310 to execute decompression operations. The operations and / or functions of the compressor 320 and decompressor 340 can be implemented in hardware as well as software and / or firmware.

[0084] When operating in compression mode, the compressor 320 included in the compression hardware architecture 300A receives a control input that includes the DMA descriptors generated by the flexible DMA unit 310. The compressor 320 then performs compression by reading the uncompressed data stored in a buffer 350 of a host interface block (HIB) 330, compressing or encoding the uncompressed data using compressor 320, generating compressed data in response to compressing or encoding the uncompressed data, and sending the compressed data to the HIB 330, which routes the compressed data for storage in memory device 130.

[0085] The buffer 350 can be a first-in-first-out (FIFO) buffer, a circular or ring buffer, or the like, in various implementations. In some implementations, the FIFO buffer 350 is an output activation FIFO buffer (or register) configured to store output activations that are generated as outputs of a layer, e.g., a neural network layer, of a neural network implemented at a host processor. The host processor can include a group of tiles, processors, or processor cores that are interconnected via a ring bus. In some implementations, each tile is configured to perfonn computations for a given neural network layer and to generate the corresponding output activations for that layer based on the computations. The output activations can be routed from one or more tiles via the ring bus and an “Outfeed” (or output feed) of activations is generated to route the output activations to the output activation FIFO 350.

[0086] The compressor hardware 320 can be located between a set of HIB FIFOs and the HIB 330. A Huffman field can be included in, or added to, a FlexibleDMA instruction 310 to enable compression for a given FlexibleDMA instruction, for example, when the destination of the compressed data is memory device 130 (e.g., DRAM). In some cases, the Huffman field is included in a FlexibleDMA instruction 310 based on an instruction set architecture (ISA) of the host processor that issues the instruction to enable compression for a given set of uncompressed data.

[0087] The FlexibleDMA instruction 310 is used to generate multiple DMA descriptors that specify or point to a specific set of data to be encoded / compressed by compressor 320. In some implementations, the data pointed by the DMA descriptors are compressed separately by the compressor 320. Before performing the compression, the compressor hardware 320 is configured to fetch a corresponding codebook that provides a mapping scheme for mapping / encoding symbols of the uncompressed data to encoded symbols of compressed data. The address of the memory location that stores the codebook can be specified in the FlexibleDMA instruction or some other instruction generated at the host processor or the SoC 102.

[0088] When operating in decompression mode, the decompressor hardware 340 included in the decompression hardware architecture 300B reads the compressed data based on the control input that include the DMA descriptors generated by the flexible DMA unit 310. The decompressor 340 then performs decompression inline on the compressed data, generates decompressed data in response to decompressing or decoding the compressed data, and sends or writes the un / decompressed data to the buffers 360, 370 of the HIB 330. The buffers 360. 370 can each be a first-in-first-out (FIFO) buffer, a circular or ring buffer, or the like, in various implementations.

[0089] In the example of Fig. 3B, the FIFO buffer 360 is an input activation FIFO buffer configured to store input activations that are generated as outputs of a prior neural network layer or inference pass performed at a host or other processor of the SoC 240. As indicated above, the host processor can include a group of tiles, processors, or processor cores. Each tile can perform computations for a given neural network layer using a subset of the input activations and a corresponding set of weights / parameters for the layer.

[0090] The input activations are stored in the input activation FIFO buffer 360. whereas the set of weights / parameters for the layer are stored in the parameter FIFO buffer 370. Each of the inputs and parameters can be routed from their respective FIFO buffers 360, 370 via a “Popinput” operation. In some implementations, the computations for that layer result in a corresponding set of output activations that generated as an output of that layer, can be compressed using compressor 320 and stored in the memory device 130, as discussed above. For additional clarify in reviewing the disclosure, a representative table of terminologies is included below with example descriptions for terminologies associated with compressor 320 and decompressor 340.Table 1. Terminologies of the compressor and decompressor

[0091] Fig. 4 shows an example memory layout of uncompressed superblocks 410. densely packed compressed superblocks 420, sparsely packed compressed superblocks 430, and metadata 440. In some implementations, a tensor is a multi-dimensional array. For example, a zero-dimensional tensor is a scalar value, a one-dimensional tensor is a vector, and a two-dimensional tensor is a matrix.

[0092] A tensor can logically be divided into multiple superblocks, such that particular dimensions of a tensor correspond to one or more superblocks. For example, elements along an X-dimension of a tensor can correspond to an uncompressed superblock 0 that is later compressed to become a compressed superblock 0, whereas elements along a Y- (or Z-) dimension of a tensor can correspond to an uncompressed superblock 1 that is later compressed to become a compressed superblock 1. In the example of Fig. 4, after compression, if the packing mode is densely packed, the densely packed compressed superblocks 420 are stored contiguously without any zero padding.

[0093] Alternatively, if the packing mode is sparsely packed, the sparsely packed compressed superblock is first zero padded to superblock size bytes bytes aligned before being stored contiguously in memory. For a given tensor, system 100 can be configured to select between one of the two packing modes for use in a compression / encoding operation. The metadata 440 for each superblock is stored contiguously in a separate memory buffer. In some implementations, system 100 does not impose a requirement on the base address between the compressed superblocks and the metadata or metadata tags.

[0094] Fig. 5 shows an example memory' layout 500 of an uncompressed superblock 510, a compressed superblock 520, and metadata of a superblock 530. In the example of Fig. 5. the uncompressed superblock 510 includes multiple uncompressed data blocks, e.g., Block 0, Block 1, Block 2, ... , Block N. The uncompressed superblock 510 can represent a set of uncompressed data blocks corresponding to an uncompressed superblock of outputs. In some implementations, the outputs are activations (e.g., “output activations”) generated by a neural network layer as part of an inference computation performed for an ML workload. In some other implementations, the outputs are generated based on some other algorithm (or ML algorithm) that is implemented using a host processor of the SoC 240.

[0095] The example memory' layout 500 of Fig. 5 includes a superblock that is divided into num split _points _per super block blocks, where num split _points _per superblock = 8. The number of split points can be more than eight or fewer than eight. For example, the number of split points can be defined based on a variable such as N, which can be an integer greater than or equal to one (1). In at least one example, the number of split points can be zero (0). In some implementations, the number of split points can be predefined or dynamically configured, for example, at compile time or runtime. In the example of Fig. 5, there are num split _points _per superblock slots in the superblock metadata 530.

[0096] Some (or all) of the uncompressed data blocks 0, 1, 2, 3, 4, 5, 6, and 7 are compressed to generate a compressed superblock 520. Each uncompressed data Block 0, 1, 2, etc., in the uncompressed superblock 510 can be compressed separately from one another, for example, using a respective encoder / compressor for each data block. In some implementations, two or more of the uncompressed data blocks 0-7 are compressed concurrently (i.e., in parallel).

[0097] Information about the size of the compressed block i can be stored in the -th slot of the superblock metadata 530. For example, compressed superblock 520 includes compressed Block 0 (522), compressed Block 1 (523), compressed Block 2 (524), compressed Block 3 (525), and compressed Block 6 (526). The sizes indicated at each block of the superblock metadata 530 indicate a respective size of a corresponding compressed data block in the compressed superblock 520. For example, a metadata tag for compressed Block 0 (522) indicates the size of block 522 post-compression is 24 bytes, a metadata tag for compressed Block 2 (524) indicates the size of block 524 post-compression is 33 bytes, and a metadata tag for compressed Block 6 (526) indicates the size of block 526 post-compression is 20 bytes. Notably, a metadata tag for compressed Block 4 indicates the size of Block 4 post-compression is 36 bytes.

[0098] In some implementations, the superblock metadata 530 includes a metadata superblock size 535 that is established in bytes. For the uncompressed superblock 510, all of the slots in the metadata superblock 530 can be set to block size bytes bytes.

[0099] Fig. 6 shows an example superblock partitioning associated with tensor spill operation 605. Upon receiving a DMA descriptor, the compressor 320 can be configured to partition or break up a tensor into multiple superblocks 610, where the size of each superblock is superblock size bytes bytes. Because the start address and the size of a descriptor are arbitrary and cannot guarantee to be a multiple of superblock size, some superblocks 620 could have partial data. In some implementations, system 100 can be configured to such that partial superblocks are not compressed and can be bypassed by the compressor 320, which can translate to power savings and improved compression efficiency. For example, if the start address of the descriptor is not aligned to superblock size bytes, the first superblock may include partial data (identified as superblock 620 in the example of Fig. 6). As another example, if {size - (superblock size bytes + start address)} % superblock size bytes are not aligned to superblock size bytes, the last superblock may also include partial data.

[0100] Fig. 7 show s an example of compressed block padding. Upon partitioning data identified by the descriptor into superblocks, the compressor 320 fetches each superblock size bytes of data from the HIB buffer as specified in the instruction and performs compression. For example, the buffer can be a data buffer that stores activation outputs of one or more layers of a neural netw ork.

[0101] To compress a superblock, the compressor divides the superblock into superblock size bytes ' block size bytes blocks, where each block has block size bytes. The compressor then compresses each block independently based on a codebook obtained using a codebook request generator, which is described below7with reference to Fig. 10. Due to variable length coding, the compressed block may not be byte aligned. If a block 702 is not byte aligned, the compressor 320 can pad the compressed block to be byte aligned, as indicated at block 704 in the example of Fig. 7.

[0102] Fig. 8 shows an example of partitioning a descriptor into multiple superblocks. In some implementations, to perform decompression, the decompressor 340 partitions data identified by a descriptor generated by the FlexibleDMA instruction into superblocks, such as superblocks 802, 804. Because the start address and size of a descriptor (810) are arbitrary and might not be a multiple of the superblock size, an example DMA descriptor may read partial data from the superblock at the beginning or end.

[0103] When the minimum access granularity is a superblock, even though only partial data of a superblock is needed, the hardware can be configured to fetch the entire compressed superblock and corresponding metadata, decompress the fetched compressed superblock and the corresponding fetched metadata, and then extract the bytes that are needed / required. Fig.8 shows that a total of 4 superblocks are fetched for the DMA descriptor. The DMA descriptor only accesses part of the data in the first and last superblocks. The superblocks that are required by this DMA descriptor are superblocks 802, 804. 806, 808.

[0104] Fig. 9 is a block diagram that indicates an example calculation of block offset_bytes and size bytes within a superblock. In some implementations, the system 100 is configured to fetch metadata (or metadata tags) for a superblock prior to decompressing the superblock. In these implementations, to decompress a superblock, its metadata must be fetched first. The metadata address for a given superblock offset i of a compressed tensor can be calculated using the equation 910. Once the metadata is received, the system 100 can perform a prefix sum of each split point in the metadata to calculate the byte offset of each compressed block in the compressed superblock. In some implementations, the byte offset of compressed block j is the sum of all the split points from 0 to / . Then, the system 100 can decompress each compressed block individually by extracting the compressed block.

[0105] As an example, Fig. 9 shows the calculation result 915 of block offset bytes and size bytes within a superblock. In this example, the size bytes field indicates the valid bytes (including padding) that are required to produce uncompressed data in the size of block size bytes. After decompressing the compressed blocks, system 100 can reconstruct the original uncompressed superblock by concatenating the uncompressed blocks.

[0106] Fig. 10 is a block diagram of an example architecture 1000 of a compressor 320, described above with reference to the example of Fig. 3 A. Each item or component of the architecture 1000 of the compressor 320 can be implemented in hardware, software, or both, including firmware. In some implementations, the compressor 320 is used with a host processor such as a TPU, and the TPU subsystems (e.g., some (or all) processing context, processor cores and / or clusters) share a single compressor instance.

[0107] The compressor 320 includes A encoders 1002-0, 1002-1. 1002-2, ... , 1002-A. where A is an integer greater than or equal to one (1). The A encoders can be included in an example encoder circuit corresponding to compressor 320. Each encoder 1002 can receive an uncompressed block as input and generate a compressed block as an output over one or more clock cycles. The encoder 1002 can generate the compressed block in block size bytes cycles or per clock cycle.

[0108] The compressor 320 includes a codebook request generator 1004 that fetches codebooks from a memory device of system 100, such as memory’ device 130 (e.g., DRAM), disk memory 260, or another memory, and stores them in a codebook cache 1006. The compressor 320 includes a descriptor converter 1008 that converts an uncompressed descriptor into descriptors of superblocks. The descriptor converter 1008 receives its input from the Flexible DMA unit 310 that generates DMA descriptors, which comprise control information for the compressor 320.

[0109] The compressor 320 includes a metadata generator 101 that generates the DMA descriptors for the metadata for compressed superblocks. The compressor 320 includes a deinterleaver 1012 that perfonns alignment detection / checking. The compressor 320 includes a descriptor generator 1018 that receives, as input, an output of the descriptor converter 1008 and generates descriptors that can be duplicated per processing context of a host processor that generates the uncompressed data blocks.

[0110] Each encoder 1002 of the compressor 320 includes a respective Huffman encoding engine 1014 that encodes or compresses one or more uncompressed data blocks based on the Huffman compression algorithm or modified version of the Huffman compression / encoding algorithm. For example, each respective Huffman encoding engine 1014 cooperates with the requested codebook to replace the uncompressed symbols with Huffman code in accordance with the requested codebook. The compressor 320 includes an interleaver engine 1016 that reassembles the compressed blocks into compressed superblocks. The compressor 320 includes a compression ratio checker 1020 that performs selective compression based on the compression threshold.

[0111] In some implementations, generating a compressed superblock includes: i) determining whether a compressed size for a set of uncompressed data blocks exceeds a threshold size; ii) generating, based on a selective encoding / compression technique, multiple compressed data blocks only when the compressed size is less than the threshold size; and iii) generating the compressed superblock in response to compressing the set of uncompressed data blocks based on the selective encoding technique. The threshold size can be a current uncompressed size of the set of uncompressed data blocks. The system 100 can dynamically determine the threshold size, for example, based on a ML workload assigned to a host processor in data communication with the encoder circuitry’.

[0112] Before performing compression for a FlexibleDMA instruction, the codebook request generator 1004 receives the codebook address from the FlexibleDMA instruction. Using the <context_id, codebook_base_address>, the codebook request generator 1004 firstchecks if the codebook is already stored in the codebook cache. If the codebook does not exist in the cache, the codebook request generator 1004 fetches the codebook from a memory device of system 100, such as a RAM module, a dedicated codebook memory or register . After the codebook has been stored in the cache, compression can begin.

[0113] FlexibleDMA generates DMA descriptors, which contain control information for the compressor 320. The data (e.g., an uncompressed data block(s)) to be compressed is stored in a buffer of the HIB. If the Huffman field is enabled in the instruction, each descriptor from a FlexibleDMA instruction is compressed sequentially. The de-interleaver 1012 detects and / or checks alignment of descriptors with reference to uncompressed data blocks (e g., 32B) obtained from a HIB FIFO register. The de-interleaver 1012 detects and / or checks the alignment and partitions the descriptor(s) into superblocks after receiving a descriptor from the FlexibleDMA unit. In some implementations, system 100 is configured to such that partial superblocks will bypass the encoders and are stored directly in a symbol superblock buffer. System 100 can update some (or all) of the split points in the metadata of the partial superblock (including the blocks that are not present in the partial superblock) with block size bytes as the value.

[0114] Each full superblock can be further segmented into N = (superblock size bytes / block size bytes) blocks. The first superblock is scheduled to the first N encoders (encoders 0 to 7V-1), then the second superblock is scheduled to encoders N to 27V -1, and the third superblock is scheduled to encoders IN to 37V- 1, and so on. As discussed above with reference to the Huffman encoding engine 1014, each encoder 1002 -TV compresses an uncompressed block of data in block size bytes cycles by looking up the codebook and replacing the symbols with Huffman code. The compressed block can be stored in a respective encoded block FIFO / buffer 1015 of the compressor 320, while waiting to be read by the interleaver 1016.

[0115] The interleaver 1016 perform or implement reconstruction, and generate a corresponding reconstruction output, by adhering to the following rule: Encoder X (0 < X < N) will output its z-th compressed block, for example, if at least one previous Encoders Y (0 <= Y < X) have already outputted their z-th compressed block. In some implementations, a compressed superblock and its corresponding uncompressed superblock are generated as outputs of the encoded superblock FIFO / buffer 1017 and the symbol superblock FIFO / buffer 1119, respectively, at the same time (e.g., in parallel).

[0116] The compression ratio checker 1020 is configured to check the compressed data size against a threshold. For example, the compression ratio checker 1020 can perform thischeck using comparator logic / circuit of the architecture 1000. In some implementations, the comparator of the compression ratio checker 1020 compares two input signals, e.g., the compressed data size and the threshold size value, and outputs a binary (or analog) signal indicating the larger value. If the compressed data size is above the threshold, the compressed superblock is discarded; otherwise, the uncompressed superblock is discarded. Finally, the superblock and the metadata are written back to DRAM.

[0117] Fig. 1 1 is a block diagram of an example architecture 1100 of a decompressor 340, described above with reference to the example of Fig. 3B. Each item or component of the architecture 1100 of the decompressor 340 can be implemented in hardware, software, or both, including firmware. In some implementations, the decompressor 340 is used with a host processor such as a TPU, and the TPU subsystems (e.g., some (or all) processing context, processor cores and / or clusters) share a single compressor instance.

[0118] The decompressor 340 includes N decoders 1102-0, 1102-1, 1102-2, ... , I 102-A. where TV is an integer greater than or equal to one (1). The N decoders can be included in an example decoder circuitry corresponding to decompressor 340. Each decoder 1102 can receive a compressed data block as input and generate an uncompressed data block as an output over one or more clock cycles. The decoder 1102 can generate the uncompressed block in block size bytes cycles or per clock cycle.

[0119] The decompressor 340 includes a codebook request generator 1104 that fetches codebooks from a memory device, such as device 130 or another device of system 100, and stores them in a codebook cache 1106. The decompressor 340 includes a request generator 1108 that issues DMA descriptors 1109 for the compressed superblocks and the corresponding metadata (e.g., metadata tags). The decompressor 340 includes an encoded data router 1112 that distributes the metadata and respective blocks (from each superblock(s)) to a corresponding decoder 1102-0, 1102-1, 1102-2, ... , 1102-JV.

[0120] Each decoder 1102 of the decompressor 340 includes a respective Huffman decoding engine 1114 that decodes or decompresses one or more compressed data blocks of a superblock based on the Huffman decompression algorithm or modified version of the Huffman decompression / decoding algorithm. For example, each respective Huffman decoding engine 1114 cooperates with the requested codebook to replace the Huffman code corresponding to the compressed symbols with uncompressed symbols / data blocks based on the Huffman decoding / decompression operation performed in accordance with the requested codebook. The decompressor 340 includes an interleaver 1118 that reassembles the decodedblocks into a superblock via the superblock FIFO 1120 before sending (1130) the decompressed superblocks to the buffer of the HIB, such as the HIB FIFOs.

[0121] In some implementations, prior to performing decompression based on a FlexibleDMA instruction, the codebook request generator 1104 receives a codebook address from the FlexibleDMA instruction. The codebook request generator 1104 can use a <context_ID. codebook_base_address>, to detennine whether the codebook is already stored in the codebook cache 1106. If the codebook control logic of decompressor 340 (or system 100) determines that the codebook does not exist in the codebook cache 1 106, then the codebook request generator 1104 fetches the codebook from a memory7of system 100, such as DRAM, disk memory 260, or some other memory7of system 100 configured for storing codebooks. The decompressor 340 is configured to initiate decompression after a codebook has been stored in the codebook cache 1106.

[0122] The request generator 1108 is configured to manage and execute superblock partitioning and address calculation for both the densely packed mode and the sparsely- packed mode. After partitioning, the request generator 1108 is configured to fetch the superblocks and metadata and store them in the metadata FlFO / buffer 1113 and the encoded data FIFO / buffer 1115, respectively.

[0123] Similar to the compressor 320 in Fig. 10, the encoded data router 1112 parses the metadata and breaks a superblock into A blocks. The first superblock is scheduled to the first N decoders (decoder 0 to A-l), then the second superblock is scheduled to decoder A to 2A-1, and the third superblock is scheduled to decoder 2Ato 3A-1, and so on. As discussed above with reference to the Huffman decoding engine 1114, each decoder 1102-A decompresses a compressed block of data in block size bytes cycles by looking up the codebook and replacing the Huffman code with symbols. In each decoder 1102-A, the decompressed block for that decoder 1102-A can be stored in the decoded block FIFO / buffer 11 17, while waiting to be read by the interleaver 1118. In some implementations, the encoded data router 1112 is configured with a bypass path to the superblock buffer directly (through the interleaver 1118) for uncompressed superblocks.

[0124] To simplify and / or streamline the reconstruction of an uncompressed output, the interleaver 1118 can be configured to apply the following reconstruction rule: Decoder X (0 < X < N) will output its z-th decoded block, for example, if all previous decoders Y (0 <= Y < X) have already outputted their z-th decoded block. The output data of the interleaver 1118 is written to the superblock FIFO / buffer 1120, which is then written (1130) to the HIB buffer. In some implementations, subsequent data flows remain the same as other DMA requestors.The POP INPUT instruction can be issued to pop the data from the HIB buffer and write it to shared memory.

[0125] Fig. 12 is an example process 1200 for block-based encoding that generates compressed data blocks for storing in memory. The process 1200 is implemented or executed using one or more components of the system 100 as described in relation to the various Figures and examples herein. In some implementations, process 1200 is implemented or executed using encoder circuitry for a hardware integrated circuit of the SoC 240. Thus, the descriptions of process 1200 will reference the above-mentioned computing resources of system 100 and device / system 200. In some examples, the steps or actions of process 1200 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0126] The system 100 receives an instruction comprising an opcode that specifies a compression operation (1202). In some implementations, the instruction is used by a firmware control block of the host processor 114. The instruction can be issued based on an instruction set architecture (ISA) of the host processor 114 on the SoC 240. The ISA is used to issue instructions to compute tiles of a host or other processor of the SoC 240. The instruction can be a direct memory access (“DMA”) instruction issued by, or on behalf of, a host interface block (HIB) of system 100. In some implementations, is a FlexibleDMA opcode that indicates whether compression is enabled for data or outputs that are being passed from the host processor to the memory device 130.

[0127] Based on the opcode, the system 100 identifies a set of uncompressed data blocks corresponding to an uncompressed superblock of outputs (1204). In some implementations, the outputs are output activations generated while processing inputs through a layer of a neural network implemented on a host processor on an integrated circuit of the system 100.

[0128] For the set of uncompressed data blocks, the system 100 can compress each uncompressed data block in parallel using encoders at the encoder circuitry (1206). In some implementations, the encoder circuitry includes multiple individual encoders, and each encoder is used to encode or compress a distinct uncompressed data block, as described above with reference to Fig 10. The system 100 includes encoder circuitry configured to generate multiple compressed data blocks in response to compressing each uncompressed data block in parallel (1208). The system 100 generates a compressed superblock using encoder circuitry of the compressor 320 (1210). In some implementations, the compressorgenerates the superblock by selectively including one or more of the compressed data blocks in the superblock.

[0129] As indicated above, in some implementations, the respective steps of process 1200 are performed at a hardware integrated circuit of system 100 as part of a larger compute operation to generate a ML output, including an output for a layer of a neural network that is the basis for one or more ML models. For example, the output can be a portion of a computation for a ML task or inference workload to generate an image processing, speech processing, or image recognition output. Workload tasks other than image and speech processing are also supported and within the scope of this disclosure. As indicated above, a portion of the integrated circuit can include special-purpose processors or hardware ML accelerators configured to accelerate computations for generating a broad range of data processing outputs.

[0130] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.

[0131] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0132] The term “computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. l ' l

[0133] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0134] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0135] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

[0136] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random-access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.

[0137] Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0138] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by wayof example semiconductor memory' devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0139] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory' feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0140] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e g., as a data sen' er, or that includes a middleware component, e.g., an application sen’ er, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleyvare, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“‘LAN”) and a w ide area network (“WAN”), e.g., the Internet.

[0141] The computing system can include clients and servers. A client and server are generally remote from each other and ty pically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0142] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of yvhat may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment.Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0143] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0144] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

What is claimed is:

1. A method performed using encoder circuitry for an integrated circuit, the method comprising: receiving an instruction comprising an opcode that specifies a compression operation; identifying, based on the opcode, a set of uncompressed data blocks corresponding to an uncompressed superblock of outputs generated by a host processor on the integrated circuit; for the set of uncompressed data blocks: compressing, by the encoder circuitry, each uncompressed data block in parallel using a plurality of encoders at the encoder circuitry: generating, by the encoder circuitry', a plurality' of compressed data blocks in response to compressing each uncompressed data block in parallel; and generating, using the encoder circuitry, a compressed superblock by selectively including one or more of the compressed data blocks.

2. The method of claim 1, further comprising: generating, based on the compressed superblock, a stream of compressed data for routing to a memory device that exchanges data communications with the host processor.

3. The method of claim 1, wherein generating the compressed superblock comprises: determining whether a compressed size for the set of uncompressed data blocks exceeds a threshold size; generating, based on a selective encoding technique, the plurality of compressed data blocks only when the compressed size is less than the threshold size; and generating the compressed superblock in response to compressing the set of uncompressed data blocks based on the selective encoding technique.

4. The method of claim 1, further comprising: generating a metadata tag that indicates a compression state of at least one of the plurality of superblocks.

5. The method of claim 4, wherein the threshold size is a current uncompressed size of the set of uncompressed data blocks.

6. The method of claim 4, further comprising: dynamically determining the threshold size based on a machine-learning workload assigned to a host processor in data communication with the encoder circuitry.

7. The method of claim 1, wherein a superblock represents a minimum granularity for compressing a set of uncompressed data blocks.

8. The method of claim 1 , wherein the opcode: i) is a field in the instruction; and ii) represents a compression enable parameter of the instruction.

9. The method of claim 8, further comprising: determining, based on the opcode, that compression by the encoder circuitry is enabled; and providing a descriptor as a control input to a compressor of the encoder circuitry in response to determining that compression is enabled.

10. The method of claim 1, wherein: i) each data block in the set of uncompressed data blocks comprises output activations of a neural network layer; and ii) at least one data block in the set of uncompressed data blocks is obtained from an output activation first-in first-out (FIFO) registers of the encoder circuitry.

11. A system comprising: a processing device; encoder circuitry for an integrated circuit; and a non-transi tory machine-readable storage device for storing instructions that are executable by the processing device to cause performance of operations comprising: receiving an instruction comprising an opcode that specifies a compression operation; identifying, based on the opcode, a set of uncompressed data blocks corresponding to an uncompressed superblock of outputs generated by a host processor on the integrated circuit;for the set of uncompressed data blocks: compressing, by the encoder circuitry, each uncompressed data block in parallel using a plurality of encoders at the encoder circuitry; generating, by the encoder circuitry, a plurality of compressed data blocks in response to compressing each uncompressed data block in parallel; and generating, using the encoder circuitry, a compressed superblock by selectively including one or more of the compressed data blocks.

12. The system of claim 11, wherein the operations further comprise: generating, based on the compressed superblock, a stream of compressed data for routing to a memory device that exchanges data communications with the host processor.

13. The system of claim 11, wherein generating the compressed superblock comprises: detennining whether a compressed size for the set of uncompressed data blocks exceeds a threshold size; generating, based on a selective encoding technique, the plurality of compressed data blocks only when the compressed size is less than the threshold size; and generating the compressed superblock in response to compressing the set of uncompressed data blocks based on the selective encoding technique.

14. The system of claim 11, wherein the operations further comprise: generating a metadata tag that indicates a compression state of at least one of the plurality of superblocks.

15. The system of claim 14, wherein the threshold size is a current uncompressed size of the set of uncompressed data blocks.

16. The system of claim 14, wherein the operations further comprise: dynamically determining the threshold size based on a machine-learning workload assigned to a host processor in data communication with the encoder circuitry.

17. The system of claim 11 , wherein a superblock represents a minimum granularity for compressing a set of uncompressed data blocks.

18. The system of claim 11, wherein the opcode: i) is a field in the instruction; and ii) represents a compression enable parameter of the instruction.

19. The system of claim 18, wherein the operations further comprise: determining, based on the opcode, that compression by the encoder circuitry is enabled; and providing a descriptor as a control input to a compressor of the encoder circuitry in response to determining that compression is enabled.

20. The system of claim 11 , wherein: i) each data block in the set of uncompressed data blocks comprises output activations of a neural network layer; and ii) at least one data block in the set of uncompressed data blocks is obtained from an output activation first-in first-out (FIFO) registers of the encoder circuitry’.

21. A non-transitory machine-readable storage device storing instructions for data operations performed using encoder circuitry’ for an integrated circuit, the instructions being executable by a processing device to cause performance of operations comprising: receiving an instruction comprising an opcode that specifies a compression operation; identifying, based on the opcode, a set of uncompressed data blocks corresponding to an uncompressed superblock of outputs generated by a host processor on the integrated circuit; for the set of uncompressed data blocks: compressing, by the encoder circuitry, each uncompressed data block in parallel using a plurality of encoders at the encoder circuitry; generating, by the encoder circuitry, a plurality of compressed data blocks in response to compressing each uncompressed data block in parallel; and generating, using the encoder circuitry, a compressed superblock by selectively including one or more of the compressed data blocks.

Images