System and method for compressing activation map of neural network and decompression method thereof

Abstract

Disclosed are a system, method, and decompression method for compressing an activation map of a neural network. The system and method provide compression and decompression of activation maps of a layer of a neural network. For compression, the values ​​of the activation map are sparsified, and the activation map is configured as a tensor having a tensor size of H×W×C, where H represents the height of the tensor, W represents the width of the tensor, and C represents the number of channels of the tensor. The tensor is formatted into at least one block having values. Each block is encoded independently of other blocks of the tensor using at least one lossless compression mode. For decompression, each block is decoded independently of other blocks using at least one decompression mode corresponding to the at least one compression mode used to compress the block; and each block is deformatted into a tensor having a size of H×W×C.

Description

[0001] This patent application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 756,067, filed on November 5, 2018, and U.S. Non-Provisional Patent Application No. 16 / 223,092, filed on December 17, 2018, the disclosures of which are incorporated herein by reference in their entireties. Technical Field

[0002] The subject matter disclosed herein generally relates to a method and system that provides lossy encoding / decoding of activation maps of a neural network to reduce memory requirements and accelerate execution of the neural network. Background Art

[0003] Recently, deep neural networks have dominated a wide range of applications, from computer vision (image classification and segmentation) and natural language processing (word-level prediction, speech recognition, and machine translation) to medical imaging. Specialized hardware has been designed to run deep neural networks as efficiently as possible. However, on the software side, some research has focused on minimizing the memory and computational requirements of these networks during operation.

[0004] When trying to train a neural network on an embedded device with limited memory, it is very important to minimize the memory requirements of the algorithm as much as possible. During training, most of the memory is actually consumed by the activation map. For example, the activation map of current deep neural network systems consumes between approximately 60% and 85% of the total memory required by the system. Therefore, reducing the memory footprint associated with the activation map becomes an important part of reducing the overall memory footprint of the training algorithm.

[0005] In neural networks where rectified linear units (ReLUs) are used as activation functions, activation maps tend to become sparse. For example, in the Inception-V3 model, most activation maps have a sparsity greater than 50%, and in some cases, most activation maps have a sparsity exceeding 90%. Therefore, there is a strong market demand for compression systems that can target this sparsity to reduce the memory requirements of training algorithms. Summary of the Invention

[0006] An example embodiment provides a system for compressing an activation map of a layer of a neural network, wherein the system may include: a processor programmed to initiate executable operations, the executable operations including: sparsifying a number of non-zero values ​​of the activation map using the processor; configuring the activation map into a tensor having a tensor size of H×W×C, where H represents the height of the tensor, W represents the width of the tensor, and C represents the number of channels of the tensor; formatting the tensor into at least one block of values; and encoding the at least one block independently of other blocks of the tensor using at least one lossless compression mode. In one embodiment, the at least one lossless compression mode may be selected from the group consisting of: exponential Golomb coding, sparse exponential Golomb coding, sparse exponential Golomb stripped minimum coding, Golomb-Rice coding, exponential mantissa coding, zero coding, fixed length coding, and sparse fixed length coding. In another embodiment, the at least one lossless compression mode selected for encoding the at least one block may be different from the lossless compression mode selected for encoding another block of the tensor. In another embodiment, the at least one block may be encoded independently of other blocks of the tensor using multiple lossless compression modes.

[0007] Another example embodiment provides a method for compressing an activation map of a neural network. The method may include: sparsifying, using a processor, a plurality of non-zero values ​​of the activation map; configuring the activation map into a tensor having a tensor size of H×W×C, where H represents the height of the tensor, W represents the width of the tensor, and C represents the number of channels of the tensor; formatting the tensor into at least one block having values; and encoding the at least one block using at least one lossless compression mode independently of other blocks of the tensor. In one embodiment, the at least one lossless compression mode may be selected from the group consisting of: exponential Golomb coding, sparse exponential Golomb coding, sparse exponential Golomb stripped minimum coding, Golomb-Rice coding, exponential mantissa coding, zero coding, fixed-length coding, and sparse fixed-length coding. In another embodiment, the at least one lossless compression mode selected for encoding the at least one block may be different from the lossless compression mode selected for compressing another block of the tensor. In another embodiment, encoding the at least one block may further include encoding the at least one block independently of other blocks of the tensor using multiple lossless compression modes. In one embodiment, the method may further include outputting the at least one block encoded as a bitstream.

[0008] Another example embodiment provides a method for decompressing a sparsified activation map of a neural network, the method comprising: decompressing, using a processor, compressed blocks of a bitstream representing values ​​of the sparsified activation map to form at least one decompressed block, the decompressed block being decompressed independently of other blocks of the activation map using at least one decompression mode corresponding to at least one lossless compression mode used to compress the at least one block; and deformatting the decompressed block into portions of a tensor having a size of H×W×C, where H represents the height of the tensor, W represents the width of the tensor, and C represents the number of channels of the tensor, the tensor being the decompressed activation map. In one embodiment, the at least one lossless compression mode may be selected from the group consisting of: exponential Golomb coding, sparse exponential Golomb coding, sparse exponential Golomb stripped minimum coding, Golomb-Rice coding, exponential mantissa coding, zero coding, fixed length coding, and sparse fixed length coding. In one embodiment, the method may further include: using a processor to sparsify a number of non-zero values ​​of the activation map; configuring the activation map into a tensor having a tensor size of H×W×C; formatting the tensor into at least one block having values; and encoding the at least one block using at least one lossless compression mode independently of other blocks of the tensor. In one embodiment, the at least one lossless compression mode selected for compressing the at least one block may be different from the lossless compression mode selected for compressing another block of the received tensor of the at least one activation map, and the step of compressing the at least one block may further include: compressing the at least one block using multiple lossless compression modes independently of other blocks of the received tensor of the at least one activation map. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the following sections, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the accompanying drawings, in which:

[0010] Figure 1A A functional block diagram depicting an example embodiment of a system for lossy compression and decompression of activation maps of a neural network in accordance with the subject matter disclosed herein;

[0011] Figure 1B depicts a functional block diagram of a compressor according to the subject matter disclosed herein;

[0012] Figure 1C depicts a functional block diagram of a decompressor in accordance with the subject matter disclosed herein;

[0013] Figure 2A and Figure 2B Depicting example embodiments of methods for encoding and decoding activation maps of a deep neural network, respectively, according to the subject matter disclosed herein; and

[0014] Figure 3 Depicting a flow of operations on activation maps at layer L of a neural network in accordance with the subject matter disclosed herein. DETAILED DESCRIPTION

[0015] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, those skilled in the art will appreciate that the disclosed aspects may be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits have not been described in detail in order not to obscure the subject matter disclosed herein.

[0016] References throughout this specification to "one embodiment" or "an embodiment" mean that the specific features, structures, or characteristics described in conjunction with that embodiment may be included in at least one embodiment disclosed herein. Therefore, the phrases "in one embodiment" or "in an embodiment" or "according to an embodiment" (or other phrases with similar meanings) that appear in different places throughout this specification may not necessarily refer to the same embodiment. Furthermore, in one or more embodiments, specific features, structures, or characteristics may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" should not be construed as necessarily preferred or advantageous over other embodiments. Furthermore, depending on the context of the discussion herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. It should also be noted that the various figures (including component diagrams) shown and discussed herein are for illustrative purposes only and are not drawn to scale. Similarly, various waveforms and timing diagrams are shown for illustrative purposes only. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals have been repeated among the figures to indicate corresponding and / or analogous elements.

[0017] The terms used herein are only used for the purpose of describing specific exemplary embodiments and are not intended to limit the subject matter of the claims. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that when the terms "comprising" and / or "including" are used in this specification, the features, wholes, steps, operations, elements and / or components described are described, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their groups is not excluded. Unless clearly defined as such, the terms "first", "second" and the like as used herein are used as labels for the nouns preceding them and do not imply any type of ordering (e.g., space, time, logic, etc.). In addition, the same reference numerals may be used across two or more figures to represent parts, components, blocks, circuits, units or modules with the same or similar functions. However, such usage is only for simplicity of description and ease of discussion and does not mean that the construction or structural details of such components or units are the same between all embodiments or that such commonly referenced parts / modules are the only way to implement the teachings of the specific embodiments disclosed herein.

[0018] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the subject matter belongs. It will also be understood that, unless expressly defined as such herein, terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and will not be interpreted as an idealized or overly formal meaning.

[0019] The subject matter disclosed herein relates to a system and method for providing lossy compression of activation maps of neural networks to reduce memory requirements and accelerate the execution of the neural network. In one embodiment, a lossy compression pipeline is provided using three general stages: a sparsification stage, a quantization stage, and an entropy coding stage. In the sparsification stage, the activation map of the neural network may be sparsified to reduce the number of non-zero values ​​in the activation map. In the quantization stage, the activation map of each layer is quantized. In the entropy coding stage, the quantized activation map may be divided into smaller units, referred to herein as compressed blocks, which are compressed using various compression schemes. In one embodiment, the compressed blocks are compressed to produce a bitstream representing the compressed activation maps of the layers of the neural network. The compressed blocks may be decompressed, dequantized, and reformatted to the original shape of the sparsified activation maps. The techniques disclosed herein can be implemented using hardware with relatively low complexity. It should be noted that even if sparsification makes the processing lossy, the techniques disclosed herein can be used to compress activation maps of a neural network without compromising accuracy.

[0020] The activation maps of each layer of the neural network can be encoded and decoded independently of the encoding of the activation maps of other layers and as required by the training algorithm. Although the lossless encoding / decoding techniques disclosed herein can compress all degrees of sparsity (including 0% and near 100% sparsity), if the number of zero values ​​in the activation map is relatively high, the techniques disclosed herein can be optimized. That is, the systems and methods disclosed herein achieve higher degrees of compression for correspondingly higher degrees of sparsity. In addition, the subject matter disclosed herein provides several modifications to existing compression algorithms, wherein the modifications can be used to exploit the sparsity of the activation map data to obtain greater degrees of compression.

[0021] In one embodiment, the encoder may be configured to receive as input a tensor of size H×W×C, where H corresponds to the height of the input tensor, W corresponds to the width of the input tensor, and C corresponds to the number of channels of the input tensor. The received tensor may be formatted into smaller blocks referred to herein as "compression units." Compression units may be independently compressed using a variety of different compression modes. The output produced by the encoder is a bitstream of compressed compression units. When a compression unit is decompressed, it is reformatted to its original shape as at least a portion of a tensor of size H×W×C.

[0022] The techniques disclosed herein can be applied to reduce the memory requirements of activation maps of neural networks configured to provide applications such as, but not limited to, computer vision (image classification, image segmentation), natural language processing (word-level prediction, speech recognition, and machine translation), and medical imaging. Neural network applications can be used in autonomous vehicles, mobile devices, robots, and / or other low-power devices (such as drones). The techniques disclosed herein reduce the memory consumption of neural networks during training and / or when embedded in a dedicated device. The techniques disclosed herein can be implemented on a general-purpose processing device or in a dedicated device.

[0023] Figure 1A A functional block diagram depicts an example embodiment of a system 100 for lossy compression and decompression of activation maps of a neural network in accordance with the subject matter disclosed herein. System 100 includes a processor 101, a memory 102, a compressor 103, and a decompressor 104. During training and / or during inference, compressor 103 and decompressor 104 compress activation maps 106 of neural network 105 to form a bitstream 114 and decompress bitstream 114 to re-form the activation maps, respectively. Prior to compressing the activation maps, compressor 103 and decompressor 104 are configured to use corresponding compression and decompression modes. System 100 may also include one or more additional processors (not shown), mass storage devices (not shown), and input / output devices such as, but not limited to, a keyboard (not shown), a display (not shown), and a pointing device (not shown).

[0024] The compressor 103 and the decompressor 104 can be implemented as modules. As used herein, the term "module" refers to any combination of software, firmware, and / or hardware configured to provide the functionality described herein related to the module. Software can be implemented as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in any embodiment described herein can include, for example, hardwired circuits, programmable circuits, state machine circuits, and / or firmware that stores instructions executed by programmable circuits, either alone or in any combination. Modules can be implemented together or individually as circuits that form part of a larger system (e.g., but not limited to, an integrated circuit (IC), a system on a chip (SoC), etc.). In addition, the processor 101 and the memory 102 can be components of the modules that form the compressor 103 and / or the decompressor 104. Alternatively, the processor 101 and the memory 102 can be used by the modules that form the compressor 103 and / or the decompressor 104.

[0025] Figure 1B Depicts a functional block diagram of a compressor 103 according to the subject matter disclosed herein. The compressor 103 may include a sparsifier 107, a quantizer 108, a formatter 109, and a lossless encoder 110. It should be noted that although the sparsifier 107 and the quantizer 108 are Figure 1B 108 are depicted as being separate from the compressor 103 , but in other embodiments, the sparsifier 107 and / or the quantizer 108 may be part of the compressor 103 .

[0026] The activation map 106 generated at the layer of the neural network 105 may be configured as a tensor of a predetermined size, for example, by the processor 101 and the memory 102. In one embodiment, the activation map 106 may be configured as a tensor of size H×W×C, where H corresponds to the height of the input tensor, W corresponds to the width of the input tensor, and C corresponds to the number of channels of the input tensor. The activation map 106 may be formed and stored as a single tensor of size H×W×C.

[0027] The activation map 106 of the neural network 105 is sparsified by a sparsifier 107 to form a sparse activation map 111 having an increased number of values ​​equal to zero so that the lossless compression performed by the encoder 110 will be more efficient. The sparsifier 107 uses additional regularization in the cost function to fine-tune the pre-trained neural network. Typically, when a neural network is trained, a cost function L(w) is minimized with respect to the weights w. The cost function L(w) contains two terms: a data term and a regularization term. The data term is typically a cross-entropy loss, while the regularization term is typically an L2 norm with respect to the network weights. During fine-tuning of the pre-trained network, the cost function L(w) is modified by adding a new regularization term:

[0028] L′(w)=L(W)+∑ j λ j ||A j ||1 (1)

[0029] Among them, A j =A j ·(w j ) is the activation map of layer j, w j is the weight of layer j, λ j is the Lagrange multiplier that controls the sparsity.

[0030] Optional Lagrange multiplier λ j To control the sparsity of the activation map of layer j. A larger λ j Will produce a sparse A j Intuitively, adding L1 regularization to A constrains the weights w to produce sparse outputs. Backpropagation adjusts the weights w to form a sparser A. Fine-tuning starts with a pre-trained network and minimizes a modified cost function L′(w). Fine-tuning lasts for a number of epochs (typically between 10 and 50 epochs, depending on the network and dataset).

[0031] If, after sparsification, the values ​​of the sparse activation map 111 have not been quantized from floating point numbers to integers, the unquantized values ​​of the sparse activation map 111 may be quantized by the quantizer 108 to integer values ​​of any bit width q (i.e., 8 bits, 12 bits, 16 bits, etc.) to form the sparsified and quantized activation map 112. Quantization by the quantizer 108 may also be considered as a way to introduce additional compression, if desired, but at the expense of accuracy. Typically, linear (uniform) quantization is used, and q can be any value between 1 bit and 16 bits. In one embodiment, because a new activation map is generated for each input image, quantization may occur during runtime.

[0032] To facilitate compression, the H×W×C sparsified and quantized activation map 112 may be formatted by the formatter 109 into blocks of values, where each block is referred to herein as a “compression unit” 113. That is, the H×W×C sparsified and quantized activation map 112 may be divided into smaller compression units. A compression unit 113 may include K elements (or values) in channel-major order, where K>0; scan lines (i.e., each block may be a row of the activation map); or K elements (or values) in row-major order, where K>0. Other techniques or methods for forming compression units 113 are also possible. For example, the loading pattern of the activation map of the corresponding neural network hardware may be used as the basis for the block (or compression unit) formatting technique.

[0033] Each compression unit 113 can be losslessly encoded or compressed by encoder 110 independently of the other compression units to form a bitstream 114. Bitstream 114 can be stored in memory 102 or in a memory associated with neural network 105. Each compression unit 113 can be losslessly encoded or compressed using any of several compression techniques referred to herein as "compression modes" or simply "modes." Example lossless compression modes include, but are not limited to: Exponential-Golomb encoding, Sparse-Exponential-Golomb encoding, Sparse-Exponential-Golomb-RemoveMin encoding, Golomb-Rice encoding, Exponent-Mantissa encoding, Zero-encoding, fixed-length encoding, and Sparse fixed-length encoding. It should be understood that other lossless encoding techniques may be used in addition to the example compression modes, or as alternatives to the example compression modes. However, it should also be noted that, with the exception of the Sparse Index Golomb compression mode and the Sparse Index Golomb minus compression mode, many of the example compression modes are publicly available or based on publicly available compression modes. Details of the Sparse Index Golomb compression mode and the Sparse Index Golomb minus compression mode are provided herein.

[0034] Exponential Golomb coding is a well-known compression scheme that assigns variable-length codes, where smaller numbers are assigned shorter codes. The number of bits used to encode a number increases exponentially, and a parameter, often called the order k parameter, controls the rate at which the number of bits increases. The following pseudocode provides example details of the Exponential Golomb compression scheme.

[0035] Let x, x>=0 be input, let k be parameter (order)

[0036] Generate output bitstream: <quotient code><remainder code>:

[0037] Business Code:

[0038] Encode q = floor(x / 2^k) using an order 0 exponential Golomb code

[0039] z=binary(q+1)

[0040] numBits = len(z)

[0041] Writes numBits-1 zero bits followed by z, represented by u

[0042] Remainder code:

[0043] Encode r=x%2^k in binary and represent it by f=binary(r)

[0044] Cascade u, f to generate output bit stream

[0045] An example of Exponential Golomb compression is:

[0046] x=23, k=3

[0047] q = floor(23 / 2^3) = 2

[0048] z=binary(2+1)=binary(3)=11

[0049] numbits=Len(z)=2

[0050] u=011 (2-1=1 zero followed by z)

[0051] f=binary(r)=binary(23%8)=binary(7)=111

[0052] Final output = 011 + 111 = 011111

[0053] Table 1 sets forth the Exponential Golomb compression pattern values ​​for input values ​​x=0-29 and for order k=0-3.

[0054] Table 1

[0055]

[0056]

[0057] The sparse Exponential Golomb compression mode is an extension or variation of the Exponential Golomb compression mode in which, if the value x to be encoded is 0, then the value x is represented by a "1" in the output bitstream. Otherwise, the Exponential Golomb encoding adds a "0" and then encodes x-1 using standard Exponential Golomb. In one embodiment where the block (compression unit) value is eight bits, an order k=4 provides the best results.

[0058] The sparse Exponential Golomb minus compression mode is an extension or variation of the sparse Exponential Golomb compression mode that uses the following rules: (1) Before encoding the value in the compression unit, determine the smallest non-zero value that can be represented by the variable y. (2) Then encode the variable y using the Exponential Golomb compression mode. (3) If the value x to be encoded is 0, encode it as "1"; (4) Otherwise, add "0" to the bitstream and then encode xy using the Exponential Golomb compression mode.

[0059] The Golomb-Rice compression mode and the exponent-mantissa compression mode are well-known compression algorithms. The following pseudo code illustrates example details of the Golomb-Rice compression mode.

[0060] Let x, x>=0 be the input and M be the parameter. M is a power of 2.

[0061] q=floor(x / M)

[0062] r=x%M

[0063] Generate output bitstream: <quotient code><remainder code>:

[0064] Business Code:

[0065] Write a string of length q with bits 1

[0066] Write bit 0

[0067] Remainder code: binary (r) with log2(M) bits

[0068] An example of the Columbus-Rice compression pattern is:

[0069] x=23, M=8, log2(M)=3

[0070] q = floor (23 / 8) = 2

[0071] r=7

[0072] Business code: 110

[0073] Remainder code: 111

[0074] Output = 110111

[0075] The zero encoding compression mode checks whether the compression unit is formed entirely of zeros and returns an empty bitstream if so. It should be noted that the zero compression mode cannot be used if the compression unit contains at least one non-zero value.

[0076] The fixed length encoding compression mode is a baseline or default compression mode that performs no compression and simply encodes the value of the compression unit using a fixed number of bits.

[0077] Finally, the sparse fixed length coding compression mode is the same as the fixed length coding compression mode, except that if the value x to be encoded is 0, it is encoded as 1, otherwise 0 is added and a fixed number of bits are used to encode non-zero values.

[0078] Return to reference Figure 1B In one embodiment, the encoder 110 may start with a 48-bit compressed bitstream 114, of which 16 bits may be used to represent the H, W, and C of the input tensor, respectively. Each compression unit 113 may be iteratively compressed for each available compression mode. The compression mode available for each compression unit may be fixed during the compression of the activation map. In one embodiment, the entire range of available compression modes may be represented by L bits. For example, if four compression modes are available, a two-bit prefix may be used to indicate the corresponding index of the four available compression modes (i.e., 00, 01, 10, and 11). In an alternative embodiment, a prefix variable-length encoding technique may be used to save some bits. For example, the index of the compression mode most commonly used by the encoder 108 may be represented by "0", and the second most commonly used compression mode, the third most commonly used compression mode, and the fourth most commonly used compression mode may be represented by "10", "110", and "111", respectively. If only one compression mode is used, it is not necessary to append the index to the beginning of the compression unit's bitstream 114.

[0079] In one embodiment, when compression unit 113 is compressed, all available compression modes can be run, and the compression mode that has produced the shortest bitstream can be selected. The corresponding index of the selected compression mode can be appended as a prefix to the beginning of the bitstream of the particular compression unit, and the resulting bitstream of the compression unit can then be added to the bitstream of the entire activation map. This process can then be repeated for all compression units of the activation map. Each respective compression unit of the activation map can be compressed using a compression mode that is different from the compression mode used for adjacent or neighboring compression units. In one embodiment, a small number of compression modes (such as two compression modes) can be used to reduce the complexity of compressing the activation map.

[0080] Figure 1C Depicts a functional block diagram of a decompressor 104 according to the subject matter disclosed herein. The decompressor 104 decompresses the bitstream 114 to form an activation map 120 ( Figure 1A ), where activation map 120 is a lossy decompression corresponding to the original non-sparsed activation map 106. Thus, neural network 105′ may be a modified version of the original neural network 105.

[0081] The decompressor 104 may include a decoder 115, a deformatter 116, and a dequantizer 117. It should be noted that although the dequantizer 117 is Figure 1C Dequantizer 117 is depicted as being separate from decompressor 104 , but in other embodiments, dequantizer 117 may be part of decompressor 104 .

[0082] In one embodiment, the decompressor 104 reads the first 48 bits of the bitstream 114 to recover H, W, and C, and then processes one compression unit of the bitstream 114 at a time. The decompressor 104 is aware of both the number of bits used for the index of the mode and the number of elements in the compression unit (W or K, depending on the compression mode used). That is, the bitstream 114 corresponding to the original (sparse) activation map 106 is decompressed by the decoder 115 to form a compression unit 118. The compression unit 118 is deformatted by the deformatter 116 to form a sparse and quantized activation map 119 of size H×W×C. The sparse and quantized activation map 119 can be dequantized by the dequantizer 117 to form a sparse activation map 120 corresponding to the original non-sparse activation map 106. It should be noted that the sparse activation map 120 is a lossy decompression of the corresponding original non-sparse activation map 106.

[0083] Figure 2A and Figure 2B Depicting example embodiments of a method 200 for encoding and a method 210 for decoding activation maps of a deep neural network according to the subject matter disclosed herein. The activation map of each layer of the neural network can be obtained by Figure 2Aand Figure 2B Before compressing the activation map, the compressor 103 and the decompressor 104 (such as Figure 1B and Figure 1C ) is configured to use corresponding compression mode and decompression mode.

[0084] exist Figure 2A In , processing begins at 201. At 202, an activation map of a neural network is sparsified to form a sparsified activation map, wherein the sparsified activation map has an increased number of values ​​equal to zero so that lossless compression performed later will be more efficient. An example sparsification technique is described in conjunction with equation (1). Other sparsification techniques may be used.

[0085] At 203, the sparse activation map is configured to be encoded. In one embodiment, the sparse activation map generated at the layer of the neural network is configured as a tensor of size H×W×C, where H corresponds to the height of the input tensor, W corresponds to the width of the input tensor, and C corresponds to the number of channels of the input tensor. If the values ​​of the sparse activation map are not quantized from floating point numbers to integers, then at 204, the unquantized values ​​of the sparse activation map can be quantized to integer values ​​of any bit width to form a sparse quantized activation map.

[0086] At 205, the sparsified quantized activation map may be formatted into compression units. At 206, each compression unit may be losslessly encoded or compressed independently of the other compression units to form a bitstream. Each compression unit may be losslessly encoded or compressed using any of several compression modes. Example lossless compression modes include, but are not limited to: exponential Golomb coding, sparse exponential Golomb coding, sparse exponential Golomb removed minimum coding, Golomb-Rice coding, exponential mantissa coding, zero coding, fixed length coding, and sparse fixed length coding. Each compression unit may be iteratively compressed for each compression mode available. In one embodiment, when a compression unit is compressed, all available compression modes may be run and the compression mode that has produced the shortest bitstream may be selected. When all compression units of an activation map have been encoded, processing ends at 207 for the activation map. For each activation map of the neural network, Figure 2A Process 200 continues in the same manner.

[0087] exist Figure 2BIn

[0045] , processing begins at 211. At 212, a bitstream is received and the first 48 bits are read to recover the encoded compression units. At 213, each encoded compression unit is decoded to form a decoded compression unit. At 214, each decoded compression unit is deformatted to form a sparse and quantized activation map. At 215, these values ​​(i.e., the sparse and quantized activation maps) are dequantized to form a sparse dequantized activation map. Processing ends at 216 for the activation map. Figure 2B The process 210 continues in the same manner to decompress each activation map of the neural network.

[0088] The following example pseudocode corresponds to method 200 .

[0089]

[0090] The following example pseudocode corresponds to method 210 .

[0091]

[0092]

[0093] Figure 3 Depicts an operational flow 300 of an activation map at a layer L of a neural network in accordance with the subject matter disclosed herein. Operational flow 300 represents both a forward processing direction and a backward processing direction through layer L. That is, operational flow 300 represents an operational flow for training a neural network and for forming inferences from inputs to the neural network. The encoded (sparsed and compressed) representation of the original activation map (not shown) of the neural network is represented as it is read from a memory (such as, Figure 1A 306 , the sparsely packed activation map 305 is converted into a bitstream 301 when it is read out of the memory 102 in the memory. At the decoder 302, the bitstream is decoded to form a compressed unit 303. At the deformatter 304, the compressed unit 303 is deformatted to form a sparse quantized activation map 305. Again, it should be noted that quantization of the activation map can be optional. At the dequantizer 306, the sparsely packed activation map 305 (i.e., the sparse quantized activation map 305) is dequantized to form a sparse activation map 307 for layer L.

[0094] The sparsified activation map 307 is used in layer L of the neural network to compute an output activation map 308 (i.e., sparsified activation map 308). At 309, the output activation map 308 is (optionally) quantized to form a sparsified and quantized activation map 310. At 311, the sparsified and quantized activation map 310 is formatted to form a compressed unit 312. At 313, the compressed unit 312 is encoded to form a bitstream 314, wherein the bitstream 314 is stored in a memory (such as a memory device). Figure 1A In the memory 102).

[0095] As those skilled in the art will recognize, the innovative concepts described herein can be modified and varied over a wide range of applications. Accordingly, the scope of the claimed subject matter should not be limited to any specific exemplary teachings discussed above, but rather is defined by the appended claims.

Claims

1. A system for compressing activation maps of a layer of a neural network, the system comprising: A processor is programmed to initiate executable operations, the executable operations comprising: Use the processor to sparse several non-zero values ​​of the activation map; Configure the sparse activation map as a tensor with a tensor size of H×W×C, where H represents the height of the tensor, W represents the width of the tensor, and C represents the number of channels of the tensor; Formatting the tensor into at least one block having values; and encoding the at least one block independently of other blocks of the tensor using at least one lossless compression mode, Wherein the neural network is configured to provide applications in image classification, image segmentation, word-level prediction, speech recognition, machine translation, or medical imaging, wherein the at least one lossless compression mode is selected from the group consisting of: sparse exponential Golomb coding, sparse exponential Golomb removed minimum coding, and sparse fixed length coding, In sparse exponential Golomb coding, if the value x to be encoded is 0, then the value x is represented by 1 in the output bit stream; otherwise, 0 is added and then x-1 is encoded using exponential Golomb coding. In sparse exponential Golomb coding, before encoding the value, the minimum non-zero value represented by the variable y is determined; the variable y is then encoded using exponential Golomb coding; if the value x to be encoded is 0, it is encoded as 1; otherwise, 0 is added to the bit stream, and then xy is encoded using exponential Golomb coding. In sparse fixed-length coding, if the value x to be encoded is 0, it is encoded as 1, otherwise, 0 is added and a fixed number of bits are used to encode the non-zero value.

2. The system according to claim 1, wherein: The at least one lossless compression mode selected for encoding the at least one block is different from a lossless compression mode selected for encoding another block of the tensor.

3. The system according to claim 1, wherein: The step of encoding the at least one block includes encoding the at least one block independently of other blocks of the tensor using a plurality of lossless compression modes.

4. The system according to claim 1, wherein: The at least one block comprises 48 bits.

5. The system according to claim 1, wherein The executable operations also include outputting the at least one block encoded as a bitstream.

6. The system according to claim 5, wherein: The executable operations also include: decoding the at least one block independently of other blocks of the tensor using at least one decompression mode corresponding to the at least one compression mode used to compress the at least one block; and The at least one block is deformatted into a tensor having a size of H×W×C.

7. The system according to claim 1, wherein: The sparse activation map consists of floating point values, The executable operation further includes: quantizing the floating-point value of the activation map into an integer value.

8. A method for compressing an activation map of a neural network, the method comprising: Use the processor to sparse several non-zero values ​​of the activation map; Configure the activation map as a tensor with a tensor size of H×W×C, where H represents the height of the tensor, W represents the width of the tensor, and C represents the number of channels of the tensor; Formatting the tensor into at least one block having values; and encoding the at least one block independently of other blocks of the tensor using at least one lossless compression mode, Wherein the neural network is configured to provide applications in image classification, image segmentation, word-level prediction, speech recognition, machine translation, or medical imaging, wherein the at least one lossless compression mode is selected from the group consisting of: sparse exponential Golomb coding, sparse exponential Golomb removed minimum coding, and sparse fixed length coding, In sparse exponential Golomb coding, if the value x to be encoded is 0, then the value x is represented by 1 in the output bit stream; otherwise, 0 is added and then x-1 is encoded using exponential Golomb coding. In sparse exponential Golomb coding, before encoding the value, the minimum non-zero value represented by the variable y is determined; the variable y is then encoded using exponential Golomb coding; if the value x to be encoded is 0, it is encoded as 1; otherwise, 0 is added to the bit stream, and then xy is encoded using exponential Golomb coding. In sparse fixed-length coding, if the value x to be encoded is 0, it is encoded as 1, otherwise, 0 is added and a fixed number of bits are used to encode the non-zero value.

9. The method according to claim 8, wherein The at least one lossless compression mode selected for encoding the at least one block is different from a lossless compression mode selected for encoding another block of the tensor.

10. The method according to claim 8, wherein The step of encoding the at least one block further comprises encoding the at least one block independently of other blocks of the tensor using a plurality of lossless compression modes.

11. The method according to claim 8, wherein The at least one block comprises 48 bits.

12. The method according to claim 8, further comprising: The at least one block encoded as a bitstream is output.

13. The method according to claim 12, further comprising: decompressing, using a processor, the at least one block independently of other blocks of the tensor using at least one decompression mode corresponding to the at least one compression mode used to compress the at least one block; as well as The at least one block is deformatted into a tensor having a size of H×W×C.

14. The method according to claim 8, wherein The sparse activation map consists of floating point values, The method also includes quantizing floating point values ​​of the sparsified activation map to integer values.

15. A method for decompressing a sparsified activation map of a neural network, the method comprising: decompressing, using a processor, compressed blocks of a bitstream representing values ​​of the sparsified activation map having values ​​to form at least one decompressed block having values, the at least one decompressed block being decompressed independently from other blocks of the sparsified activation map using at least one decompression mode corresponding to at least one lossless compression mode used to compress the at least one block; and decompressing the at least one decompressed block into portions of a tensor having a size of H×W×C, where H represents a height of the tensor, W represents a width of the tensor, and C represents a number of channels of the tensor, the tensor being the decompressed activation map, Wherein the neural network is configured to provide applications in image classification, image segmentation, word-level prediction, speech recognition, machine translation, or medical imaging, wherein the at least one lossless compression mode is selected from the group consisting of: sparse exponential Golomb coding, sparse exponential Golomb removed minimum coding, and sparse fixed length coding, In sparse exponential Golomb coding, if the value x to be encoded is 0, then the value x is represented by 1 in the output bit stream; otherwise, 0 is added and then x-1 is encoded using exponential Golomb coding. In sparse exponential Golomb coding, before encoding the value, the minimum non-zero value represented by the variable y is determined; the variable y is then encoded using exponential Golomb coding; if the value x to be encoded is 0, it is encoded as 1; otherwise, 0 is added to the bit stream, and then xy is encoded using exponential Golomb coding. In sparse fixed-length coding, if the value x to be encoded is 0, it is encoded as 1, otherwise, 0 is added and a fixed number of bits are used to encode the non-zero value.

16. The method according to claim 15, further comprising: Use the processor to sparse several non-zero values ​​of the activation map; Configure the sparsified activation map as a tensor with a tensor size of H × W × C; Formatting the tensor into at least one block having values; as well as The at least one block is encoded using the at least one lossless compression mode independently of other blocks of the tensor.

17. The method according to claim 16, wherein The at least one lossless compression mode selected for compressing the at least one block is different from a lossless compression mode selected for compressing another block of the tensor, and The step of compressing the at least one block further comprises: compressing the at least one block independently of other blocks of the tensor using a plurality of lossless compression modes.

Images