Compression apparatus and method using parameters of quadtree method

Abstract

An apparatus configured to compress a tensor including a plurality of pixels includes a quadtree generator configured to generate a quadtree searching for non-zero pixels included in the tensor and to extract at least one parameter from the quadtree, a mode selector configured to determine a compression mode based on the at least one parameter, and a bitstream generator configured to generate a bitstream by compressing the tensor based on the compression mode.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of priority to Korean Patent Application No. 10-2020-0054770, filed on May 7, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The present invention relates to an apparatus and method for compressing data, and more specifically, to an apparatus and method for parametrically compressing tensors using a quadtree approach of a neural network. Background Technology

[0004] Neural networks are implemented using computational architectures that model biological brains. Because neural network processors perform massive computations on large amounts of input data, rapid data processing, storage, and retrieval are essential.

[0005] The concept of tensors is used in neural network architectures. A tensor is a generalized representation of a vector, and a tensor can include multiple weights and feature maps. Neural networks can use tensors as the basic processing units for computation, storage, and / or compression. Summary of the Invention

[0006] The present invention provides a neural tensor compressor that effectively compresses tensors, including a neural network processor for the neural tensor compressor and a method for operating the neural network processor.

[0007] The present invention provides a quantification method that takes into account data characteristics.

[0008] According to one aspect of the present invention, an apparatus configured to compress a tensor comprising a plurality of cells is provided, the apparatus comprising: a quadtree generator configured to generate a quadtree for searching non-zero cells included in the tensor and extracting at least one parameter from the quadtree; a mode selector configured to determine a compression mode based on the at least one parameter; and a bitstream generator configured to generate a bitstream by compressing the tensor based on the compression mode.

[0009] According to another aspect of the present invention, a neural network processor is provided, the neural network processor comprising: an arithmetic circuit configured to perform computations on input data using a neural network to generate a tensor comprising a plurality of pixels; and a neural tensor compressor configured to output a bitstream by compressing the tensor, wherein the neural tensor compressor is further configured to: generate a quadtree corresponding to a repetitive space partitioning method to search for non-zero pixels included in the tensor, extract at least one parameter from the quadtree, and determine a compression mode of the tensor based on the at least one parameter.

[0010] According to another aspect of the present invention, a method is provided, the method comprising: receiving a tensor, the tensor being the result of performing repeated arithmetic calculations on a feature map and weights; extracting at least one parameter, the at least one parameter being the result of repeatedly spatially partitioning the tensor to compress zero pixels among a plurality of pixels included in the tensor; determining a compression mode based on the at least one parameter; and outputting a bitstream based on the compression mode.

[0011] According to another aspect of the present invention, a compression method for a neural network processor is provided, the neural network processor being configured to perform computations on feature maps and weights using a neural network, the method comprising: receiving a tensor as a result of the computation, the tensor comprising a plurality of pixels; setting a quantization range of the tensor based on the maximum value of a pixel among the plurality of pixels; selectively quantizing the tensor based on the quantization range; and compressing the quantized tensor. Attached Figure Description

[0012] The embodiments of the inventive concept will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 This is a block diagram of an external memory and a neural network processor according to an exemplary embodiment of the present invention.

[0014] Figure 2 This is a block diagram of a neural tensor compressor according to an exemplary embodiment of the present invention;

[0015] Figure 3A and Figure 3B This is a diagram illustrating an exemplary embodiment of a quadtree-based compression method performed in a quadtree generator according to a concept of the present invention;

[0016] Figure 4 This is a flowchart of a method for determining a compression mode according to an exemplary embodiment of the present invention;

[0017] Figure 5This is a structural diagram of a bitstream according to an exemplary embodiment of the present invention;

[0018] Figure 6 This is a flowchart of an operation method of a neural tensor compressor according to an exemplary embodiment of the present invention;

[0019] Figure 7 This is a flowchart of an operation method of a neural tensor compressor according to an exemplary embodiment of the present invention;

[0020] Figure 8 This is a diagram illustrating a neural network based on an exemplary embodiment of the concept of the present invention;

[0021] Figure 9 This is a diagram illustrating the convolution calculation of a neural network according to an exemplary embodiment of the present invention.

[0022] Figure 10 This is a block diagram of a neural tensor compressor that also includes a quantizer, representing an exemplary embodiment of the present invention.

[0023] Figure 11 This is a diagram illustrating the pixel distribution based on pixel values ​​according to an exemplary embodiment of the present invention.

[0024] Figure 12 This is a flowchart of an operation method of a quantizer according to an exemplary embodiment of the present invention;

[0025] Figure 13 This is a flowchart of an operation method of a neural tensor compressor according to an exemplary embodiment of the present invention;

[0026] Figure 14 This is a flowchart of an operation method of a neural tensor compressor according to an exemplary embodiment of the present invention;

[0027] Figure 15 This is a flowchart of an operation method of a neural tensor compressor according to an exemplary embodiment of the present invention; and

[0028] Figure 16 This is a block diagram of an electronic system according to an exemplary embodiment of the present invention. Detailed Implementation

[0029] In the following description, embodiments of the inventive concept will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals consistently denote the same elements. Figure 1 This is a block diagram of an electronic device 10 including a neural network processor 100 and an external memory 300, according to an exemplary embodiment of the present invention.

[0030] Electronic device 10 can extract useful information by analyzing input data in real time based on neural networks, determine the status based on the extracted information, or control at least one component included in electronic device 10. For example, electronic device 10 can be applied to drones, advanced driver assistance systems (ADAS), robotic devices, smart TVs, smartphones, medical devices, mobile devices, image display devices, measuring devices, Internet of Things (IoT) devices, etc., and can also be used as various other electronic devices.

[0031] Electronic device 10 may include neural network processor 100 and external memory 300. However, embodiments are not limited thereto and may also include at least one intellectual property (IP) block. For example, in addition to neural network processor 100 and external memory 300, electronic device 10 may also include at least one IP block, such as memory storage and sensors, that are required to process the neural network processor 100.

[0032] The neural network processor 100 can generate neural networks, train (or learn) neural networks, perform computations based on received input data, and generate information signals based on the results of operations, or retrain the neural network. The neural network model can include, but is not limited to, various models such as: convolutional neural networks (CNNs) such as GoogleNet, AlexNet, and VGG networks; region (R) CNNs (RCNNs); R proposal networks (RPNs); recurrent neural networks (RNNs); stack-based deep neural networks (DNNs) (SDNNs); state-space (SS) DNNs (SSDNNs); deconvolutional networks; deep belief networks (DBNs); restricted Boltzmann machines (RBMs); fully convolutional networks; long short-term memory (LSTM) networks; and classification networks. The neural network processor 100 can include one or more processors (e.g., central processing units, etc.) for performing computations according to the model of the neural network.

[0033] The neural network processor 100 may include a separate memory as internal memory for storing programs corresponding to the model of the neural network. The neural network processor 100 may also be referred to as a neural network processing device, a neural network integrated circuit, a neural network processing unit (NPU), etc.

[0034] According to an example embodiment, the information signal generated by the neural network processor 100 may include at least one of various types of recognition signals, such as speech recognition signals, target recognition signals, image recognition signals, and biometric information recognition signals.

[0035] In an example embodiment, the neural network processor 100 may receive frame data included in a video stream as input data and generate recognition signals for objects included in an image represented by the received frame data. For example, the neural network processor 100 may generate a face recognition signal based on frame data provided by a camera.

[0036] In an example embodiment, the neural network processor 100 may receive frequency data included in an audio stream as input data and generate a speech recognition signal of speech extracted from the frequency data. However, as another example, the inventive concept is not limited thereto, and the neural network processor 100 may receive various types of input data and generate recognition signals based on the input data.

[0037] Because neural networks generate a lot of data with zero values ​​due to their computational characteristics, neural network processor 100 can compress data by removing data with zero values.

[0038] According to an exemplary embodiment of the present invention, the neural network processor 100 can remove zero cells with a data value of zero from the plurality of cells included in the input feature map data used in convolution calculations, and compress the data by using the data values ​​of non-zero cells that do not have a data value of zero and by using the position information of the non-zero cells. The neural network processor 100 can improve the speed of processing, storing, loading, and retrieving data by compressing the data. Furthermore, the neural network processor 100 can improve the data input / output speed by storing the compressed data in or loading the compressed data from the external memory 300.

[0039] The neural network processor 100 may include a neural tensor decompressor 110, an internal memory 120, an arithmetic circuit 130, and a neural tensor compressor 140.

[0040] The neural tensor decompressor 110 can load data stored in compressed form into the external memory 300 and decompress the data. In an example embodiment, the neural tensor decompressor 110 can decompress data that has already been compressed by the neural tensor compressor 140 in reverse order. For example, the neural tensor compressor 140 can compress data, and the neural tensor decompressor 110 can decompress the compressed data in reverse order of the compression process performed by the neural tensor compressor 140.

[0041] In an example embodiment, the neural tensor decompressor 110 can determine the compression algorithm by referring to the storage address of data already stored in the external memory 300, and can decompress the compressed data based on the determined compression algorithm. The external memory 300 may include storage areas corresponding to compression modes. For example, the external memory 300 may include a first storage area corresponding to a first compression mode, a second storage area corresponding to a second compression mode, and a third storage area corresponding to a third compression mode. The neural tensor decompressor 110 can determine the compression mode from the storage area where the data is loaded (i.e., the storage address of the storage area), and can apply a decoding method according to the compression mode. According to the technical concept of the present invention, because data can be decompressed according to the storage area, the stored bitstream may not include bit information of the compression mode. The decompressed data can be temporarily stored in the internal memory 120.

[0042] The internal memory 120 can temporarily store decompressed data or temporarily store calculation results (e.g., tensors) output from the arithmetic circuit 130.

[0043] Internal memory 120 can temporarily store data in use for fast data processing within neural network processor 100. Because data processing bandwidth is limited between neural network processor 100 and external memory 300 located outside neural network processor 100, neural network processor 100 may include internal memory 120 alone for fast data processing. In example embodiments, internal memory 120 may have faster processing speeds and higher stability than external memory 300, but is not limited thereto. For example, internal memory 120 may include static random access memory (RAM) (SRAM), while external memory 300 may include dynamic RAM (DRAM) and synchronous DRAM (SDRAM).

[0044] Arithmetic circuit 130 can receive input data, including input feature maps and weights, from internal memory 120. Arithmetic circuit 130 can generate tensors by performing convolution calculations using the received input feature maps and weights. Tensors can include feature maps and weights.

[0045] Arithmetic circuit 130 can repeatedly perform arithmetic calculations on the input feature map and weights. Arithmetic circuit 130 can perform multiplication, division, addition, subtraction, and logical operations, and can be referred to as a multiplication accumulator (MAC). Arithmetic circuit 130 can use combinations of arithmetic calculations of the input feature map and weights to solve complex mathematical calculations (e.g., differentiation and integration).

[0046] The neural tensor compressor 140 can load the computation results output from the arithmetic circuit 130 via internal memory 120. The computation results output from the arithmetic circuit 130 can be called tensors. A tensor can be a generalized representation of a vector and can include multiple pixels. In an example embodiment, multiple pixels can form a feature map by arranging them in a matrix, and multiple feature maps can exist depending on the depth of the neural network. The neural network processor 100 can process data in units of tensors. The neural tensor compressor 140 can compress tensors and store the compression result in external memory 300.

[0047] In an example embodiment, the neural tensor compressor 140 can output the generated bitstream to a storage region corresponding to the compression mode. As described above, the external memory 300 may include storage regions corresponding to compression modes. For example, the external memory 300 may include a first storage region corresponding to a first compression mode, a second storage region corresponding to a second compression mode, and a third storage region corresponding to a third compression mode. For example, a bitstream based on the first compression mode can be output to the first storage region, a bitstream based on the second compression mode can be output to the second storage region, and a bitstream based on the third compression mode can be output to the third storage region. According to the technical concept of the present invention, since data can be stored in different spaces in response to the compression mode, the bitstream may not include bit information of the compression mode.

[0048] The neural tensor decompressor 110, arithmetic circuit 130, and neural tensor compressor 140 can be implemented using processing circuitry (e.g., hardware including logic circuitry), or using a combination of hardware and software (e.g., a processor running software that performs compression operations). Specifically, the processing circuitry can be implemented using, but is not limited to, central processing units (CPUs), arithmetic logic units (ALUs) performing arithmetic and logical calculations, shifting, etc., digital signal processors (DSPs), microprocessors, application-specific integrated circuits (ASICs), etc.

[0049] For ease of description, in this specification, a cell in a tensor that has a value of 0 as its data value can be called a zero cell, and a cell in a tensor that has a value other than 0 as its data value can be called a non-zero cell. Due to the computational characteristics of neural networks, the proportion of zero cells in a tensor can be very high.

[0050] Figure 2 This is a block diagram of a neural tensor compressor 140 according to an exemplary embodiment of the present invention.

[0051] The neural tensor compressor 140 may include a quadtree generator 141, a mode selector 143, and a bitstream generator 145.

[0052] Quadtree generator 141 can generate a quadtree for searching non-zero cells included in a tensor. The quadtree can be a data structure that hierarchically divides space into four squares. In an example embodiment, quadtree generator 141 can designate regions where no non-zero cells exist as 0 and regions where non-zero cells exist as 1 by repeatedly dividing the tensor spatially into four regions. All cells included in the region designated as 0 can have a cell value "0" and can be compressed, and cells included in the region designated as 1 can be spatially subdivided again to search for non-zero cells in the next lower level. Quadtree generator 141 can extract maximum bit data representing information about the maximum bit width of non-zero cells, cell position data representing information about the location of non-zero cells, and at least one tensor data by generating a quadtree for the tensor, wherein the at least one tensor data includes non-zero data representing the value of the non-zero cell.

[0053] The quadtree generator 141 may include a non-zero buffer (not shown). The non-zero buffer may buffer non-zero cells included in the input tensor.

[0054] Quadtree generator 141 can extract at least one parameter from the generated quadtree. In an example embodiment, quadtree generator 141 can extract a parameter of quadtree bit number QTB corresponding to the number of bits generated as a compression result of the quadtree, a parameter of non-zero pixel bit width NZW corresponding to the maximum bit width of non-zero pixels, and a parameter of zero count ZC corresponding to the number of zero pixels or zero count included in the tensor.

[0055] In the example embodiment, a quadtree bit count QTB can be generated as the quadtree result, and the quadtree bit count QTB can correspond to the total number of bits representing the position information of non-zero pixels. Additionally, when the pixel with the maximum value among the non-zero pixels is represented in binary, the non-zero pixel bit width NZW (hereinafter also referred to as "non-zero bit width NZW") can correspond to the number of bits from the least significant bit to the highest non-zero bit. Furthermore, the zero count ZC can correspond to the number of zero pixels among multiple pixels. The quadtree generator 141 can output the quadtree bit count QTB, the zero count ZC, and the non-zero bit width NZW.

[0056] Reference Figure 3A and Figure 3B Describe in detail the methods for generating quadtrees from tensors and extracting parameters.

[0057] Mode selector 143 can select a compression mode based on at least one of the quadtree bit count QTB, zero count ZC, and non-zero bit width NZW received from quadtree generator 141. In an example embodiment, mode selector 143 can determine whether a quadtree compression mode is selected based on the quadtree bit count QTB. In an example embodiment, mode selector 143 can select one of a zero-bitmap compression mode and a fixed-length compression mode based on the zero count ZC and non-zero bit width NZW. Mode selector 143 can output a mode signal MODE that commands the compression mode. See also... Figure 4 Describe in detail the method for selecting compression modes based on parameters.

[0058] Bitstream generator 145 can receive a tensor, compress the received tensor based on a mode signal MODE, and output the compressed result as a bitstream BITSTREAM. In an example embodiment, bitstream generator 145 can generate a bitstream BITSTREAM based on the mode signal MODE. The mode signal MODE may include at least one of a signal controlling a quadtree compression mode, a zero-bitmap compression mode signal, and a fixed-length compression mode signal.

[0059] Bitstream generator 145 can compress the provided tensor based on a quadtree using a mode signal MODE that controls the quadtree compression mode. However, the inventive concept is not limited thereto; bitstream generator 145 can also compress the tensor based on the mode signal MODE that controls the quadtree compression mode, using the result used in quadtree generator 141 without regenerating the quadtree.

[0060] Because the Rectified Linear Unit (ReLU) operation, which converts negative numbers to zero, is often applied to convolutional computations in neural networks, many zero pixels may be distributed across the feature map. When non-zero pixels are clustered in a specific region, the compression efficiency is not significantly reduced even if they appear in the feature map. However, when non-zero pixels are scattered across the feature map, the overhead of quadtree-compressed data becomes greater than that of uncompressed data.

[0061] Compared to compression that does not consider zero pixels, the neural tensor compressor 140 and the neural network processor 100 including the neural tensor compressor 140, based on the technical concept of the present invention, improve compression efficiency by adaptively compressing the TENSOR, which is characterized by a small number of pixels with zero values. Furthermore, since the neural tensor compressor 140 based on the technical concept of the present invention can determine the compression mode using parameters extracted when compressing the TENSOR based on a quadtree, the determination of the compression mode can be faster, thus improving the compression speed, and its implementation is readily apparent.

[0062] Furthermore, the neural tensor compressor 140 according to the technical concept of the present invention can take into account the characteristic that multiple pixels included in the tensor are mainly distributed in low pixel value regions, thereby reducing data loss and improving compression efficiency by quantizing only a portion of the multiple pixels.

[0063] Figure 3A and Figure 3B This is a diagram illustrating a quadtree-based compression method executed in a quadtree generator 141 according to an exemplary embodiment of the present invention. Figure 2 Refer to together Figure 3A and Figure 3B .

[0064] Reference Figure 3A A tensor can include multiple feature maps (FMs), and each feature map (FM) can be generated by arranging a set of cells (CGs) in matrix form. For example, a set of four cells arranged in a 2×2 matrix can be called a set of cells (CGs), and a set of four set of cells (CGs) arranged in a 2×2 matrix can be called a feature map (FM), and a set of four feature map (FM) can be called a tensor (TENSOR). However, the inventive concept is not limited to this; in the set of cells (CGs), the cells can be arranged in a 4×4 matrix, and in the feature map (FM), the set of cells (CGs) can be arranged in a 4×4 matrix. The matrix form can vary.

[0065] A cell can be the smallest unit of data representation, and cell values ​​can be represented in hexadecimal. In an example embodiment, because a cell can be represented by two hexadecimal digits, the data represented by a cell can include 8 bits (i.e., 1 byte). However, the inventive concept is not limited to this, and the data represented by a cell can be configured differently according to data representation formats such as 10-bit or 12-bit.

[0066] In a pixel group CG, four adjacent pixels can be arranged in a 2×2 matrix, and the size of the pixel group CG can be 4 bytes. In an example embodiment, the pixel group CG can be referred to as a sub-region of the feature map FM, which is generated when the feature map FM is partitioned by the first quadtree.

[0067] Feature maps (FMs) can be generated by arranging multiple cells in a 4×4 matrix. A tensor can be formed by arranging four feature maps (FMs). However, the inventive concept is not limited to this, and a tensor can be formed by arrays of various numbers of feature maps (FMs).

[0068] According to the example embodiment, the number of cells included in a tensor can depend on the depth of the neural network. For example, when the depth of the neural network is 3, the number of cells can be 64 (i.e., 4). 3 This is the result of having 4 cells on each of the three directional axes (i.e., the width axis, the height axis, and the channel axis). In this case, the feature map FM can be arranged in the form of a 4×4 matrix with 4 cells in width and 4 cells in height, and the number of feature maps FM can be 4, which can be the same as the number of cells on the channel axis.

[0069] For ease of description, a tensor TENSOR is shown to include 64 cells, but the embodiment is not limited to this. In the example embodiment, assuming the depth of the neural network is M, the number of cells included in a tensor TENSOR can be N (i.e., 4). M For example, when the depth of the neural network is 4, the number of cells in a tensor can be 256, while when the depth of the neural network is 5, the number of cells in a tensor can be 1,024.

[0070] Quadtree generator (e.g., Figure 2 The quadtree generator 141 can compress the first tensor TENSOR1 based on a quadtree method. The quadtree generator 141 can search for the first tensor TENSOR1 in the row direction for quadtree compression.

[0071] Quadtree generator 141 can determine that a first compression length LENGTH1 is 1, which is the length of a first tensor TENSOR1 based on quadtree compression. In an example embodiment, quadtree generator 141 may include a non-zero buffer (not shown). The non-zero buffer can buffer non-zero cells included in the input tensor. According to an example embodiment, quadtree generator 141 can search the first tensor TENSOR1 in the row direction for quadtree compression, and as a result, it can be identified that no buffered non-zero cells exist. The minimum length allocated to the tensor to indicate that no non-zero cells exist in the first tensor TENSOR1 (i.e., all cells are zero cells) can be 1 byte. As a result, a first compression length LENGTH1 of 1 can be understood as 1 byte being allocated to indicate that no non-zero cells exist.

[0072] According to the example embodiment, the size of the data allocated to the first compression length LENGTH1 can be 6 bits. The first tensor TENSOR1 can include 64 (i.e., 4...) bits. 3 The first tensor TENSOR1 can have up to 64 (i.e., 2^64) pixels, and each pixel can be 1 byte in size. Therefore, when all pixels are non-zero, the length of the first tensor TENSOR1 can be up to 64 (i.e., 2^64) bytes. 6 )byte.

[0073] As a result, information from 1 byte to 64 bytes can be represented in 6 bits, and can be represented by binary numbers 000000. (2) This indicates the first compression length, LENGTH1. Since 1 byte consists of 8 bits, the remaining two bits can be zero-padded.

[0074] Because the first tensor TENSOR1 does not contain any non-zero pixels, the first non-zero bit width NZW1 and the first non-zero value NZV1 can both be 0. The first non-zero bit width NZW1 is the bit length of the largest non-zero pixel included in the first tensor TENSOR1, and the first non-zero value NZV1 is the value of the non-zero pixel. Since the information where the first compression length LENGTH1 is 1 corresponds to the information where the first tensor TENSOR1 does not contain non-zero pixels, the first non-zero bit width NZW1 and the bits of the first non-zero value NZV1, which represents the value of the non-zero pixel, can be excluded from the compressed data.

[0075] Reference Figure 3B The second tensor TENSOR2 can include four feature maps (e.g., FM1, FM2, FM3, and FM4). A quadtree generator (e.g., Figure 2The quadtree generator (141) can search the second tensor TENSOR2 in the row direction for quadtree compression. A non-zero buffer can buffer non-zero cells in the second tensor TENSOR2. As a result of this buffering, the maximum value of the non-zero cells included in the second tensor TENSOR2 is hexadecimal "0E" (or binary 00001110). (2) Therefore, the non-zero bit width NZW can be 4.

[0076] In the example embodiment, the pixel values ​​of the first feature map FM1, the second feature map FM2, and the fourth feature map FM4 in the second tensor TENSOR2 can all be 0, and only the third feature map FM3 can have non-zero pixels (quadtree: 0010). (2) The first feature map FM1, the second feature map FM2, and the fourth feature map FM4 can all be compressed into 1 byte (00000000). (2) ),like Figure 3A As shown.

[0077] In an example embodiment, a quadtree can be applied to a third feature map FM3 that includes non-zero pixels. As a result of applying the quadtree, the third feature map FM3 can be divided into four equal parts relative to its center (e.g., upper left, upper right, lower left, and lower right), and these four equal parts can be a first pixel group CG1, a second pixel group CG2, a third pixel group CG3, and a fourth pixel group CG4, respectively. Non-zero pixels may not exist in the first pixel group CG1 and the second pixel group CG2, while non-zero pixels may exist in the third pixel group CG3 and the fourth pixel group CG4 (quadtree: 0011). (2) Compression can terminate in the first pixel group CG1 and the second pixel group CG2 where no non-zero pixels exist, and the quadtree can be reapplied to the third pixel group CG3 (inclusive of "0E") and the fourth pixel group CG4 (inclusive of "06") where non-zero pixels exist. Only the top-left pixel of the third pixel group CG3 can have non-zero pixels (quadtree: 1000). (2) ), and only the bottom left pixel of the fourth pixel group CG4 can have a non-zero value (quadtree: 0010). (2) Since the non-zero pixels of the third pixel group CG3 and the fourth pixel group CG4 have been reached, quadtree compression can be terminated. Because the non-zero value of the third pixel group CG3 is hexadecimal 0E (i.e., 0x0E), 0E can be converted to binary 1110. (2) Because the non-zero value of the fourth pixel CG4 is hexadecimal 06 (i.e., 0x06), 06 can be converted to binary 0110. (2) .

[0078] The quadtree generator 141 according to the technical concept of the present invention can generate a quadtree with multiple parameters as the result of quadtree compression. These multiple parameters may include: compression length LENGTH, which is the compression length based on the quadtree tensor; non-zero bit width NZW, which is the bit width of the non-zero pixels with the maximum value included in the tensor; zero count ZC, which is the number of zero pixels among the multiple pixels included in the tensor; non-zero value NZV, which is the non-zero pixel value; and quadtree bit count QTB, which is the number of bits generated as the result of quadtree compression, but the present invention is not limited thereto.

[0079] When applying quadtree-based compression to a tensor TENSOR, the quadtree generator 141 can extract at least one parameter.

[0080] In the example embodiment, the quadtree generator 141 can identify that the sum of the number of bits generated in the quadtree compression is 33 bits. 33 bits cannot be represented by 4 bytes of space; therefore, the quadtree generator 141 can determine that the second compression length LENGTH2 is 5 bytes (i.e., 000100). (2) Since 5 bytes include 40 bits, the remaining 7 bits beyond the 33 bits can be zero-padded.

[0081] In an example embodiment, the quadtree generator 141 can determine that the second non-zero bit width NZW2 has 4 bits based on the cells buffered in the non-zero buffer.

[0082] In the example embodiment, since four 4-bit units have already been generated as a result of quadtree compression, the quadtree generator 141 can determine that the quadtree bit count QTB is 16. The quadtree bit count QTB can be in the range of 12 to 84.

[0083] In an example embodiment, the quadtree generator 141 can extract the zero count ZC. The zero count ZC can be extracted by subtracting the number of non-zero cells buffered in the non-zero buffer from the number of cells included in the tensor TENSOR.

[0084] However, the inventive concept is not limited to non-zero buffers, and the zero count ZC can be calculated in reverse based on the bits generated as a result of quadtree compression. For example, among the bits already generated as a result of quadtree compression, the topmost bit could be 0010. (2) Because the top-level bits correspond to whether a non-zero value is present in the feature map FM, a 0 can be interpreted as indicating that all 16 pixels in a feature map FM are zero pixels. For example, three "0"s can represent 48 (i.e., 16 × 3) zero counts ZC. Similarly, because the second-level bits include 0011... (2)Since there are two "0"s and the quadtree has already been applied once in the second level, it can be understood that each "0" represents four zero pixels in a tuple. For example, two "0"s can represent a zero count ZC of 8 (i.e., 4 × 2). Similarly, because the quadtree has already been applied twice in the third level, and the number of "0"s in the third level is 6, it can be understood that six "0"s represent 6 zero pixels. As a result, the total zero count ZC can be calculated in reverse as 62 (i.e., 48 + 8 + 6).

[0085] Figure 4 This is a flowchart of a method for determining a compression mode according to an exemplary embodiment of the present invention. (See also...) Figure 2 and Figure 4 .

[0086] When compressing a tensor using a quadtree method, a quadtree with at least one parameter can be generated. The quadtree parameter may include: a non-zero bit width NZW, which is the bit width of the non-zero pixels with the maximum value included in the tensor TENSOR; a zero count ZC, which is the number of zero pixels among the multiple pixels included in the tensor TENSOR; and a quadtree bit count QTB, which is the number of bits generated as a result of performing quadtree compression, but is not limited to these.

[0087] Reference Figure 4 Pattern selector (e.g., Figure 2 The mode selector 143 in the tensor compares the number of quadtree bits QTB with the number of pixels N included in the tensor TENSOR to determine whether the number of quadtree bits QTB is less than or equal to the number of pixels N (S11). The number of pixels N can depend on the depth M of the neural network and can satisfy N = 4. M .

[0088] When the number of bits in the quadtree QTB is less than or equal to the number of pixels N (S11, yes), the tensor TENSOR can be compressed based on the quadtree method (S12).

[0089] When the number of bits in the quadtree QTB is greater than the number of pixels N (S11, No), the mode selector 143 can compare the product of the non-zero bit width NZW and the zero count ZC with the number of pixels N to determine whether the product is greater than the number of pixels N (S13).

[0090] When the product of the non-zero bit width NZW and the zero count ZC is greater than the number of pixels N (S13, yes), the tensor TENSOR can be compressed based on the zero-bitmap method (S14). The zero-bitmap method can be a method that treats non-zero pixels as 1, zero pixels as 0, and uses a prefix table that includes positional information about non-zero and zero pixels for compression.

[0091] When the product of the non-zero bit width NZW and the zero count ZC is less than or equal to the number of cells N (S13, no), the tensor TENSOR can be compressed based on a fixed-length method (S15). The fixed-length method can be a method that fixes the length of a cell to the bit width of the cell with the largest cell value among a plurality of cells without using a prefix table.

[0092] According to the technical concept of this invention, it is necessary to compare the product of the non-zero bit width NZW and the zero count ZC with the number of pixels N. The following formula can be used as a reference.

[0093] (N+NZW×NZC)>(NZW×N) (Equation 1)

[0094] The terms on the left correspond to the zero-bitmap method, and the terms on the right correspond to the fixed-length method. The direction of the inequality sign indicates that the fixed-length method has better compression efficiency.

[0095] Equation 1 can be summarized as follows.

[0096] N+NZW×(N-ZC)>NZW×N (Equation 2)

[0097] Equation 2 can be summarized as follows.

[0098] N>NZW×ZC (Equation 3)

[0099] In summary, according to Equation 3, the mode selector 143 can compare the product of the non-zero bit width NZW and the zero count ZC with the number of pixels N and determine that the fixed-length method has better compression efficiency when the number of pixels N is greater than the product.

[0100] Refer to later Figure 5 This describes the bitstream generated as a result of compression applied to the quadtree method, zero-bitmap method, and fixed-length method.

[0101] Figure 5 This is a structural diagram of a bitstream according to an exemplary embodiment of the present invention. See also: Figure 2 and Figure 5 .

[0102] Reference Figure 5 In (a), the bitstream generator 145 can generate zero pixels as a bitstream of size 8 bits. Since a pixel has 2 bits of hexadecimal information, the size of a pixel can be 8 bits. The pixel value of a zero pixel can be 0, and when decimal 0 is converted to hexadecimal, it can be represented as 0x00.

[0103] Reference Figure 5In (b), the bitstream generator 145 can generate an uncompressed tensor as a bitstream of size 8 × (N+1). Since the tensor contains N cells and each cell is 8 bits in size, an uncompressed bitstream of 8 * N cells can be generated. Furthermore, the 8 bits representing the maximum value of a cell can be placed at the beginning of the bitstream. For example, when the maximum value of a cell is 2, the maximum value can be represented as hexadecimal 0x02 or binary 00000010. (2) And it can be located in the first 8 bits of the bit stream.

[0104] Reference Figure 5 In (c), bitstream generator 145 can generate a bitstream corresponding to the quadtree method selected by mode selector 143. According to the example embodiment, when the quadtree bit count QTB is equal to or less than the pixel count N, it can generate... Figure 5 The bitstream of (c) in the middle.

[0105] Bitstream generator 145 can place bits corresponding to the number of pixels N included in the tensor TENSOR in the first 6 bits of the bitstream. For example, when N equals 64, the binary number 111111 can be represented in a 6-bit space. (2) (i.e., 63) (10) ).

[0106] The bitstream generator 145 can write a 1-bit identifier in the 1-bit space at the next position to indicate whether the quadtree has been applied.

[0107] Bitstream generator 145 can place the bits used for the non-zero bit width NZW in the next position. Since a cell has 8 bits of information, the maximum non-zero bit width NZW can also be 8 bits. When the non-zero bit width NZW is 1, 000 can be placed bit by bit. (2) (That is, 0 = NZW-1) is written into the bit space, while when the non-zero bit width NZW is 8, 111 is written in bits. (2) (i.e., 7 = NZW-1).

[0108] Bitstream generator 145 can write bits for the quadtree bit count QTB in the bit space at the next position. In the example embodiment, when N is 64, the total number of bits generated as a result of quadtree compression, resulting in the quadtree bit count QTB, can be distributed in the range of 12 to 84. Because Figure 5 (c) is determined such that when the number of quadtree bits QTB is less than or equal to a number of pixels (i.e., N is 64), quadtree compression is performed by mode selector 143, and the number of quadtree bits QTB can have a distribution range of 12 to 64.

[0109] Bitstream generator 145 can write bits for k non-zero values ​​NZV in the bit space of the next position. For example, bitstream generator 145 can write bits for NZV 0 to NZV k-1. Because the non-zero bit width NZW includes the cell with the maximum value, each non-zero value NZV can be represented by using only the number of bits corresponding to the non-zero bit width NZW.

[0110] The bitstream generator 145 can perform zero padding to fit byte units into the bit space at the next position. Because zero padding is required to fit byte units, the number of bits to be zero-padding can be between 0 and 7.

[0111] Reference Figure 5 In (d), bitstream generator 145 can generate a bitstream corresponding to the quadtree method selected by mode selector 143. According to the example embodiment, when the quadtree bit count QTB and the product of the non-zero bit width and the zero count (NZW×ZC) are both greater than the number of pixels N, a bitstream can be generated. Figure 5 The bitstream of (d) in the text. (The part related to...) is omitted. Figure 5 The description of bit space repetition in (c) is as follows.

[0112] and Figure 5 Compared to the bitstream shown in (c) in the middle, Figure 5 The bitstream shown in (d) illustrates the difference in bitstream length due to the inclusion of bits written for the prefix table instead of the quadtree bit count (QTB) and bits that indicate whether flexible length has been applied. Additionally, compared to... Figure 5 Compared to the bitstream shown in (c) in the middle, Figure 5 The difference in the bitstream shown in (d) is that the bits that can identify whether a quadtree has been applied are written as 0. The prefix table can include 64 bits of information by treating non-zero pixels as 1 and zero pixels as 0, where the 64 bits of information contain positional information about non-zero and zero pixels.

[0113] According to the example embodiment, when the quadtree bit count (QTB) exceeds 64 bits, since the quadtree bit count (QTB) includes up to 84 bits, the zero-bitmap method, which always includes a prefix table of 64 bits, will always be more efficient in terms of compression.

[0114] Reference Figure 5 In (e), bitstream generator 145 can generate a bitstream corresponding to the fixed-length method selected by mode selector 143. According to an example embodiment, when the quadtree bit count QTB is greater than the number of pixels N, but the product of the non-zero bit width and the zero count (NZW×ZC) is less than or equal to the number of pixels N, a bitstream can be generated. Figure 5The bitstream of (e) in the text. (The part related to...) is omitted. Figure 5 (c) and Figure 5 The description of bit space repetition in (d) is as follows.

[0115] and Figure 5 Compared to the bitstream shown in (d) in the image, Figure 5 The bitstream shown in (e) can be seen as a result of the following: the prefix table and the non-zero value NZV are not used, but fixed-length values ​​are used, and a bit of 1 is used to indicate whether a flexible length has been applied.

[0116] According to the example embodiment, when N is 64 and most of them are non-zero pixels, since there is no need to provide location information, the maximum value of the pixels among the 64 pixels can be determined as the non-zero bit width NZW, and fixed-length compression can be performed. When the product of the non-zero bit width and the zero count (NZW × ZC) is less than or equal to the number of pixels N, the fixed-length method is better because the 64 bits of the prefix table incur overhead.

[0117] Figure 6 This is a flowchart of the operation method of a neural tensor compressor 140 according to an exemplary embodiment of the present invention.

[0118] The neural tensor compressor 140 can receive a tensor that is the result of repeated arithmetic operations on the feature map and weights (S110).

[0119] The neural tensor compressor 140 can extract at least one parameter as a result of quadtree compression, in which the zero pixel among multiple pixels is compressed by spatially partitioning the tensor (S120).

[0120] The neural tensor compressor 140 can determine the compression mode based on at least one parameter (S130).

[0121] The neural tensor compressor 140 can output a bitstream based on a compression mode (S140).

[0122] Figure 7 This is a flowchart of the operation method of a neural tensor compressor 140 according to an exemplary embodiment of the present invention.

[0123] After operation S110, the neural tensor compressor 140 can extract the quadtree bit count QTB generated as a result of quadtree compression (S121).

[0124] The neural tensor compressor 140 can extract a non-zero bit width NZW (S122) corresponding to the bit width of the non-zero pixel with the largest pixel value among multiple pixels.

[0125] The neural tensor compressor 140 can extract a zero count ZC (S123) corresponding to the number of pixels with zero values ​​among a plurality of pixels. Next, the process can proceed to operation S130.

[0126] Figure 8 This is a diagram illustrating a convolutional neural network structure as an example of a neural network architecture. Figure 9 This is a diagram illustrating the convolution operation of a neural network NN according to an example embodiment of the present invention.

[0127] Reference Figure 8 The neural network NN may include a first layer L1 to an nth layer Ln. Each of the first layer L1 to the nth layer Ln may include a linear layer or a non-linear layer, and in embodiments, at least one linear layer and at least one non-linear layer may be combined and may be referred to as a single layer. For example, a linear layer may include a convolutional layer and a fully connected layer, and a non-linear layer may include a pooling layer and an activation layer.

[0128] For example, the first layer L1 can be a convolutional layer, the second layer L2 can be a pooling layer, and the nth layer Ln can be a fully connected layer that serves as the output layer. The neural network NN can also include activation layers, and may also include layers that perform other types of operations.

[0129] Layers L1 through Ln can each receive input data (e.g., image frames) or feature maps generated in the previous layer as input feature maps, and generate output feature maps or recognition signals REC by calculating the input feature maps. In this case, the feature maps can be referred to as data representing various characteristics of the input data. The first feature maps FM1 through Ln can have shapes such as two-dimensional matrices or three-dimensional matrices (or tensors). The first feature maps FM1 through Ln can have width W (or number of columns), height H (or number of rows), and depth D corresponding to the x-axis, y-axis, and z-axis, respectively. In this case, the depth D can be referred to as the number of channels.

[0130] The first layer L1 generates the second feature map FM2 by convolving the first feature map FM1 with the weight map WM. The weight map WM filters the first feature map FM1 and can also be referred to as a filter or kernel. The depth D (i.e., number of channels) of the weight map WM can be the same as the depth D (i.e., number of channels) of the first feature map FM1, and the same channels of the weight map WM and the first feature map FM1 can be convolved with each other. The weight map WM can move as a sliding window by traversing the first feature map FM1. The amount of movement can be referred to as the "stride length" or "step". During each movement, the weight values ​​included in the weight map WM can be multiplied by all pixel data in the region of the first feature map FM1 that overlaps with the weight map WM, and the results of the multiplications are summed. The data in the region of the first feature map FM1 that overlaps with each weight value included in the weight map WM can be referred to as the extracted data. When the first feature map FM1 and the weight map WM are convolved with each other, one channel of the second feature map FM2 can be generated. Although... Figure 8 The diagram shows a weight map WM, but multiple channels of a second feature map FM2 can be generated by fully convolving multiple weight maps WM with a first feature map FM1. For example, the number of channels in the second feature map FM2 can correspond to the number of weight maps WM.

[0131] The second layer (L2) can generate a third feature map (FM3) by using pooling to change the spatial size of the second feature map (FM2). Pooling can be referred to as sampling or downsampling. A two-dimensional pooling window (PW) can be moved on the second feature map (FM2) in units of the size of the pooling window (PW), and the maximum value (or average value of the image data) can be selected from the image data in the region overlapping with the pooling window (PW). Therefore, a third feature map (FM3) with a changed spatial size can be generated from the second feature map (FM2). The number of channels in the third feature map (FM3) and the second feature map (FM2) can be the same.

[0132] The nth layer Ln can classify the category CL of the input data by combining the features of the nth feature map FMn. Additionally, the nth layer Ln can generate a recognition signal REC corresponding to the category CL. In an embodiment, the input data may correspond to frame data included in a video stream, and the nth layer Ln can identify objects and generate a recognition signal REC corresponding to the identified objects by extracting the category CL corresponding to the objects included in the image indicated by the frame data based on the nth feature map FMn provided by the previous layer.

[0133] Reference Figure 9The input feature map 201 can include D channels, and each channel's input feature map can have a size of H rows and W columns (D, H, and W are natural numbers). Each kernel 202 can have a size of R rows and S columns (R and S are natural numbers), and kernel 202 can include a number of channels corresponding to the number (or depth) D of the input feature map 201. The output feature map 203 can be generated by performing a three-dimensional convolution operation between the input feature map 201 and kernel 202, and according to the convolution operation, the output feature map 203 can include Y channels (Y is a natural number).

[0134] You can refer to Figure 9 The process of generating an output feature map 203 by applying a convolution operation between an input feature map 201 and a kernel 202 is described, and reference can be performed between the input feature map 201 of all channels and the kernel 202 of all channels. Figure 9 The two-dimensional convolution operation described is used to generate the output feature map 203 for all channels.

[0135] Figure 10 This is a block diagram of a neural tensor compressor 240 that further includes a quantizer 247, according to an exemplary embodiment of the present invention.

[0136] The neural tensor compressor 240 may include a quadtree generator 241, a mode selector 243, a bitstream generator 245, and a quantizer 247. The quantizer 247 may receive a tensor TENSOR, quantize the received tensor TENSOR, and provide the quantized tensor TENSOR_Q to the quadtree generator 241. The quadtree generator 241 may receive the quantized tensor TENSOR_Q and generate a quadtree by applying a quadtree method to the quantized tensor TENSOR_Q. Figure 10 The quadtree generator 241, mode selector 243, and bitstream generator 245 in the code correspond to... Figure 2 The quadtree generator 141, mode selector 143, and bitstream generator 145 are omitted from repeated descriptions.

[0137] Because ReLU computation is often used for convolution computations in neural networks (NNs), many zero pixels can be distributed across the feature map and the tensor containing the feature map. When many zero pixels exist, it can be assumed that most pixel values ​​included in the tensor are relatively close to 0 together.

[0138] According to the technical concept of the present invention, quantizer 247 can perform non-uniform quantization. In an example embodiment, quantizer 247 can perform non-uniform quantization by using the property that a tensor has many zero pixels.

[0139] For non-uniform quantization, Equation 4 can be applied.

[0140]

[0141] Referring to Equation 4, the non-uniform quantization value Q out This can correspond to the value obtained by dividing the quantization step Qstep by 2 and adding it to the input, rounding the decimal result down by the quantization step Qstep, and adding the offset to the division result. However, the inventive concept is not limited to this, and various non-uniform quantization methods that take into account tensor properties can be applied.

[0142] The quantizer 247, based on the technical concept of the present invention, can selectively perform quantization. In an example embodiment, when high precision is required for the neural network processor 100, the quantizer 247 can pass the tensor TENSOR to the quadtree generator 241 without performing quantization. For example, when quantization is not performed, the quantized tensor TENSOR_Q can be the same as the tensor TENSOR. In an example embodiment, when the neural network processor 100 operates in a low-power mode or is mounted on a relatively inexpensive electronic device, the quantizer 247 can perform non-uniform quantization.

[0143] When performing uniform quantization, data loss may occur in low-pixel values, which can have a relatively sensitive impact on accuracy. When performing non-uniform quantization, the neural tensor compressor 240 according to the technical concept of the present invention can avoid quantizing pixels with low-pixel values ​​that are related to data processing accuracy, and can greatly reduce the data compression rate by quantizing pixels with high-pixel values ​​and reducing data loss.

[0144] Figure 11 This is a graph showing the pixel distribution based on pixel values ​​according to an exemplary embodiment of the present invention. The horizontal axis of the graph can represent pixel values, while the vertical axis can represent the number of pixels.

[0145] Reference Figure 11 Assume that each of the multiple pixels included in the tensor has a pixel value of at most 255. The quantization step size Qstep can be categorized by dividing 255 or the maximum pixel value by 4. For example, the quantization range can be divided into a first range (where pixel values ​​are 0 to 63) with a quantization step size 1 (i.e., Qstep is 1) and an offset of 0, a second range (where pixel values ​​are 64 to 127) with a quantization step size 2 (i.e., Qstep is 2) and an offset of 32, and a third range (where pixel values ​​are 128 to 255) with a quantization step size 4 (i.e., Qstep is 4) and an offset of 64. However, the embodiments are not limited to this.

[0146] According to the example embodiment, since the quantization step size is 1 (or Qstep is 1) and the offset is 0 for the first range, the non-uniform quantization value can include values ​​from 0 to 63.

[0147] According to the example embodiment, since the quantization step size is 2 (or Qstep is 2) and the offset is 32 for the second range, the non-uniform quantization value can include values ​​from 64 to 95.

[0148] According to the example embodiment, since the quantization step size is 4 (or Qstep is 4) and the offset is 64 for the third range, the non-uniform quantization value can include values ​​from 96 to 127.

[0149] A quantizer based on the technical concept of the present invention (e.g., Figure 10 The quantizer 247 in the code can quantize an 8-bit cell value with a value between 0 and 255 into a 7-bit cell value with a value between 0 and 127. Because non-uniform quantization ignores the quantization of a range of cells with low cell values, it reduces data loss compared to uniform quantization.

[0150] Figure 12 This is a flowchart illustrating the operation method of a quantizer 247 according to an exemplary embodiment of the present invention. See also: Figure 10 .

[0151] Quantizer 247 can receive a tensor (S21). The tensor can include 64 8-bit pixels.

[0152] Quantizer 247 can divide a range of 8-bit cell values ​​(S22). Quantizer 247 can identify whether a cell value is less than 64 (S22). When the cell value is less than 64 (S22, yes), the cell is not quantized and can be provided to quadtree generator 241. When the cell value is 64 or greater (S22, no), the process proceeds to operation S23.

[0153] The quantizer 247 can identify whether the cell value is 128 or greater (S23).

[0154] When the cell value is less than 128 (S23, No), the quantizer 247 can divide the cell value by 2 (S24) and add the offset 32 ​​to the divided cell value (S25).

[0155] When the cell value is 128 or greater (S23, yes), the quantizer 247 can divide the cell value by 4 (S26) and add the offset 64 to the divided cell value (S27).

[0156] Figure 13This is a flowchart illustrating the operation of a neural tensor compressor 240 according to an exemplary embodiment of the present invention. (See also...) Figure 10 and Figure 13 .

[0157] The neural tensor compressor 240 can receive a tensor comprising multiple pixels as a result of computation of feature maps and weights using a neural network (S210).

[0158] The neural tensor compressor 240 can set the quantization range of the tensor based on the cell with the maximum value among multiple cells (S220).

[0159] The neural tensor compressor 240 can selectively quantize tensors based on quantization ranges (S230). In an example embodiment, the neural tensor compressor 240 can quantize pixels included in a second range but not pixels included in a first range. For example, the first range may include low-pixel-value regions, and the second range may include relatively high-pixel-value regions, but is not limited thereto.

[0160] The neural tensor compressor 240 can extract multiple parameters by applying a quadtree data structure to the quantized tensor TENSOR_Q (S240).

[0161] The neural tensor compressor 240 can determine whether a quadtree-based bitstream has been generated based on multiple parameters (e.g., NZW, ZC, and QTB) (S250).

[0162] Figure 14 This is a flowchart of the operation method of a neural tensor compressor 240 according to an exemplary embodiment of the present invention.

[0163] After operation S210, quantizer 247 can calculate the first value obtained by dividing the maximum value of the cell by 4 and rounding down to decimal (S221).

[0164] The quantizer 247 can calculate a second value obtained by dividing the maximum value of the cell by 2 and rounding down to the decimal level (S222).

[0165] Quantizer 247 can set the first quantization range to be greater than or equal to 0 and less than or equal to the first value (S223).

[0166] Quantizer 247 can set the second quantization range to be greater than or equal to the first value and less than or equal to the second value (S224).

[0167] The quantizer 247 can set the third quantization range to be greater than or equal to the second value and less than or equal to the maximum cell value (S225).

[0168] Quantizer 247 can classify multiple pixels into quantization ranges corresponding to pixel values ​​(S226).

[0169] Next, the process can proceed to operation S230.

[0170] Figure 15 This is a flowchart of the operation method of a neural tensor compressor 240 according to an exemplary embodiment of the present invention.

[0171] After operation S220, quantizer 247 can divide the pixel value of the pixel included in the second quantization range by 2, and add the division result to the first offset to prevent overlap with the first quantization range (S231).

[0172] Quantizer 247 can divide the pixel value of a pixel included in the third quantization range by 4, and add the division result to the second offset to prevent overlap with the second quantization range (S232).

[0173] Next, the process can proceed to operation S240.

[0174] Figure 16 This is a block diagram of an electronic system 1000 according to an exemplary embodiment of the present invention.

[0175] Reference Figure 16 The electronic system 1000 can analyze input data in real time based on a neural network (NN), extract effective information, determine the status based on the extracted information, and control the components of the electronic system 1000. For example, the electronic system 1000 can be applied to drones, ADAS, robotic devices, smart TVs, smartphones, medical devices, mobile devices, image display devices, measuring devices, Internet of Things (IoT) devices, etc., and additionally, the electronic system 1000 can be installed on one of various electronic devices.

[0176] In addition to the functional units illustrated, the electronic system 1000 may also include various types of intellectual property (IP) blocks. As used herein, intellectual property (IP) or IP blocks may also be referred to as intellectual property cores and may be used to represent independent discrete units that provide macro-functionality to the system. Those skilled in the art will understand that the disclosed intellectual property or intellectual property cores are physically implemented using electronic (or optical) circuits formed by semiconductor-based manufacturing techniques or other manufacturing techniques, such as logic circuits, discrete components, microprocessors, hardwired circuits, storage elements, wiring connections, buses, communication links, etc. For example, IP blocks may include processing units, multiple cores included in processing units, multi-format codecs (MFC), video modules (e.g., camera interfaces, Joint Photographic Experts Group (JPEG)) processors, video processors, mixers, etc.), 3D graphics cores, audio systems, drivers, display drivers, volatile memory, non-volatile memory, memory controllers, input and output interface blocks, cache memory, etc.

[0177] Technologies used for connecting IP can include system bus-based connectivity methods. For example, the Advanced Microcontroller Bus Architecture (AMBA) protocol for Advanced RISC Machines (ARM) can be used as a standard bus specification. Bus types for the AMBA protocol can include Advanced High Performance Bus (AHB), Advanced Peripheral Bus (APB), Advanced Extensible Interface (AXI), AXI4, and AXI Coherence Extension (ACE). Among these bus types, AXI can be an interface protocol between IPs and can provide features such as multiple outstanding addresses and data interleaving. Additionally, other types of protocols can be used, such as the SONIC uNetwork, IBM CoreConnect, and the open core protocol of OCP-IP.

[0178] The neural network processor (or neural processing unit (NPU)) 1100 can receive various types of input data via a system bus and generate information signals based on the input data. For example, the NPU 1100 can generate information signals by performing neural network operations on the input data, and the neural network operations can include convolution operations.

[0179] Memory 1300 can be a storage area for storing data, and can store, for example, an operating system (OS), various programs, and various types of data. Memory 1300 can be dynamic random access memory (RAM) (DRAM), but is not limited to this. Memory 1300 can include volatile memory. Volatile memory can include DRAM, static RAM (SRAM), synchronous DRAM (SDRAM), phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FeRAM), etc.

[0180] The CPU 1500 can control all operations of the electronic system 1000. For example, the CPU 1500 may include a central processing unit (CPU). The CPU 1500 may include one processor core (or a single core) or multiple processor cores (or multiple cores). The CPU 1500 can process or run programs and / or data stored in the memory 1300. For example, the CPU 1500 can control the functions of the electronic system 1000 by running programs stored in the memory 1300.

[0181] Storage device 1700 can be a storage location for storing data, and can store various programs and various types of data. Storage device 1700 may include non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable PROM (EEPROM), flash memory, phase-change random access memory (RAM) (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Furthermore, in embodiments, storage device 1700 may include at least one of hard disk drive (HDD), solid-state drive (SSD), compact flash memory (CF) storage device, secure digital storage (SD) device, micro secure digital storage device, extreme digital (xD), and memory stick.

[0182] Sensor 1900 can collect information about the surroundings of electronic system 1000. Sensor 1900 can sense or receive image signals from outside electronic system 1000 and convert the sensed or received image signals into image data (i.e., image frames). For this purpose, sensor 1900 can include at least one of various types of sensing devices such as imaging devices, image sensors, light detection and ranging (LIDAR) sensors, ultrasonic sensors, and infrared sensors, or can receive sensing signals from sensing devices. In embodiments, sensor 1900 can provide image frames to neural network processor 1100. For example, sensor 1900 can include an image sensor that generates a video stream by capturing images of the external environment of electronic system 1000 and sequentially provides consecutive image frames of the video stream to neural network processor 1100.

[0183] Although the inventive concept has been specifically shown and described with reference to embodiments thereof, it should be understood that various changes in form and detail may be made without departing from the spirit and scope of the appended claims.

Claims

1. An apparatus configured to compress a tensor comprising a plurality of pixels, the apparatus comprising: A quadtree generator is configured to generate a quadtree that searches for non-zero pixels in the tensor and extract at least one parameter from the quadtree. A mode selector configured to determine a compression mode based on the at least one parameter; as well as A bitstream generator configured to generate a bitstream by compressing the tensor based on the compression mode.

2. The device according to claim 1, wherein, The at least one parameter includes: The first parameter is generated as a result of the quadtree and corresponds to the total number of bits representing the location information of the non-zero cell. The second parameter, in response to the pixel having the maximum value among the non-zero pixels being represented in binary, corresponds to the number of bits from the least significant bit to the most significant non-zero bit; and The third parameter corresponds to the number of zero pixels among the plurality of pixels.

3. The device according to claim 2, wherein, The mode selector is further configured to select a first compression mode as the compression mode in response to the value of the first parameter being less than or equal to the number of pixels included in the tensor, wherein the tensor is compressed using a quadtree method in the first compression mode.

4. The device according to claim 2, wherein, The mode selector is further configured to: in response to the value of the first parameter exceeding the number of pixels included in the tensor and the product of the value of the second parameter and the value of the third parameter exceeding the number of pixels, select a second compression mode that compresses the tensor using a zero-bitmap method as the compression mode, wherein, in the zero-bitmap method, non-zero pixels are treated as 1 and zero pixels are treated as 0.

5. The device according to claim 2, wherein, The mode selector is further configured to: in response to the value of the first parameter exceeding the number of pixels included in the tensor and the product of the value of the second parameter and the value of the third parameter being less than or equal to the number of pixels, select a third compression mode that compresses the tensor using a fixed-length method as the compression mode, wherein, in the fixed-length method, the tensor is compressed based on the bit width of the pixel with the maximum value among the plurality of pixels.

6. The device according to claim 1, wherein, The tensor includes 4 M There are pixels, where M is a natural number.

7. The device according to claim 1, wherein, The tensor includes at least one of a feature map and weights.

8. The device according to claim 1, wherein, The bitstream generator is further configured to output the bitstream to at least one of a plurality of storage regions corresponding to the compression mode.

9. A neural network processor, comprising: An arithmetic circuit configured to generate a tensor comprising multiple pixels by performing computations on input data using a neural network; as well as A neural tensor compressor, configured to output a bitstream by compressing the tensor. The neural tensor compressor is further configured to: generate a quadtree corresponding to the repetitive spatial partitioning method to search for non-zero pixels included in the tensor, extract at least one parameter from the quadtree, and determine the compression mode of the tensor based on the at least one parameter.

10. The neural network processor according to claim 9, wherein, The at least one parameter includes: The first parameter is generated as a result of the quadtree and corresponds to the total number of bits representing the location information of the non-zero pixels; The second parameter, in response to the pixel having the maximum value among the non-zero pixels being represented in binary, corresponds to the total number of bits from the least significant bit to the most significant non-zero bit; and The third parameter corresponds to the number of zero pixels among the plurality of pixels.

11. The neural network processor according to claim 10, wherein, The neural tensor compressor is further configured to compress the tensor by applying a quadtree method when the value of the first parameter is less than or equal to the number of pixels included in the tensor.

12. The neural network processor according to claim 10, wherein, The neural tensor compressor is further configured to compress the tensor by applying a zero-bitmap method when the value of the first parameter is greater than the number of pixels included in the tensor and when the product of the value of the second parameter and the value of the third parameter is greater than the number of pixels, wherein, in the zero-bitmap method, non-zero pixels are treated as 1 and zero pixels are treated as 0.

13. The neural network processor according to claim 10, wherein, When the value of the first parameter is greater than the number of pixels included in the tensor and when the product of the value of the second parameter and the value of the third parameter is less than or equal to the number of pixels, the tensor is compressed by applying a fixed-length method, wherein the tensor is compressed based on the bit width of the pixel with the maximum value among the plurality of pixels.

14. The neural network processor according to claim 9, wherein, The tensor includes 4 M There are pixels, where M is a natural number.

15. The neural network processor according to claim 9, wherein, The bitstream is output to one of the multiple storage areas, which is configured to correspond to the compression mode.

16. A method for compressing a tensor comprising multiple pixels, comprising: Receive the tensor, which is the result of performing repeated arithmetic calculations on the feature map and weights; Extract at least one parameter, the at least one parameter being the result of repeatedly partitioning the tensor to compress zero pixels among the plurality of pixels included in the tensor; The compression mode is determined based on the at least one parameter; as well as The bit stream is output based on the compression mode.

17. The method according to claim 16, wherein, The extraction of at least one parameter includes: Extract the first parameter, which is generated as a result of the quadtree and corresponds to the total number of bits representing the location information of non-zero pixels; Extract the second parameter, in response to the pixel having the maximum value among the non-zero pixels being represented in binary, the second parameter corresponding to the total number of bits from the least significant bit to the most significant non-zero bit; and Extract a third parameter, which corresponds to the number of zero pixels among the plurality of pixels.

18. The method of claim 17, wherein, The determination of the compression mode includes: Compare the value of the first parameter with the number of pixels included in the tensor; and In response to the first parameter's value being greater than the number of pixels, the product of the second parameter's value and the third parameter's value is compared with the number of pixels.

19. The method of claim 18, further comprising: In response to the value of the first parameter being less than or equal to the number of pixels, the tensor is compressed using a quadtree method.

20. The method of claim 18, further comprising: When the product of the value of the second parameter and the value of the third parameter is greater than the number of pixels, the tensor is compressed using a zero-bitmap method, wherein, in the zero-bitmap method, non-zero pixels are treated as 1 and zero pixels are treated as 0; and When the product of the value of the second parameter and the value of the third parameter is less than or equal to the number of pixels, the tensor is compressed using a fixed-length method, wherein the tensor is compressed based on the bit width of the pixel with the maximum value among the plurality of pixels.

Images