Neural network device, computing system, and method of processing feature maps

Abstract

A neural network device, a computing system, and a method of processing a feature map are provided. The neural network device includes at least one processor configured to implement: an arithmetic circuit configured to generate third data including a plurality of pixels based on a neural network performing an arithmetic operation on first data and second data; and a compressor configured to generate compressed data by compressing the third data, wherein the compressor is further configured to generate bitmap data as the compressed data based on a quadtree structure, the bitmap data including position information of non-zero pixels having non-zero data values among the plurality of pixels.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2019-0055157, filed on May 10, 2019, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The exemplary embodiments of this application relate to a neural network processor, and more specifically, to a neural network processor for compressing feature map data and a computing system including the neural network processor. Background Technology

[0004] A neural network is a computational architecture used to model the biological brain. In recent years, with the development of neural network technology, research has been conducted on using one or more neural network models in various types of electronic systems to analyze input data and extract information using neural network devices.

[0005] Neural network processors require extensive arithmetic operations on complex input data, including writing to and reading input data from external memory. Therefore, due to the large volume of input data, writing or reading input data from external memory can take a considerable amount of time. Summary of the Invention

[0006] One or more example embodiments provide a neural network processor for efficiently compressing input data and a neural network device including the neural network processor.

[0007] One or more example embodiments also provide an efficient method for compressing input data.

[0008] According to one aspect of an example embodiment, a neural network device is provided, comprising: at least one processor configured to implement: an arithmetic circuit configured to generate third data including a plurality of pixels based on a neural network, the neural network being configured to perform arithmetic operations on first data and second data; and a compressor configured to generate compressed data by compressing the third data, wherein the compressor is further configured to generate bitmap data as compressed data based on a quadtree structure, the bitmap data including position information of non-zero pixels having non-zero data values ​​among the plurality of pixels.

[0009] According to another aspect of the example embodiment, a computing system is provided, including: a memory configured to store at least one program; and a processor configured to perform arithmetic operations of a neural network on a feature map including a plurality of pixels by executing the at least one program, wherein the processor is further configured to compress the feature map by generating bitmap data, the bitmap data including position information of non-zero pixels having non-zero data values ​​among the plurality of pixels.

[0010] According to another aspect of the example embodiment, a method for processing feature maps of a neural network in a computing system including memory is provided, the method comprising: generating compressed data based on location information and data of non-zero pixels having non-zero data values ​​among a plurality of pixels included in the feature map, the location information repeatedly indicating the positions of non-zero pixels on a partition plane of the feature map; and storing the compressed data in memory. Attached Figure Description

[0011] The above and / or other aspects will become apparent and more readily understood from the following description of exemplary embodiments taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 This is a block diagram of an electronic system according to an example embodiment;

[0013] Figure 2 This is a block diagram of an electronic system according to another example embodiment;

[0014] Figure 3 A neural network is shown;

[0015] Figure 4A and Figure 4B The convolution operation of a neural network is shown;

[0016] Figure 5 This is a block diagram of a neural network processor according to an example embodiment;

[0017] Figure 6 This is a flowchart of a method for operating a compressor according to an example embodiment;

[0018] Figure 7 This is a flowchart of a method for operating a compressor according to another example embodiment;

[0019] Figure 8 This is a flowchart of a method for operating a compressor according to another example embodiment;

[0020] Figure 9 An input feature map according to an example embodiment is shown;

[0021] Figure 10 Feature diagrams according to an example embodiment are shown;

[0022] Figure 11 A method for generating compressed data according to an example embodiment is shown;

[0023] Figure 12 A method for generating compressed data according to another example embodiment is shown;

[0024] Figure 13A A method for generating compressed data according to another example embodiment is shown;

[0025] Figure 13B A method for generating compressed data according to another example embodiment is shown;

[0026] Figure 13C A method for generating compressed data according to another example embodiment is shown;

[0027] Figure 14 A method for generating compressed data according to another example embodiment is shown; and

[0028] Figure 15 This is a block diagram of an application processor according to an example embodiment. Detailed Implementation

[0029] In the following description, exemplary embodiments will be described in detail with reference to the accompanying drawings.

[0030] Figure 1 This is a block diagram of an electronic system 10 according to an example embodiment.

[0031] Reference Figure 1 Electronic system 10 can extract useful information by analyzing real-time input data based on neural networks, and can determine the context or control components of electronic devices, including electronic system 10, based on the extracted information. For example, electronic system 10 can be applied to drones, advanced driver assistance systems (ADAS), robotic devices, smart TVs (TVs), smartphones, medical devices, mobile devices, image display devices, measuring devices, Internet of Things (IoT) devices, etc., but the example embodiments are not limited thereto, and electronic system 10 can be applied to various other types of electronic devices.

[0032] Electronic system 10 may include at least one intellectual property (IP) block and neural network processor 100. For example, electronic system 10 may include a first IP block IP1, a second IP block IP2, and a third IP block IP3, as well as neural network processor 100.

[0033] Electronic system 10 may include various types of IP blocks. For example, IP blocks may include processing units, multiple cores included in processing units, multi-format codecs (MFC), video modules (e.g., camera interfaces, Joint Picture Experts Group (JPEG) processors), video processors, mixers, etc.), three-dimensional (3D) graphics cores, audio systems, drivers, display drivers, volatile memory, non-volatile memory, memory controllers, input / output interface blocks, cache memory, etc. Each of the first IP block IP1 to the third IP block IP3 may include at least one of various types of IP blocks.

[0034] Technologies for connecting IP blocks include system bus-based connectivity schemes. For example, the Advanced Microcontroller Bus Architecture (AMBA) protocol for Advanced RISC Machines (ARM) can be used as the standard bus specification. Bus types for the AMBA protocol can include Advanced High-Performance Bus (AHB), Advanced Peripheral Bus (APB), Advanced Extensible Interface (AXI), AXI4, and AXI Coherence Extension (ACE). Among these bus types, AXI is the interface protocol between IP blocks and can provide multiple outstanding address functions, data interleaving, etc. In addition to the above, other types of protocols (such as SONICs Inc.'s uNetwork, IBM's CoreConnect, and OCP-IP's open core protocol) can also be applied to the system bus.

[0035] The neural network processor 100 can generate neural networks, train neural networks (or allow neural networks to learn), perform arithmetic operations based on received input data, and generate information signals based on the results of arithmetic operations, or retrain neural networks. The neural network model can include various types of models, such as, but not limited to, convolutional neural networks (CNNs) including GoogleNet, AlexNet, and VGG networks, region-based neural networks (R-CNNs), region proposal networks (RPNs), recurrent neural networks (RNNs), stack-based deep neural networks (S-DNNs), state-space dynamic neural networks (S-SDNNs), deconvolutional networks, deep abstract networks (DBNs), restricted Boltzmann machines (RBMs), fully convolutional networks, long short-term memory (LSTM) networks, classification networks, etc. The neural network processor 100 can include one or more processors configured to perform arithmetic operations according to the model of the neural network. Additionally, the neural network processor 100 can include a separate memory storing programs corresponding to the model of the neural network. The neural network processor 100 can be a neural network processing device, a neural network integrated circuit, a neural network processing unit (NPU), etc.

[0036] The neural network processor 100 can receive various types of input data from at least one IP block via a system bus and generate information signals based on the input data. For example, the neural network processor 100 can generate information signals by performing neural network operations on the input data, and the neural network operations may include convolution operations. (Refer to...) Figure 4A and Figure 4B The convolution operation of the neural network processor 100 is described in detail. The information signal generated by the neural network processor 100 may include at least one of various types of recognition signals, such as speech recognition signals, object recognition signals, image recognition signals, and biometric recognition signals. For example, the neural network processor 100 may receive frame data included in a video stream as input data and generate recognition signals for objects included in an image indicated by the frame data based on the frame data. However, the neural network processor 100 is not limited to this, and the neural network processor 100 may receive various types of input data and generate recognition signals based on the input data.

[0037] According to the electronic system 10 of the example embodiment, among a plurality of pixels included in the input feature map data to be used for convolution operations, the neural network processor 100 can identify zero pixels with zero (“0”) as data values ​​and non-zero pixels with non-zero values ​​as data values. Based on this identification, the neural network processor 100 can generate compressed data by using the data values ​​of the non-zero pixels and the positional information of the non-zero pixels. The neural network processor 100 can generate compressed data with a relatively small amount of data by compressing the feature map data and store the generated compressed data in external memory, thereby improving the data input and output speed.

[0038] In this specification, for ease of description, a pixel with a zero ("0") data value among the plurality of pixels included in the feature map is called a zero pixel, and a pixel with a non-zero value among the plurality of pixels is called a non-zero pixel.

[0039] Figure 2 This is a block diagram of an electronic system 10 according to another example embodiment. (Hereinafter, it refers to...) Figure 2 Electronic systems 10 non-repeating references Figure 1 The description is as follows.

[0040] Reference Figure 2 The electronic system 10 may include a neural network processor 100, random access memory (RAM) 200, a processor 300, a memory 400, and a sensor module 500. The neural network processor 100 may correspond to... Figure 1 The neural network processor 100.

[0041] RAM 200 can temporarily store programs, data, or instructions. For example, programs and / or data stored in memory 400 can be temporarily loaded into RAM 200 according to the control or boot code of processor 300. RAM 200 can be implemented using memory such as dynamic RAM (DRAM) or static RAM (SRAM).

[0042] Processor 300 can control the general operation of electronic system 10, and for example, processor 300 can be a central processing unit (CPU). Processor 300 may include a single core or multiple cores. Processor 300 can process or execute programs and / or data stored in RAM 200 and memory 400. For example, processor 300 can control the functions of electronic system 10 by executing programs stored in memory 400.

[0043] Memory 400 is a memory for storing data and can store, for example, an operating system (OS), various programs, and various types of data. Memory 400 may include DRAM, but is not limited to it. Memory 400 may include at least one of volatile memory and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FeRAM), etc. Volatile memory may include DRAM, SRAM, synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, FeRAM, etc. In addition, according to the example embodiment, memory 400 may include at least one of the following: hard disk drive (HDD), solid-state drive (SSD), high-density flash memory (CF) card, secure digital card (SD card), micro-secure digital card (Micro-SD card), mini-secure digital card (Mini-SD card), extreme digital card (XD card), or memory stick.

[0044] Sensor module 500 can collect information from the periphery of electronic system 10. Sensor module 500 can sense or receive image signals from outside electronic system 10 and convert the sensed or received image signals into image data, i.e., image frames. For this purpose, sensor module 500 may include sensing devices, such as at least one of various types of sensing devices, such as imaging devices, image sensors, light detection and ranging (LIDAR) sensors, ultrasonic sensors, infrared sensors, or sensor module 500 may receive sensing signals from sensing devices. According to an example embodiment, sensor module 500 can provide image frames to neural network processor 100. For example, sensor module 500 may include an image sensor and generate a video stream by capturing images of the external environment of electronic system 10, and sequentially provide consecutive image frames of the video to neural network processor 100.

[0045] According to the electronic system 10 of the example embodiment, the neural network processor 100 can generate compressed data by using non-zero pixel data values ​​and non-zero pixel position information as described above, and store the generated compressed data in RAM 200 or memory 400. The neural network processor 100 can temporarily store the compressed data in RAM 200 and use the temporarily stored compressed data to process corresponding arithmetic operations, thereby improving the processing speed of the neural network. Alternatively, the neural network processor 100 can store the compressed data in memory 400, thereby storing a relatively small amount of feature map data.

[0046] Figure 3 The structure of a convolutional neural network is shown as an example of a neural network structure.

[0047] Reference Figure 3 A neural network NN may include multiple layers, for example, a first layer L1 to an nth layer Ln. Each of the multiple layers L1 to Ln may be a linear layer or a non-linear layer, and according to an example embodiment, at least one linear layer and at least one non-linear layer may be combined and referred to as a layer. For example, a linear layer may include a convolutional layer and a fully connected layer, while a non-linear layer may include a pooling layer and an activation layer.

[0048] For example, the first layer L1 can be a convolutional layer, the second layer L2 can be a pooling layer, and the nth layer Ln can be a fully connected layer that serves as the output layer. The neural network NN can further include activation layers, and can further include layers configured to perform another arithmetic operation.

[0049] Each of the multiple layers L1 to Ln can receive input data (e.g., image frames) or a feature map generated in the previous layer as an input feature map, and perform arithmetic operations on the input feature map to generate an output feature map or recognition signal REC. In this case, the feature map indicates data representing various features of the input data. The first to nth feature maps FM1, FM2, ..., FMn can have, for example, a 2D matrix or a 3D matrix (or tensor) format. The first to nth feature maps FM1, FM2, ..., FMn can have a width (or column) W, a height (or row) H, and a depth D corresponding to the x-axis, y-axis, and z-axis of the coordinate system, respectively. Here, the depth D can be referred to as the number of channels.

[0050] The first layer L1 generates the second feature map FM2 by convolving the first feature map FM1 and the weight map WM. The weight map WM filters the first feature map FM1 and can also be referred to as a filter or kernel. The depth (i.e., number of channels) of the weight map WM is the same as the depth (i.e., number of channels) of the first feature map FM1, and the same channels of the weight map WM and the first feature map FM1 can be convolved. Using the first feature map FM1 as a sliding window, the weight map WM can be shifted in an iterative manner. The shift amount can be referred to as the "stride length" or "step". During each shift, each weight value included in the weight map WM is multiplied and added to all pixel data in the region overlapping with the first feature map FM1. The data in the region of the first feature map FM1 where each weight value included in the weight map WM overlaps with the first feature map FM1 can be referred to as extracted data. One channel of the second feature map FM2 can be generated based on the convolution of the first feature map FM1 and the weight map WM. Although... Figure 3 A weight map WM is shown, but essentially multiple weight maps can be convolved with a first feature map FM1 to generate multiple channels of a second feature map FM2, and the number of channels of the second feature map FM2 can correspond to the number of weight maps.

[0051] The second layer L2 can generate a third feature map FM3 by modifying the spatial size of the second feature map FM2 via pooling. Pooling can be referred to as sampling or downsampling. A 2D pooling window PW can be shifted on the second feature map FM2 in units of the pooling window PW, and the maximum value (or mean value) of the pixel data in the region overlapping with the pooling window PW can be selected. Therefore, a third feature map FM3 with a spatial size modified from the second feature map FM2 can be generated. The number of channels in the third feature map FM3 is the same as the number of channels in the second feature map FM2.

[0052] The nth layer Ln can classify the category CL of the input data by combining the features of the nth feature map FMn. Additionally, the nth layer Ln can generate a recognition signal SEC corresponding to the category. According to an example embodiment, the input data can correspond to frame data included in a video stream, and the nth layer Ln can extract the category corresponding to the object included in the image indicated by the frame data based on the nth feature map FMn provided from the previous layer to identify the object, and generate a recognition signal REC corresponding to the identified object.

[0053] Figure 4A and Figure 4B The convolution operation of a neural network is shown.

[0054] Reference Figure 4A The input feature map 201 can include D channels, and each channel of the input feature map can have a size of H rows and W columns, where D, H, and W are natural numbers. Each kernel 202 can have a size of R rows and S columns, and kernel 202 can include a number of channels corresponding to the number of channels (or depth) D of the input feature map 201, where R and S are natural numbers. The output feature map (203) can be generated by a 3D convolution operation between the input feature map 201 and kernel 202, and can include Y channels according to the convolution operation.

[0055] You can refer to this. Figure 4B This describes the operation of generating an output feature map through a convolution operation between an input feature map and a kernel. A reference operation can be performed between the input feature map 201 (all channels) and the kernel 202 (all channels). Figure 4B The 2D convolution operation is described to generate output feature maps for all channels (203).

[0056] Reference Figure 4B As an example, the input feature map 210 has a size of 6×6, the original kernel 220 has a size of 3×3, and the output feature map 230 has a size of 4×4. However, the sizes of the input feature map 210, the original kernel 220, and the output feature map 230 are not limited to these, and the neural network can be implemented with feature maps and kernels of various sizes. Furthermore, all values ​​defined in the input feature map 210, the original kernel 220, and the output feature map 230 are merely illustrative, and the embodiments are not limited thereto.

[0057] Convolution operations can be performed when the original kernel 220 slides across the input feature map 210 in 3×3 window units. The convolution operation can instruct the following arithmetic operation: multiplying the pixel data in a specific window by the weight values ​​at corresponding positions in the original kernel 220, and summing all the values ​​obtained through multiplication to obtain each pixel data of the output feature map 230. The data multiplied by the weight values ​​included in the window of the input feature map 210 can be referred to as the extracted data extracted from the input feature map 210. Specifically, firstly, a convolution operation can be performed between the original kernel 220 and the first extracted data 211 of the input feature map 210. That is, the pixel data 0, 2, 0, 4, 0, 0, 0, 0, and 0 of the first extracted data 211 can be multiplied by the weight values ​​of the original kernel 220—-1, -3, 4, 7, -2, -1, -5, 3, and 1, respectively, and the result of the multiplication is 0, -6, 0, 28, 0, 0, 0, 0, and 0. Next, 22 can be calculated, which is the result of summing all the obtained values, and the pixel data 231 in the first row and first column of the output feature map 230 can be determined as 22. Here, the pixel data 231 in the first row and first column of the output feature map 230 corresponds to the first extracted data 211. Similarly, a convolution operation can be performed between the second extracted data 212 of the input feature map 210 and the original kernel 220 to determine that the pixel data 232 in the first row and second column of the output feature map 230 is 2. Finally, a convolution operation can be performed between the 16th extracted data 213 (i.e., the last extracted data) of the input feature map 210 and the original kernel 220 to determine that the pixel data 233 in the fourth row and fourth column of the output feature map 230 is 0.

[0058] The convolution operation between an input feature map 210 and a raw kernel 220 can be processed by repeatedly performing the product of the extracted data from the input feature map 210 and the corresponding weight values ​​of the raw kernel 220, and then adding the product results. As a result of the convolution operation, an output feature map 230 can be generated.

[0059] According to an example embodiment, when the Corrected Linear Unit (ReLU) function is applied to the input feature map 210, negative pixel data can be replaced with "0". As a result, the input feature map 210 may include a number of zero pixels with "0" as data values. According to an example embodiment, the neural network processor 100 can reduce the data in the input feature map 210 by generating compressed data using only the positional information of non-zero pixels and the data values ​​of non-zero pixels. The method and apparatus for compressing data in the input feature map 210 according to the example embodiment, performed by the neural network processor 100, will be described in more detail below with reference to the accompanying drawings.

[0060] Figure 5This is a block diagram of a neural network processor 100 according to an example embodiment.

[0061] Reference Figure 5 The neural network processor 100 may include control logic 110, compressor 120, internal memory 130, and arithmetic circuits 160.

[0062] Internal memory 130 can receive external data from outside the neural network processor 100. The external data can be input data. Internal memory 130 can store various data and weight values ​​required for arithmetic operations, as well as the external data. For this purpose, internal memory 130 may include: kernel memory 140 storing kernel data KD; input feature map memory 150 storing input feature map IF; and output feature map memory 155 storing output feature map OF. Kernel memory 140, input feature map memory 150, and output feature map memory 155 may comprise a single piece of hardware or multiple separate pieces of hardware, but embodiments are not limited thereto. For example, kernel memory 140, input feature map memory 150, and output feature map memory 155 may refer to memories corresponding to different regions within a single piece of hardware. Each of kernel memory 140, input feature map memory 150, and output feature map memory 155 may be implemented using various types of memory such as DRAM, SRAM, and synchronous dynamic RAM (SDRAM).

[0063] Input feature map memory 150 can store input feature map IF and provide input feature map IF to arithmetic circuit 160 and compressor 120. Kernel memory 140 can include kernel data KD, which includes multiple weight values ​​to be used for convolution operations, and kernel memory 140 can provide kernel data KD to arithmetic circuit 160. Output feature map memory 155 can store output feature map OF received from arithmetic circuit 160 and output the stored feature map FM to compressor 120.

[0064] Arithmetic circuit 160 can receive an input feature map IF from input feature map memory 150 and kernel data KD from kernel memory 140. Arithmetic circuit 160 can generate an output feature map OF by performing a convolution operation using the received input feature map IF and kernel data KD.

[0065] Control logic 110 can control the general operation of neural network processor 100 and can be, for example, a CPU. Control logic 110 may include a single core or multiple cores. Control logic 110 can control the functions of neural network processor 100 by executing programs stored in internal memory 130.

[0066] Compressor 120 can receive a feature map FM and generate compressed data CD by using the position information of non-zero pixels with non-zero values ​​("0") among the multiple pixels included in the feature map FM, and the data values ​​of non-zero pixels. The feature map FM may include an input feature map IF received from an external source or an output feature map OF received from arithmetic circuit 160. Compressor 120 may include data processing circuitry or software.

[0067] Compressor 120 can be used as a decompressor and is configured to decompress from an external source (e.g., Figure 2 The RAM 200 receives compressed data CD and decompresses the received compressed data CD. The decompression method can be performed in the reverse order of the compression method, so the example embodiment can be applied not only to the compression method but also to the decompression method for recovering the feature map FM from the compressed data. For example, the compressor 120 can receive compressed data CD from an external source, decompress the received compressed data CD to generate an input feature map IF, and store the generated input feature map IF in the input feature map memory 150.

[0068] Example embodiments of compressed input data include, for example, compressed feature maps FM, but embodiments are not limited thereto, and other input data, such as kernel data KD, can be compressed.

[0069] Figure 6 This is a flowchart of a method for operating a compressor according to an example embodiment.

[0070] Reference Figure 5 and Figure 6 In operation S10, compressor 120 can generate compressed data CD by using position information and data values ​​of non-zero pixels, where the position information repeatedly indicates the position of the non-zero pixels on the dividing plane. In operation S20, compressor 120 can output the generated compressed data CD to a memory (e.g., ...). Figure 2 The RAM 200 allows the generated compressed data CD to be stored in the memory.

[0071] Figure 7 This is a flowchart of a method for operating a compressor according to another example embodiment. Figure 7 It can be at least a part of a method for generating compressed data, executed by a compressor. Figure 6 S10), and a method for generating bitmap data that indicates the position information of non-zero units, including non-zero pixels, is shown.

[0072] Reference Figure 5 and Figure 7In operation S110, compressor 120 can generate group mapping data that indicates the position information of non-zero groups among multiple groups divided into rows from multiple cells, the non-zero group including at least one non-zero pixel.

[0073] According to the example embodiment, a plurality of units indicate pixel units, each pixel unit including N×N pixels, where N is a natural number, and a unit including at least one non-zero pixel can be called a non-zero unit.

[0074] In operation S120, compressor 120 can generate cell mapping data indicating the position information of non-zero cells in the non-zero group. In operation S130, compressor 120 can generate pixel count data indicating the number of non-zero pixels included in the non-zero cells.

[0075] According to an example embodiment, compressor 120 can generate bitmap data including hierarchical data indicating the positions of non-zero pixels based on a partition plane, and as described above in this specification, the method of indicating the positions of non-zero pixels by generating hierarchical data based on a partition plane is called a quadtree scheme. (Refer to...) Figure 11 Describe the quadtree solution in detail.

[0076] According to an example embodiment, compressor 120 can generate bitmap data including group mapping data indicating position information of non-zero groups, cell mapping data indicating position information of non-zero cells in non-zero groups, and pixel count data indicating the number of non-zero pixels.

[0077] Figure 8 This is a flowchart of a method for operating a compressor according to another example embodiment. In detail, Figure 8 It is at least a part of the method for generating compressed data performed by the compressor. Figure 6 S10) and a method for generating cell data that indicates the position information of non-zero pixels and the data value of non-zero pixels in non-zero cells is shown.

[0078] Reference Figure 5 and Figure 8 In operation S210, compressor 120 can generate maximum bit data indicating the maximum number of bits of information about non-zero pixels. In operation S220, compressor 120 can generate pixel position data indicating the position information of non-zero pixels by repeatedly indicating the position of non-zero pixels in the quadrant of non-zero cells. In operation S230, compressor 120 can generate non-zero pixel data indicating the value of non-zero pixels.

[0079] According to an example embodiment, compressor 120 can generate at least one cell data, which includes maximum bit data indicating information about the maximum bit width of a non-zero pixel, pixel position data indicating the position information of a non-zero pixel in a non-zero cell, and non-zero pixel data indicating the value of a non-zero pixel.

[0080] Figure 9 An input feature map according to an example embodiment is shown.

[0081] Reference Figure 9 The input feature map FM can include 8×8 pixels, and the multiple pixels can include zero pixels ZP with "0" as data value and non-zero pixels NP with non-"0" as data value. As mentioned above, the ReLU operation can be applied to the feature map FM, so the pixel with a data value corresponding to a negative number can be replaced with the zero pixel ZP.

[0082] although Figure 9 A pixel with eight bits of data is shown, but this is merely an example, and each of multiple pixels can have more or less eight bits of data. Additionally, although... Figure 9 An example of a feature map FM consisting of 8×8 pixels is shown, but a feature map FM can include pixels larger or smaller than 8×8.

[0083] Figure 10 A feature diagram according to an example embodiment is shown.

[0084] Reference Figure 10 The feature map FM can include three groups divided by rows, for example, group 1 G1, group 2 G2, and group 3 G3. Group 1 G1 can include unit 1 C1, unit 2 C2, and unit 3 C3; group 2 G2 can include unit 4 C4, unit 5 C5, and unit 6 C6; and group 3 C3 can include unit 7 C7, unit 8 C8, and unit 9 C9. Figure 10 In the example, the cell indicator includes a pixel cell of 4×4 pixels, and the second cell C2 includes non-zero pixels NP, so the second cell C2 can be called a non-zero cell. Additionally, the first group G1, including the second cell C2 as a non-zero cell, can be called the non-zero group.

[0085] Figure 11 A method for generating compressed data according to an example embodiment is illustrated. In detail, Figure 11 A method for generating compressed data is shown when the feature map includes a non-zero pixel.

[0086] Reference Figure 5 and Figure 11The fifth unit C5 constituting the feature map FM may include a non-zero pixel NP. According to an example embodiment, the compressor 120 may generate compressed data CD based on the quadtree QT using the feature map FM.

[0087] Compressor 120 can generate group mapping data (GMD) by indicating at corresponding bits whether each of the multiple groups includes a non-zero pixel (NP). Figure 11 In the example, the first group G1 and the third group G3 may not include non-zero pixels NP, while the second group G2 may include non-zero pixels NP. Compressor 120 can configure the first bit corresponding to the first group G1 and the third bit corresponding to the third group G3 to "0", and configure the second bit corresponding to the second group G2 to "1", resulting in compressor 120 generating "010" as group mapping data GMD. Because both the first group G1 and the third group G3 have "00" as pixel data, compressor 120 can ignore the data from the first group G1 and the third group G3.

[0088] Compressor 120 can generate cell mapping data CMD by indicating at corresponding bits whether each of the plurality of cells included in the second group G2 includes a non-zero pixel NP. Figure 11 In the example, the fourth unit C4 and the sixth unit C6 may not include non-zero pixels NP, while the fifth unit C5 may include non-zero pixels NP. The compressor 120 can configure the first bit corresponding to the fourth unit C4 and the third bit corresponding to the sixth unit C6 to "0", and configure the second bit corresponding to the fifth unit C5 to "1", resulting in the compressor 120 generating "010" as the unit mapping data CMD. Because both the fourth unit C4 and the sixth unit C6 have "00" as pixel data, the compressor 120 can ignore the data of the fourth unit C4 and the sixth unit C6. Although Figure 11 The diagram shows the allocation of three bits as group mapping data (GMD) and cell mapping data (CMD), but this is merely an example, and more or fewer bits than three may be allocated as group mapping data (GMD) and cell mapping data (CMD) based on the number of groups and cells.

[0089] Compressor 120 can generate pixel count data PND based on the number of non-zero pixels NP. Figure 11 In the example, the number of non-zero pixels NP is 1, so compressor 120 can generate "001" as pixel count data PND corresponding to the number of non-zero pixels NP. Although Figure 11 The diagram shows the allocation of three bits as the pixel count data PND, but this is only an example, and more or fewer bits can be allocated as the pixel count data PND.

[0090] As described above, compressor 120 can generate bitmap data BMD including group mapping data GMD, cell mapping data CMD, and pixel number data PND. According to the example embodiment, compressor 120 can add arbitrary padding bits to the bitmap data BMD to satisfy the number of bits (e.g., a multiple of eight bits).

[0091] Compressor 120 can divide the fifth unit C5 into multiple quadrants and indicate at corresponding bits whether each quadrant includes a non-zero pixel NP, thereby generating at least one pixel position data. Figure 11 In the example, the first quadrant Q1, the second quadrant Q2, and the fourth quadrant Q4 may not include non-zero pixels NP, while the third quadrant Q3 may include non-zero pixels NP. Therefore, the compressor 120 can configure the first bit corresponding to the first quadrant Q1, the second bit corresponding to the second quadrant Q2, and the fourth bit corresponding to the fourth quadrant Q4 as "0", and configure the third bit corresponding to the third quadrant Q3 as "1", so the compressor 120 can generate "0010" as the first pixel position data PLD1.

[0092] Compressor 120 can generate pixel position data as described above until a non-zero pixel NP is specified. Figure 11 In the example, the third quadrant Q3 comprises four pixels, and the non-zero pixel NP is located in the first position (top left) within the third quadrant Q3. Therefore, the compressor 120 can configure only the bit corresponding to the non-zero pixel NP as "1" and configure the bits located in the remaining positions as "0". As a result, the compressor 120 can generate "1000" as the second pixel position data PLD2.

[0093] exist Figure 11 In this embodiment, a non-zero pixel is specified by two pixel position data, and thus the two pixel position data are included in the compressed data CD. However, the embodiment is not limited to this, and two or more or fewer pixel position data can be included in the compressed data CD.

[0094] Compressor 120 can generate non-zero pixel data NPD based on the data values ​​of non-zero pixel NP. Figure 11 In the example, the non-zero pixel NP has an eight-bit data value "02", so the compressor 120 can generate a two-bit data value "00000010" as the non-zero pixel data NPD to correspond to the eight-bit data value "02".

[0095] Compressor 120 can generate cell data CLD including first pixel position data PLD1, second pixel position data PLD2, and non-zero pixel data NPD, and can generate compressed data CD including bitmap data BMD and cell data CLD. According to an example embodiment, compressor 120 can add arbitrary padding bits to the cell data CLD to meet the number of bits (e.g., a multiple of eight bits).

[0096] According to the example embodiment, the position of non-zero pixels can be specified based on a quadtree, and compressed data CD can be generated by using the specified position of non-zero pixels, thereby reducing the amount of data in the compressed data CD and improving the data transmission rate.

[0097] According to an example embodiment, compressor 120 can recover feature map FM based on compressed data CD. According to the example embodiment, compressor 120 can: recover the positions of non-zero cells based on group mapping data GMD and cell mapping data CMD, recover the number of non-zero pixels based on pixel number data PND, and recover the positions of non-zero pixels based on first pixel position data PLD1 and second pixel position data PLD2. Compressor 120 can recover feature map FM by inputting the data value of a non-zero pixel at the recovery position of the non-zero pixel based on non-zero pixel data NPD and inputting "00" at the remaining positions.

[0098] although Figure 11 An example is shown where compressed data CD sequentially includes group mapping data GMD, cell mapping data CMD, pixel number data PND, first pixel position data PLD1, second pixel position data PLD2, and non-pixel data NPD. However, this is only an example, and compressed data CD can be generated by various sequential combinations of the above data.

[0099] Figure 12 A method for generating compressed data according to another example embodiment is shown. In detail, Figure 12 This illustrates a method for generating compressed data when the feature map includes a non-zero pixel. References are not repeated here. Figure 11 The description is as follows.

[0100] Reference Figure 5 and Figure 12 The compressor can generate Maximum Bit Data (MBD). MBD indicates the maximum number of bits in the data value of a non-zero pixel NP. According to an example embodiment, MBD can have "the maximum number of bits in the data value of a non-zero pixel NP - 1" as the data value. Figure 12In the example, the data value of the non-zero pixel NP is "02", so the maximum number of bits for the data value of the non-zero pixel NP can be 2. Therefore, the maximum bit data MBD can be configured as "001" corresponding to two bits. According to the example embodiment, there is at least one bit as the data value of the non-zero pixel NP, so "000" can correspond to one bit, and "001" can correspond to two bits. For example, the maximum bit data MBD corresponding to eight bits can be "111".

[0101] When the maximum bit data (MBD) is present, the compressor 120 can change the non-zero pixel data (NPD) from "00000010" to "10" by removing unnecessary "0"s. During the recovery process, the compressor 120 can treat the pixel data as "02" by taking the maximum bit data (MBD) into account.

[0102] Figure 13A A method for generating compressed data according to another example embodiment is shown. In detail, Figure 13A This illustrates a method for generating compressed data when a cell comprises multiple non-zero pixels. References are not repeated here. Figure 11 and 12 The description is as follows. Specifically, the above has already referenced... Figure 11 The method for generating group mapping data (GMD) and cell mapping data (CMD) in bitmap data (BMD) has been described, so its description will not be repeated here.

[0103] Reference Figure 5 and Figure 13A Since the fifth unit C5 includes two non-zero pixels, for example, the first non-zero pixel NP1 and the second non-zero pixel NP2, the compressor 120 can generate "010" as the pixel count data PND. Since the data value of the second non-zero pixel NP2 is "0E" and can be represented by four bits, the compressor 120 can generate "011" corresponding to four bits as the maximum bit data MBD.

[0104] Compressor 120 can configure the first bit corresponding to the first quadrant Q1 and the fourth bit corresponding to the fourth quadrant Q4 as "0", and configure the second bit corresponding to the second quadrant Q2 and the third bit corresponding to the third quadrant Q3 as "1". Therefore, since the first non-zero pixel NP1 is located in the second quadrant Q2 and the second non-zero pixel NP2 is located in the third quadrant Q3, compressor 120 can generate "0110" as the first pixel position data PLD1. In addition, since the first non-zero pixel NP1 is located in the second position (upper right) in the second quadrant Q2, compressor 120 can generate "0100" as the second pixel position data PLD2, and since the second non-zero pixel NP2 is located in the first position (upper left) in the third quadrant Q3, compressor 120 can generate "1000" as the third pixel position data PLD3.

[0105] The compressor 120 can generate "1110" corresponding to "0E" as the data value of the first non-zero pixel NP1 as the first non-zero pixel data NPD1, generate "0010" corresponding to "01" as the data value of the second non-zero pixel NP2 as the second non-zero pixel data NPD2, and generate compressed data CD by arranging the generated data according to the previously defined rules.

[0106] Figure 13B A method for generating compressed data according to another example embodiment is shown. In detail, Figure 13B This illustrates a method for generating compressed data when a cell comprises multiple non-zero pixels. References are not repeated here. Figure 13A The description is as follows.

[0107] Reference Figure 5 and Figure 13B The compressor 120 can generate first maximum bit data MBD1 and second maximum bit data MBD2 for the first non-zero pixel NP1 and the second non-zero pixel NP2, respectively. Corresponding to the first non-zero pixel NP1, the compressor 120 can generate first sub-unit data SCLD1, including the first maximum bit data MBD1, first pixel position data PLD1, second pixel position data PLD2, and first non-zero pixel data NPD1; and corresponding to the second non-zero pixel NP2, it can generate second sub-unit data SCLD2, including the second maximum bit data MBD2, third pixel position data PLD3, fourth pixel position data PLD4, and second non-zero pixel data NPD2. The method for generating each data has already been described above, so it will not be repeated here.

[0108] Figure 13C A method for generating compressed data according to another example embodiment is shown. In detail, Figure 13CThis illustrates a method for generating compressed data when a cell comprises multiple non-zero pixels. References are not repeated here. Figure 13A The description is as follows.

[0109] Reference Figure 5 and Figure 13C The compressor 120 can generate first maximum bit data MBD1 and second maximum bit data MBD2 for the first non-zero pixel NP1 and the second non-zero pixel NP2, respectively. The compressor 120 can generate first pixel position data PLD1, second pixel position data PLD2, and third pixel position data PLD3 based on the positions of the first non-zero pixel NP1 and the second non-zero pixel NP2 in the fifth unit C5. Furthermore, the compressor 120 can generate first maximum bit data MBD1 and first non-zero pixel data NPD1 corresponding to the first non-zero pixel NP1, and generate second maximum bit data MBD2 and second non-zero pixel data NPD2 corresponding to the second non-zero pixel NP2. The method for generating each data has already been described above, so it will not be repeated here.

[0110] Figure 14 A method for generating compressed data according to another example embodiment is shown. In detail, Figure 14 A method for generating compressed data is illustrated when multiple units include at least one of non-zero pixels. References are not repeated here. Figure 11 and 12 The description is as follows. Specifically, the above has already referenced... Figure 11 The method for generating bitmap data (BMD) has been described, so it will not be repeated here.

[0111] Reference Figure 5 and Figure 14 Because the feature map FM includes three non-zero pixels, for example, a first non-zero pixel NP1, a second non-zero pixel NP2, and a third non-zero pixel NP3, the compressor 120 can generate "011" as the pixel count data PND. The compressor 120 can be configured to use the above reference... Figure 12 The same method described is used to generate the first unit data CD1.

[0112] Compressor 120 can configure "101" as the second unit mapping data CMD2 because the seventh unit C7 in the third group G3 includes the second non-zero pixel NP2, and the ninth unit C9 includes the third non-zero pixel NP3. Additionally, compressor 120 can configure "100" as the second maximum bit data MBD2 based on "19", where "19" is the data value of the second non-zero pixel NP2 with the larger data value between the second non-zero pixel NP2 and the third non-zero pixel NP3.

[0113] Compressor 120 can generate third pixel position data PLD3 and fourth pixel position data PLD4 based on the position of the second non-zero pixel NP2, and can generate second non-zero pixel data NPD2 based on the data value of the second non-zero pixel NP2. Furthermore, compressor 120 can generate fifth pixel position data PLD5 and sixth pixel position data PLD6 based on the position of the third non-zero pixel NP3, and can generate third non-zero pixel data NPD3 based on the data value of the third non-zero pixel NP3.

[0114] Compressor 120 can generate compressed data CD by arranging the generated data according to previously defined rules.

[0115] Figure 15 This is a block diagram of an application processor 1000 according to an example embodiment. The application processor 1000 may be implemented as a semiconductor chip by a system-on-a-chip (SoC).

[0116] Reference Figure 15 The application processor 1000 may include a processor 1010 and working memory 1020. Additionally, the application processor 1000 may include one or more IP modules connected to a system bus. The working memory 1020 may store software such as various programs and instructions relating to the operation of a system employing the application processor 1000, and may include, for example, an OS 1021, a neural network (NN) module 1022, and a compressor module 1023. The compressor module 1023 may be used as a compressor according to the above example embodiment.

[0117] The NN module 1022 can perform arithmetic operations of existing neural network processors or neural network processors according to the example embodiment. Additionally, according to the example embodiment, the compressor module 1023 can receive an input neural network model and perform a compression operation. According to the example embodiment, the NN module 1022 can be implemented in OS 1021.

[0118] although Figure 15 A processor 1010 is shown, but an application processor may include multiple processors. In this case, some of the multiple processors may correspond to general-purpose processors, while others may be dedicated processors for executing neural network models.

[0119] Although exemplary embodiments have been described with reference to the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope defined by the claims.

Claims

1. A neural network device, comprising: At least one processor is configured to implement: Arithmetic circuits are configured to generate feature maps comprising multiple pixels using neural network-based arithmetic operations. as well as A compressor is configured to generate compressed data by compressing the feature map. The compressor is further configured to generate bitmap data as the compressed data, the bitmap data including the position information of non-zero pixels with non-zero data values ​​among the plurality of pixels. The feature map comprises multiple units, and each unit comprises n×n pixels, where n is a natural number. The multiple units are divided into multiple groups according to row units. The bitmap data includes hierarchical data based on a quadtree structure, and the hierarchical data includes: Group mapping data indicating information about a non-zero group among the plurality of groups, including non-zero units, wherein the non-zero units include the non-zero pixels; and Cell mapping data, indicating information about the non-zero cells in the non-zero group.

2. The neural network device according to claim 1, wherein, The bitmap data includes information about the position of the non-zero cells among the plurality of cells.

3. The neural network device according to claim 2, wherein, The bitmap data indicates the position of the non-zero cells by repeatedly indicating the position of the non-zero cells on a plurality of partition planes for at least a portion of the feature map.

4. The neural network device according to claim 1, wherein, The group mapping data includes "1" bits corresponding to the non-zero group and "0" bits corresponding to the groups other than the non-zero group. The unit mapping data includes "1" as the bit corresponding to the non-zero unit, and includes "0" as the bit corresponding to the unit other than the non-zero unit among the plurality of units.

5. The neural network device according to claim 1, wherein, The plurality of units includes a first non-zero unit and a second non-zero unit, and The cell mapping data includes first sub-cell mapping data indicating the position of the first non-zero cell and second sub-cell mapping data indicating the position of the second non-zero cell.

6. The neural network device according to claim 1, wherein, The bitmap data also includes pixel count data, which indicates the number of non-zero pixels included in the non-zero unit.

7. The neural network device according to claim 1, wherein, The compressed data also includes at least one unit of data indicating the position information and data information of the non-zero pixels.

8. The neural network device according to claim 7, wherein, Each of the at least one unit data includes: pixel position data, indicating information about the position of the non-zero pixel in the non-zero unit; and non-zero pixel data, indicating the data value of the non-zero pixel.

9. The neural network device according to claim 8, wherein, The pixel location data is generated based on a quadtree, which indicates the location of the non-zero pixels by repeatedly indicating the location of the non-zero pixels on multiple partition planes for at least a portion of the feature map.

10. The neural network device according to claim 8, wherein, The compressed data also includes maximum bit data, which indicates the maximum number of bits in the data value of the non-zero pixel.

11. The neural network device according to claim 10, wherein, The maximum bit data value is m-1, where the maximum number of bits for the non-zero pixel is m, and m is a natural number.

12. A computing system, comprising: The memory is configured to store at least one program; as well as The processor is configured to perform neural network arithmetic operations on a feature map comprising multiple pixels by executing the at least one program. The processor is further configured to compress the feature map by generating bitmap data, the bitmap data including the position information of non-zero pixels with non-zero data values ​​among the plurality of pixels. The feature map comprises multiple units, and each unit comprises n×n pixels, where n is a natural number. The multiple units are divided into multiple groups according to row units. The bitmap data includes hierarchical data based on a quadtree structure, and the hierarchical data includes: Group mapping data indicating information about a non-zero group among the plurality of groups, including non-zero units, wherein the non-zero units include the non-zero pixels; and Cell mapping data, indicating information about the non-zero cells in the non-zero group.

13. The computing system according to claim 12, wherein, The memory is also configured to store compressed data generated based on the compression of the feature map.

14. The computing system according to claim 12, wherein, The processor is also configured to generate the bitmap data based on a quadtree, which indicates the position of the non-zero pixel by repeatedly indicating the position of the non-zero pixel on a plurality of partition planes for at least a portion of the feature map.

15. The computing system according to claim 12, wherein, The processor is also configured to: For the group mapping data, a "1" is generated as the bit corresponding to the non-zero group, and a "0" is generated as the bit corresponding to the group other than the non-zero group among the plurality of groups, and For the cell mapping data, a "1" is generated as the bit corresponding to the non-zero cell, and a "0" is generated as the bit corresponding to the cell among the plurality of cells other than the non-zero cell.

16. The computing system according to claim 12, wherein, The processor is further configured to generate pixel count data indicating the number of non-zero pixels included in the non-zero units, as the bitmap data.

17. The computing system according to claim 12, wherein, The processor is further configured to generate at least one unit data indicating the position information of the non-zero pixel and the data information of the non-zero pixel.

18. The computing system according to claim 17, wherein, The processor is also configured to generate the following data as the at least one unit data: pixel position data, indicating information about the position of the non-zero pixel in the non-zero unit; And non-zero pixel data, indicating the data value of the non-zero pixel.

19. The computing system according to claim 18, wherein, The processor is also configured to generate the pixel location data based on a quadtree, which indicates the location of the non-zero pixel by repeatedly indicating the location of the non-zero pixel on a plurality of partition planes for at least a portion of the feature map.

20. The computing system according to claim 18, wherein, The processor is also configured to generate maximum bit data indicating the data value of the non-zero pixel as the at least one unit data.

21. A method for processing feature maps of a neural network in a computing system including a memory, the method comprising: Compressed data is generated based on location information and on data of non-zero pixels with non-zero data values ​​among a plurality of pixels included in the feature map, wherein the location information repeatedly indicates the position of the non-zero pixels on the dividing plane of the feature map; as well as The compressed data is stored in a memory. The feature map comprises multiple units, each unit consisting of n×n pixels, where n is a natural number. The multiple units are divided into multiple groups according to row units. The generation of compressed data includes generating hierarchical data based on a quadtree structure, wherein generating the hierarchical data includes: Generate group mapping data indicating information about a non-zero group among the plurality of groups, the non-zero unit including the non-zero pixel; and Generate cell mapping data, which indicates information about the non-zero cells in the non-zero group.

22. The method according to claim 21, wherein, The generated compressed data also includes: Pixel count data is generated, which indicates the number of non-zero pixels included in the non-zero unit.

23. The method of claim 22, further comprising: Generate maximum bit data, which indicates information about the maximum number of bits for the non-zero pixel; as well as At least one unit of data indicating the position and data information of the non-zero pixels is generated as the compressed data. Generating the at least one unit of data includes: Pixel position data is generated to specify the position of the non-zero pixel, the pixel position data indicating information about the non-zero pixel by repeatedly indicating the position of the non-zero pixel in the quadrant of the non-zero unit; and Generate non-zero pixel data that indicates the data value of the non-zero pixel.

Images