Method and apparatus for processing data of a neural network

Abstract

A method and apparatus for processing data of a neural network are provided. The method includes obtaining one or more bit representations for processing data of a neural network from a memory and / or a neural network processor; generating a plurality of candidate profiles based on the one or more bit representations; determining a final profile by comparing compression performance of each of the candidate profiles; determining an optimal profile for compressing data of the neural network based on the determined final profile, and compressing the data of the neural network using the determined optimal profile; and storing the compressed data of the neural network to the memory.

Description

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 930,029, filed November 4, 2019, with the United States Patent and Trademark Office, and Korean Patent Application No. 10-2020-0136124, filed October 20, 2020, with the Korean Intellectual Property Office, the entire publication of which is incorporated herein by reference. Technical Field

[0002] This disclosure relates to methods and apparatus for processing data from neural networks. Background Technology

[0003] Neural networks represent computational architectures that simulate the biological brain. Recently, with the development of neural network technology, various electronic systems have been actively researched to utilize neural network devices to analyze input data and extract useful information. Neural network devices require large amounts of data transmission and computation on complex input data. Due to limited hardware resources, it is difficult to load all the parameters of the entire neural network into the neural network device's computing circuitry at once, or for the computing circuitry to calculate all the results at once. Therefore, only a portion of the neural network's data (e.g., intermediate computation data, weights, training datasets, etc.) can be stored in memory, resulting in high data throughput and large storage space requirements for neural network devices. Therefore, in order to extract desired information by analyzing large amounts of input data in real time using neural networks, techniques capable of efficiently processing the data streams associated with neural networks are needed. Summary of the Invention

[0004] Methods and apparatus for processing data from neural networks are provided. The technical problems to be solved by this disclosure are not limited to those described above; other technical problems can be inferred from the embodiments below.

[0005] Additional aspects will be set forth in part in the description which follows, and will be apparent in part from the description, or may be learned by practice of the disclosed embodiments.

[0006] According to one aspect of an embodiment, a method for processing data from a neural network includes: obtaining one or more bit representations of the data for processing the neural network from a memory and / or a neural network processor; generating a plurality of candidate profiles by dividing the one or more bit representations of the obtained data into one or more channels and applying a compression technique to each channel; determining a final profile by comparing the compression performance of each candidate profile; determining the partitioning method and the compression technique of the one or more channels used in the determined final profile as an optimal configuration for compressing the data of the neural network, and compressing the data of the neural network using the determined optimal configuration; and storing the compressed neural network data in the memory.

[0007] The method may further include: obtaining compressed neural network data from the memory; decompressing the compressed neural network data according to an optimal configuration; and sending the decompressed neural network data to a neural network processor for use in neural network computation.

[0008] According to one aspect of an embodiment, a neural network device includes: a memory; a neural network processor; and an encoder configured to obtain one or more bit representations of data for processing a neural network from the memory and / or the neural network processor; generate a plurality of candidate profiles by dividing the one or more bit representations of the obtained data into one or more channels and applying a compression technique to each channel; determine a final profile by comparing the compression performance of each of the plurality of candidate profiles; determine the partitioning method and the compression technique of one or more channels used in the determined final profile as an optimal configuration for compressing the data of the neural network, and compress the data of the neural network using the determined optimal configuration; and store the compressed neural network data in the memory.

[0009] The neural network device may further include: a decoder configured to: obtain compressed neural network data from the memory; decompress the compressed neural network data according to an optimal configuration; and send the decompressed neural network data to a neural network processor for use in neural network operations.

[0010] According to one aspect of an embodiment, an apparatus for processing data from a neural network includes: a first partitioner configured to: obtain an optimal configuration predetermined by a final profile, obtain a bit representation of the data for processing the neural network from a memory and / or a neural network processor, and partition the bit representation into one or more channels using the obtained optimal configuration, wherein in the final profile, a compression technique is defined for each channel by partitioning the neural network data into one or more channels; a plurality of compressors configured to: compress each of the partitioned channels using the obtained optimal configuration, and output compressed channels; and a first splicer configured to: splice the compressed channels, and output the compressed bit representation to the memory.

[0011] According to one aspect of an embodiment, an apparatus for processing data from a neural network includes: a second partitioner configured to: obtain an optimal configuration predetermined by a final configuration file, obtain a compressed bit representation of compressed data of the neural network from a memory, and partition the compressed bit representation into one or more compressed channels based on the obtained optimal configuration, wherein in the final configuration file, a compression technique is defined for each channel by partitioning the data of the neural network into one or more channels; a plurality of decompressors configured to: decompress each of the partitioned compressed channels based on the optimal configuration, and output a decompressed channel; and a second splicer configured to: splice the decompressed channels, and output the decompressed bit representation to a neural network processor for use in neural network operations.

[0012] According to one aspect of the embodiments, a method for processing data of a neural network includes: obtaining one or more bit representations of the data for processing the neural network; generating a plurality of candidate profiles by dividing the one or more bit representations of the obtained data into one or more channels and applying a compression technique to each channel; determining a final profile by comparing the compression performance of each of the plurality of candidate profiles; and determining the partitioning method and the compression technique of the one or more channels used in the determined final profile as the optimal configuration for compressing the data of the neural network.

[0013] According to one aspect of an embodiment, an apparatus for processing data of a neural network includes: a memory configured to store at least one program; and a processor configured to compress data by executing the at least one program, wherein the processor obtains one or more bit representations of the data for processing the neural network; generates a plurality of candidate profiles by dividing the one or more bit representations of the obtained data into one or more channels and applying a compression technique to each channel; determines a final profile by comparing the compression performance of each of the plurality of candidate profiles; and determines the partitioning method and the compression technique of the one or more channels used in the determined final profile as the optimal configuration for compressing the data of the neural network.

[0014] According to one aspect of the embodiments, a computer-readable non-transitory recording medium includes a recording medium having thereon recorded a program for performing the methods described above on a computer.

[0015] According to one aspect of the embodiments, an apparatus for processing data from a neural network includes: a first partitioner configured to: obtain an optimal configuration predetermined by a final configuration file, obtain a bit representation of the data for processing the neural network, and partition the bit representation into one or more channels using the obtained optimal configuration, wherein in the final configuration file, a compression technique set for each channel is defined by partitioning the neural network data into one or more channels; a plurality of compressors configured to: compress each of the partitioned channels using the obtained optimal configuration, and output a compressed channel; and a first splicer configured to: splice the compressed channels, and output a compressed bit representation.

[0016] According to one aspect of the embodiments, an apparatus for processing data from a neural network includes: a second partitioner configured to: obtain an optimal configuration predetermined by a final configuration file, obtain a compressed bit representation of compressed data from the neural network, and partition the compressed bit representation into one or more compressed channels based on the obtained optimal configuration, wherein in the final configuration file, a compression technique is defined for each channel by partitioning the data from the neural network into one or more channels; a plurality of decompressors configured to: decompress each of the partitioned compressed channels based on the optimal configuration, and output a decompressed channel; and a second splicer configured to: splice the decompressed channels, and output a decompressed bit representation. Attached Figure Description

[0017] The above and other aspects, features, and advantages of the disclosed specific embodiments will become clearer from the following description taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1 This is a diagram illustrating the architecture of a neural network according to an embodiment;

[0019] Figure 2 It is a diagram illustrating the operations performed in a neural network according to an embodiment;

[0020] Figure 3 This is a diagram illustrating a compression pipeline for data compression according to an embodiment;

[0021] Figure 4 This is a block diagram illustrating the hardware configuration of a neural network device according to an embodiment;

[0022] Figure 5 This is a block diagram illustrating the hardware configuration of a neural network device according to another embodiment;

[0023] Figure 6 This is a conceptual diagram illustrating a method for generating configuration files according to an embodiment;

[0024] Figure 7A , Figure 7Band Figure 7C This is a diagram illustrating the statistical distribution based on activation partitioning according to an embodiment;

[0025] Figure 7D , Figure 7E and Figure 7F It is a diagram used to illustrate the statistical distribution based on weights according to an embodiment;

[0026] Figure 8 This is a diagram illustrating a candidate configuration file generation algorithm according to an embodiment;

[0027] Figure 9 This is a diagram illustrating the algorithm for determining the optimal configuration according to an embodiment;

[0028] Figure 10 This is a diagram illustrating an algorithm for determining the optimal compression technique and algorithm parameters according to an embodiment;

[0029] Figure 11 This is a diagram used to describe the compression of bit representation according to an embodiment;

[0030] Figure 12 This is a table used to illustrate various compression techniques according to the embodiments;

[0031] Figure 13 This is a diagram illustrating a method for compression based on an optimal configuration according to an embodiment;

[0032] Figure 14 This is a diagram used to describe an encoder according to an embodiment;

[0033] Figure 15 This is a diagram used to describe the stop code insertion algorithm according to an embodiment;

[0034] Figure 16 This is a diagram used to illustrate compression by an encoder according to an embodiment;

[0035] Figure 17 This is a block diagram used to describe an encoder according to another embodiment;

[0036] Figure 18 This is a diagram used to describe the decoder according to an embodiment;

[0037] Figure 19 This is a block diagram used to describe a decoder according to another embodiment;

[0038] Figure 20 This is a table used to illustrate the statistical characteristics of neural network data according to the embodiments;

[0039] Figure 21 This is a graph used for comparing compression ratios according to an embodiment;

[0040] Figure 22 This is a block diagram illustrating the configuration of an electronic system according to an embodiment;

[0041] Figure 23 This is a flowchart of a method for processing neural network data in a neural network device according to an embodiment. Detailed Implementation

[0042] Referring now to embodiments, examples of which are shown in the accompanying drawings, wherein the same reference numerals always denote the same elements. In this respect, embodiments may take different forms and should not be construed as limited to the description set forth herein. Therefore, embodiments are described below only by reference to the accompanying drawings to explain various aspects. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. When a statement such as “at least one of…” follows a column of elements, the statement modifies the entire column of elements rather than a single element within the column.

[0043] The terminology used herein has been chosen as that commonly used by those skilled in the art in consideration of the functionality of the present embodiments, but may be changed in light of technical intent, precedent, or disclosure of new technologies. Furthermore, in certain cases, some terms are arbitrarily chosen by the applicant, and in such cases, the meaning of the terms will be described in detail in the relevant sections of the specification. Therefore, the terms used in the specification should not be simply defined by their names, but rather based on their meanings and the entirety of the specification.

[0044] In the description of the embodiments, it will be understood that when an element is referred to as being connected to another element, it can include electrical connections when the element is directly connected to the other element and when the element is indirectly connected to the other element through an intermediate constituent element. It should be understood that, unless otherwise defined, when a component “comprises” or “includes” an element in the specification, other elements are not excluded, but may also be included.

[0045] It will also be understood that the terms “comprising” or “including” should not be construed as necessarily including all the constituent elements and operations described in the specification, nor should they be construed as excluding portions of constituent elements or operations of various constituent elements and operations, or including additional constituent elements and operations.

[0046] It will be understood that although the terms "first," "second," etc., may be used herein to describe various constituent elements, these constituent elements should not be limited by these terms. These terms are only used to distinguish one constituent element from another.

[0047] This embodiment relates to a method and apparatus for compressing data included in a neural network, and detailed descriptions of subjects widely known to those skilled in the art to which the following embodiments pertain are omitted.

[0048] The description of the embodiments should not be construed as limiting the scope of the claims, and embodiments that are readily inferred by those skilled in the art from the specific implementation and embodiments will be interpreted as being included in the inventive concept. Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings.

[0049] Figure 1 This is a diagram illustrating the architecture of a neural network 1 according to some embodiments.

[0050] Reference Figure 1 Neural network 1 can be represented by a mathematical model using nodes and edges. Neural network 1 may include a deep neural network (DNN) or an n-layer neural network architecture. A DNN or an n-layer neural network may correspond to a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzmann machine, etc. For example, neural network 1 may be implemented as a CNN, but is not limited to this. Figure 1 Neural network 1 can correspond to some layers of a CNN. Therefore, neural network 1 can correspond to convolutional layers, pooling layers, or fully connected layers of a CNN. However, for convenience, in the following description, it is assumed that neural network 1 corresponds to a convolutional layer of a CNN.

[0051] In a convolutional layer, the first feature map FM1 corresponds to the input feature map, and the second feature map FM2 corresponds to the output feature map. Feature maps can represent datasets showcasing various characteristics of the input data. The first feature map FM1 and the second feature map FM2 can be two-dimensional or higher-dimensional high-dimensional matrices, each with its own activation parameters. When the first feature map FM1 and the second feature map FM2 correspond to, for example, three-dimensional feature maps, they have a width W (or columns), a height H (or rows), and a depth C. In this case, the depth C can correspond to the number of channels.

[0052] In the convolutional layer, a convolution operation is performed on the first feature map FM1 and the weight map WM, resulting in the generation of the second feature map FM2. The weight map WM can filter the first feature map FM1 and is referred to as a filter or kernel. In one example, the depth (i.e., the number of pathways) of the weight map WM is the same as the depth C (i.e., the number of pathways) of the first feature map FM1. The weight map WM is shifted by traversing the first feature map FM1 as a sliding window. In each shift, the weights included in the weight map WM are multiplied and summed with all the feature values ​​in the region overlapping with the first feature map FM1. As the first feature map FM1 is convolved with the weight map WM, one pathway of the second feature map FM2 is generated.

[0053] exist Figure 1 Although a weight map WM is depicted, multiple pathways for generating a second feature map FM2 can be generated by convolving multiple weight maps with a first feature map FM1. The second feature map FM2 of the convolutional layer can be the input feature map of the next layer. For example, the second feature map FM2 can be the input feature map of a pooling layer. This embodiment is not limited to this.

[0054] Figure 2 This is a diagram illustrating the operations performed in neural network 2 according to some embodiments.

[0055] Reference Figure 2 The neural network 2 has a structure including an input layer, a hidden layer and an output layer. The neural network 2 performs operations based on the received input data (e.g., I1 and I2) and can generate output data (e.g., O1 and O2) based on the results of the operations.

[0056] As mentioned above, neural network 2 can be an n-layer neural network or a DNN comprising two or more hidden layers. For example, such as Figure 2 As shown, neural network 2 can be a DNN comprising an input layer (layer 1), two hidden layers (layers 2 and 3), and an output layer (layer 4). When neural network 2 is implemented as a DNN architecture, it includes a larger number of layers capable of processing effective information; therefore, compared to a neural network with a single layer, neural network 2 can handle a larger number of complex datasets. However, although neural network 2 is shown as comprising four layers, this is merely an example, and neural network 2 may include fewer or more layers, or fewer or more pathways. That is, neural network 2 may include... Figure 2 The diagram shows layers with various structures.

[0057] Each layer included in neural network 2 may include multiple pathways. A pathway may correspond to multiple artificial nodes, which are referred to as neurons, processing elements (PEs), units, or similar terms. For example, as... Figure 2 As shown, layer 1 may include two pathways (nodes), and each of layers 2 and 3 may include three pathways (e.g., CH1, CH2, and CH3). However, this is just an example, and each layer included in neural network 2 may include a variety of numbers of pathways (nodes).

[0058] The pathways included in each layer of the neural network 2 can be connected to each other to process data. For example, one pathway can receive data from other pathways for operation and output the result of the operation to other pathways.

[0059] Each input and output of each path can be referred to as input activation and output activation. That is, activation can be the output of a path and can be a parameter corresponding to the input of a path included in the next layer. Meanwhile, each path can determine its own activation based on the activations and weights received from the paths included in the previous layer. Weights are parameters used to operate on the output activation in each path and can be values ​​assigned to the connection relationships between paths.

[0060] Each path can be processed by a computing unit or processing element that is activated by receiving input and outputting output, and the input-output of each path can be mapped. For example, when σ is an activation function, It is the weight from the k-th path included in layer (i-1) to the j-th path included in layer i. It is the deviation of the j-th pathway included in the i-th layer. When the j-th pathway in the i-th layer is activated, the activation can be calculated using Equation 1 below, where i is a positive integer greater than 1, and j and k are positive integers.

[0061] Equation 1

[0062]

[0063] like Figure 2 As shown, the activation of the first pathway CH1 in the second layer (layer 2) can be represented as Furthermore, according to Equation 1, It can have a value The activation function σ can be a rectified linear unit (ReLU, also known as a linear rectified function), but it is not limited to this. For example, the activation function σ can be a sigmoid function, a hyperbolic tangent function, a maxout function, etc.

[0064] As mentioned above, in Neural Network 2, large datasets are exchanged between multiple interconnect pathways, and many computational processes are performed across multiple layers. Therefore, there is a need for techniques that can reduce the memory required to process the complex data of Neural Network 2.

[0065] On the other hand, data compression can reduce the memory traffic of neural network 2 operations and improve the performance of neural network 2 operations that are limited by memory bounds by compressing neural network data. Here, the memory bound can mean that the time to complete an operation is determined by the amount of memory.

[0066] Neural network data is data used in various processes related to machine learning through the use of neural networks, and can include all static and dynamic data used or processed in the neural network processes, regardless of whether the data values ​​change. Neural network data can include all data with varying degrees of precision.

[0067] Neural network data can include all data (such as values ​​assigned to the connections of the aforementioned layers, input and output values ​​used in the various stages of processing using the neural network (such as the result of a multiply-accumulate (MAC) operation), and preset values ​​used for processing), without restrictions on the representation format or timing. For example, neural network data can include, but is not limited to, at least one of the activations, weights, and gradients of the neural network, and can be data values ​​used or processed in the inference or learning processes of the neural network.

[0068] This embodiment provides a method for compressing various neural network data without limiting the representation format and usage time, thereby reducing memory access burden and performing general compression in different neural networks.

[0069] The method for compressing neural network data used in neural network 2 will be described in detail below with reference to the accompanying drawings.

[0070] Figure 3 This is a diagram illustrating a compression pipeline 3 for data compression according to an embodiment.

[0071] Reference Figure 3 The compression pipeline 3 includes a lossy conversion block 31 and a lossless compression block 32. Because the compression pipeline 3 is used, memory throughput can be reduced and the performance of memory-bound machine learning workloads can be improved. For example, 32-bit floating-point full-precision data can be compressed for training or inference using lossy conversion and / or lossless compression. Lossy conversion compresses data by reducing the amount of information in the data, while lossless compression compresses data by increasing information density.

[0072] Lossy conversion techniques include pruning, singular value decomposition (SVD), and quantization, while lossless compression techniques include Huffman coding, run-length coding on zeros (Z-RLC), and zero-value compression (ZVC).

[0073] However, the hardware complexity of lossless compression techniques varies greatly, and some compression techniques are not feasible in hardware with limited resources, such as embedded systems. As a result, there is a need for lightweight lossless compression techniques that can reliably achieve high compression ratios at low hardware costs.

[0074] Meanwhile, the overall compression ratio of the neural network data can be the compression ratio of lossy transformation (P) and lossless compression (C) (e.g., overall compression ratio = P × C). However, in the following description, it is assumed that the compression ratio is the compression ratio of lossless compression. However, the compression method according to the embodiments is not limited to this, and of course, the overall compression ratio can be improved by combining various lossy transformation, lossless compression and / or quantization methods.

[0075] For example, quantization can improve the overall compression ratio by combining it with lossless compression. Quantization effectively reduces the random and highly entropy-laden meaningless information in neural network data. Therefore, after neural network data is quantized, the amount of information is relatively small, but more meaningful information is retained, thus improving the compression ratio.

[0076] Figure 4 This is a block diagram illustrating the hardware configuration of a neural network device according to an embodiment.

[0077] Reference Figure 4 The neural network device 4 includes a processor 410 and a memory 420. Figure 4 The neural network device 4 shown herein only includes components relevant to this embodiment. Therefore, it will be clear to those skilled in the art that the neural network device 4 may also include components other than those described herein. Figure 4 Other general components besides those shown in the diagram.

[0078] The neural network device 4 can correspond to a computing device with various processing functions, such as generating neural networks, training (or learning) neural networks, analyzing the statistical properties of neural network data, determining a profile for compressing neural network data, dividing neural network data into multiple lanes and compressing each of the multiple lanes individually, performing inference using the neural network, or retraining the neural network. For example, the neural network device 4 can be implemented as various types of devices, such as personal computers (PCs), server devices, and mobile devices. Typically, in the actual neural network operation of a processor, intermediate data of the outputs of each layer of the neural network needs to be stored and reused in the operation of subsequent layers. Due to the limited storage space of the processor's internal memory, it is usually necessary to store the data of the aforementioned layers (e.g., activations, weights, gradients, etc.) in external memory and read this data from external memory during subsequent operations. This results in a large amount of data transfer between the processor and external memory. In embodiments according to this disclosure, data read from memory 420 (e.g., weights, input activations of the initial layer, training dataset, etc.) and / or data generated in the neural network operation can be compressed via processor 410 and stored in memory 420. When processor 410 performs neural network operations (e.g., performing some or all of the operations in one or more layers or one or more nodes), processor 410 can read compressed data required for performing some or all of the operations in the one or more layers or one or more nodes from memory 420, and decompress the read compressed data for use in neural network operations. Therefore, data transmission bandwidth and external memory storage space can be saved, while reducing the cost and power consumption of the neural network device.

[0079] The neural network device 4 can determine the optimal configuration for compressing neural network data and perform compression, taking into account the processing performance of the device on which the neural network is deployed (e.g., a mobile device, an embedded device, etc.). Devices on which the neural network is deployed include, but are not limited to, autonomous vehicles, robots, smartphones, tablets, augmented reality (AR) devices, and Internet of Things (IoT) devices that use neural networks to perform speech recognition, video recognition, etc.

[0080] According to this embodiment, in addition to neural network device 4, there may be additional neural network devices. Neural network device 4 and the additional neural network devices can each perform separate functions for compressing neural network data. For example, neural network device 4 can determine a configuration file for compressing neural network data, and other neural network devices can use this configuration file to compress the neural network data. However, in the following description, for ease of explanation, it is assumed that one neural network device 4 performs all functions, and the same assumption will apply to what will be described later. Figure 5 .

[0081] The configuration file can represent a collection of information including methods for processing the bit representation of neural network data and information about various compression techniques used for the bit representation in compression processing.

[0082] The processor 410 can generate multiple candidate profiles by obtaining one or more bit representations of neural network data, and determine the final profile among the candidate profiles for compressing the neural network data.

[0083] Candidate profiles can be represented as profiles generated as candidates to be selected as the final profile and with different conditions.

[0084] The final configuration file can represent a configuration file that includes final information about how neural network device 4 compresses neural network data.

[0085] Processor 410 can determine the compression technique and partitioning method for one or more channels used in the final configuration file as the optimal configuration for compressing neural network data. Once the optimal configuration is determined, the optimal method for compressing the neural network data can be determined.

[0086] Processor 410 can divide neural network data into one or more bit segments according to an optimal configuration and compress each bit segment using an optimal compression technique. The statistical characteristics of each bit segment can be revealed by dividing the neural network data into multiple bit segments. Since processor 410 applies appropriate optimal compression techniques to the revealed statistical characteristics of each bit segment, the compression ratio can be improved.

[0087] When a data value is divided into multiple bits, a bit field can represent a group of bits that form part of each data value.

[0088] Here, a bit field can be referred to as a channel. The technique used to compress multiple channels by applying a compression technique suitable for each channel is called channel compression.

[0089] The following will describe in detail, with reference to the accompanying drawings, the method by which the processor 410 determines the optimal configuration for compressing neural network data, and the specific method for compressing neural network data according to the determined optimal configuration.

[0090] Processor 410 performs overall functions for controlling neural network device 4. For example, processor 410 controls the overall operation of neural network device 4 by executing a program stored in memory 420 within neural network device 4. Processor 410 may be implemented as a central processing unit (CPU), graphics processing unit (GPU), or application processor (AP) disposed in neural network quantization device, but is not limited thereto.

[0091] Memory 420 is hardware that stores various data processed in neural network device 4. For example, memory 420 may store data processed in neural network device 4 and data that will be processed in neural network device 4. Furthermore, memory 420 may store applications, drivers, etc., that will be driven by neural network device 4. Memory 420 may be DRAM, but is not limited to it. Memory 420 may include at least one of volatile memory and non-volatile memory. Non-volatile memory includes read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memory includes dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, FeRAM, etc. In one embodiment, the memory 420 may include a hard disk drive (HDD), a solid-state drive (SSD), a compact flash memory (CF), a secure digital card (SD), a micro-secure digital card (micro-SD), a mini-secure digital card (mini-SD), an extreme digital card (xD), and a memory stick.

[0092] Meanwhile, the memory 420 can store, for example, neural network data, and can store various programs related to the training algorithm, compression algorithm, configuration file algorithm, etc. of the neural network to be executed by the processor 410.

[0093] A configuration file algorithm can be defined as an algorithm that uses configuration files to define a method for compressing neural network data.

[0094] Figure 5 This is a block diagram illustrating the hardware configuration of a neural network device 5 according to another embodiment.

[0095] Reference Figure 5The neural network device 5 includes a neural processor 510 and a memory 520. The neural processor 510 may also include a neural network processor 512, an encoder 514, and a decoder 516. The neural processor 510 and memory 520 of the neural network device 5 can perform operations related to… Figure 4 The processor 410 and memory 420 have the same function.

[0096] The neural processor 510 can perform operations for driving the aforementioned neural network. For example, these operations may be those required for inference, training, and retraining based on input data. The neural processor 510 can compress the neural network data used for these operations using an optimal configuration, and can also decompress the compressed neural network data using an optimal configuration.

[0097] The neural network processor 512 can control the encoder 514 and the decoder 516. For example, the neural network processor 512 can control the encoder 514 to compress neural network data and control the decoder 516 to decompress the compressed neural network data. In one example, the encoder 514 can obtain neural network data (e.g., neural network activations, weights, gradients, training datasets, etc.) from the memory 520 and / or the neural network processor 512. The encoder 514 can compress the obtained neural network data and store the compressed neural network data in the memory 520. When the neural network processor 512 performs neural network operations (e.g., performing some or all of the operations of one or more layers or one or more nodes), the neural network processor 512 can read the compressed neural network data required to perform some or all of the operations of the one or more layers or one or more nodes from the memory 520. The decoder 516 can decompress the compressed neural network data and send the decompressed neural network data to the neural network processor 512 for neural network operations.

[0098] Reference Figures 14 to 19 A detailed description of compression and decompression using encoder 514 and decoder 516.

[0099] at the same time, Figure 5 The neural network device 5 can be the neural network device 4 described above. Figure 4 ), or additional neural network devices besides neural network device 4.

[0100] Figure 6 This is a conceptual diagram illustrating the compression method according to an embodiment. (Refer to...) Figure 6 Processor 410 (reference) Figure 4 An executable configuration file can be used to compress neural network data.

[0101] Processor 410 may access memory 420 (see reference) Figure 4 Obtain the bit representation 6 of the neural network data. The obtained bit representation 6 can be a bit stream. For example, the neural network data may include neural network 2 (see reference 2). Figure 2 The processor 410 can obtain a bitstream of the weight data of the neural network 2, which includes at least one of the activation, weights, and gradients of the neural network 2.

[0102] Neural network data used for processing neural networks can be represented as a bitstream consisting of 10 bits (b0, b1, b2, b3, b4, b5, b6, b7, b8, and b9). Here, for illustration purposes, the bit representation of neural network data is shown as a bitstream consisting of 10 bits, but the data format of neural network data and the number of bits constituting the bitstream are not limited to this.

[0103] On the other hand, although only one bit representation 6 is shown for illustration, processor 410 can obtain multiple bit representations. In a neural network, there can be a large amount of data defined by such bit representations. Processor 410 can determine the final profile by obtaining a single bit representation, and can also determine the final profile by obtaining multiple bit representations.

[0104] The processor 410 can generate multiple candidate profiles by dividing one or more bit representations of the acquired data into one or more channels and applying compression techniques to each channel.

[0105] For example, processor 410 can generate candidate profiles 61, 62, 63, and 64 by dividing bit representation 6 into one or more channels and applying compression techniques to each channel. However, in Figure 6 For ease of description, only four candidate profiles 61, 62, 63 and 64 are shown, but in reality, a variety of number of candidate profiles can be generated.

[0106] A channel can represent a bit segment with a predetermined bit width generated by dividing a bit stream. For example, in the bit representation 6 of configuration file 1 (61), channel 0 is a bit segment with a bit width of 2, channel 1 is a bit segment with a bit width of 3, channel 2 is a bit segment with a bit width of 4, and channel 3 can be a bit segment with a bit width of 1.

[0107] Processor 410 can generate candidate profiles 61, 62, 63 and 64 by applying at least one of partitioning methods and compression techniques for dividing a bit representation into one or more channels.

[0108] Candidate profiles 61, 62, 63, and 64 may include all combinations of methods for dividing bit representation 6. For example, candidate profiles 61, 62, 63, and 64 may include multiple profiles in which the number of channels for dividing bit representation 6 into multiple bit segments and the bit width of the bit segment corresponding to each channel are set differently.

[0109] In configuration file 1 (61), the number of channels is 4, and the bit widths corresponding to each channel are 2, 3, 4, and 1. In configuration file 2 (62), the number of channels is 3, and the bit widths corresponding to each channel are 3, 1, and 6. In configuration file n-1 (63), the number of channels is 5, and the bit widths corresponding to each channel are 1, 3, 2, 3, and 1. In configuration file n (64), the number of channels is 3, and the bit widths corresponding to each channel are 1, 3, and 6. Figure 6 Only one configuration, which differs from the other, is shown: the number of channels and the bit width corresponding to each channel. However, configurations may also be included that have the same number of channels and different compression techniques, even if the bit width is the same for each channel.

[0110] Candidate profiles 61, 62, 63, and 64 may include all combinations of multiple compression techniques that can be assigned to each channel partitioned by any of the partitioning methods.

[0111] For example, in candidate profiles 61, 62, 63, and 64, compression techniques can be applied independently to each channel. a Compression techniques can be applied to channel 0, C of configuration file 2 (62). b Compression technology can be applied to channel 1, C a The compression technique can be applied again to channel 2. Because the compression technique is applied independently to each channel, the same compression technique can be applied to different channels.

[0112] For example, in candidate profiles 61, 62, 63, and 64, different compression techniques can be applied to each channel. In profile n(64), multiple compression methods (C... a C b C c C d The different Cs in (and none) a C b The `none` option can be applied to each channel. `none` can indicate a compression technique that does not compress the channel.

[0113] Candidate configuration files 61, 62, 63, and 64 may include configuration files that apply compression techniques with different algorithm parameters.

[0114] Algorithm parameters represent the parameters used when employing a specific compression technique. Each compression technique can have corresponding algorithm parameters, and multiple compression techniques can have different algorithm parameters. For example, the maximum sequence can be an algorithm parameter for Zero-Run-Length Encoding (Z-RLC), the matrix size can be an algorithm parameter for Compressed Sparse Columns / Rows (CSC / CSR), and the block size can be an algorithm parameter for Dynamic Precision Reduction (DPRed).

[0115] However, some compression techniques may not have corresponding algorithm parameters. For example, zero-value compression (ZVC) may not have corresponding algorithm parameters.

[0116] The values ​​of the algorithm parameters can have a controllable range. When the values ​​of the algorithm parameters are controlled differently, the compression performance and the required memory resources can vary. For example, referring to configuration file 2(62), compression technology C... a It is applied to both channel 0 and channel 2, but the value of each algorithm parameter is as follows: C a1 and C a20 But they are different from each other.

[0117] Processor 410 can determine the final profile by comparing the compression performance of each of the generated candidate profiles 61, 62, 63, and 64. Compression performance may include, but is not limited to, the time complexity, required memory space, compression ratio, and statistical characteristics of each profile.

[0118] For example, processor 410 can calculate the compression ratio corresponding to each of candidate profiles 61, 62, 63, and 64, and can determine the candidate profile with the highest compression ratio among the calculated compression ratios as the final profile. Processor 410 can calculate that profile 1 (61) has a compression ratio of 40%, profile 2 (62) has a compression ratio of 20%, profile n-1 (63) has a compression ratio of 10%, and profile n (64) has a compression ratio of 50%. If the compression ratio of the profiles between profile 2 (62) and profile n-1 (63) is less than 50%, then profile n (64) will be determined as the final profile because its compression ratio is the highest among the candidate profiles.

[0119] Figure 7A , Figure 7B and Figure 7C This is a graph used to illustrate the statistical distribution based on activation partitioning according to an embodiment.

[0120] Reference Figures 7A to 7CCandidate profiles 61, 62, 63, and 64 may include profiles with different sparsity distributions of channels based on the number of channels and the bit width of the channels. In other words, different parts of the neural network data may have different sparsity distributions. Figure 7A This illustrates the sparsity distribution when only one channel exists. Second... Figure 7B The diagram illustrates the sparsity distribution when there are three channels, each with a bit width of 1, 2, and 6. The third... Figure 7C The sparsity distribution is shown when there are four channels and each bit width is 1, 2, 2, and 4.

[0121] When the second Figure 7B Channel 0 and Channel 1 are connected to the first Figure 7A When comparing channel 0, the second can be seen Figure 7B Channels 0 and 1 tend to be very sparse. When the third... Figure 7C Channel 2 and the first Figure 7A When comparing channel 0, the third can be seen Figure 7C Channel 2 still tends to be sparse. That is to say, the first Figure 7A The most significant bit (MSB) of channel 0 (undivided) tends to be very sparse, and the sparsity decreases as we move toward the least significant bit (LSB).

[0122] In this way, the sparsity distribution of each channel can vary depending on the partitioning method. Since channels with high sparsity are easier to compress than channels with low sparsity, the compression ratio can be improved depending on the partitioning method.

[0123] Candidate profiles 61, 62, 63, and 64 may include profiles that apply compression techniques that do not compress the least significant bit (LSB) of the channel. Second Figure 7B Channel 2 and the third Figure 7C Channel 3 may have a higher capacity than the first channel. Figure 7A Channel 0 has high sparsity. Data with low sparsity can have low compression ratios or larger data volumes. Therefore, conversely, compression ratios can be improved by compressing only the channels with high sparsity and by not compressing the channels with low sparsity.

[0124] Figures 7D to 7F It is a graph used to illustrate the statistical distribution based on weights according to the embodiments.

[0125] Reference Figures 7D to 7F It can be seen that even if the type of neural network data is weights, neural network data also has the characteristics of weights. Figures 7A to 7C The sparse distribution is similar to the sparse distribution.

[0126] fourth Figure 7DThis illustrates the sparsity distribution when only one channel exists. Fifth. Figure 7E The sixth figure illustrates the sparsity distribution when there are three channels, each with a bit width of 1, 2, and 7. Figure 7F The sparsity distribution is shown when there are four channels and each has a bit width of 1, 2, 3 and 4.

[0127] When the fifth Figure 7E Channel 0 and Channel 1 with the fourth Figure 7D When comparing channel 0, the fifth... Figure 7E Channels 0 and 1 tend to be very sparse. When the sixth... Figure 7F Channel 2 and the fourth Figure 7D When comparing channel 0, the sixth can be seen. Figure 7F Channel 2 still tends to be sparse.

[0128] On the other hand, Figure 7D In the bitstream, the most significant bit (MSB) corresponding to the sign bit b0 is not omitted. Figure 7A In this approach, the sign bit b0 of the bit stream is omitted, thus allowing for variations in the shape of the sparsity distribution. For example, if the neural network data values ​​are always positive, the sign bit can be omitted.

[0129] Figure 8 This is a diagram illustrating the candidate configuration file generation algorithm 8 according to an embodiment.

[0130] Reference Figure 8 Candidate configuration file generation algorithm 8 can be used to generate references. Figure 6 The candidate configuration files are described.

[0131] In operation 81, processor 410 (refer to...) Figure 4 Obtain neural network data, bit width of bit representation, compression techniques, and algorithm parameters corresponding to each compression technique.

[0132] In operation 82, processor 410 generates all partitioning methods in which the number of channels and / or the bit width of each channel are different.

[0133] In operation 83, processor 410 generates candidate profiles with different compression techniques and / or algorithm parameters for all partitioning methods.

[0134] In operation 84, processor 410 outputs the results of generating candidate configuration files.

[0135] Figure 9 This is a diagram illustrating the optimal configuration determination algorithm 9 according to an embodiment.

[0136] Reference Figure 9 Algorithm 9, the optimal configuration determination algorithm, can be used to determine the reference configuration. Figure 6 The final configuration file and optimal configuration are described.

[0137] In operation 91, processor 410 (refer to...) Figure 4 ) obtain in Figure 8 The candidate configuration file generated in Algorithm 8 includes the candidate configuration file, the bit width of the bit representation, the compression technique, the symbol-based compression technique, the algorithm parameters corresponding to each compression technique, and the constraints when implemented in hardware.

[0138] In operation 92, processor 410 (refer to...) Figure 4 Obtain the set of all partitioning methods that vary the number of channels and / or the bit width of each channel.

[0139] In operation 93, processor 410 considers the set of all partitions obtained in operation 92 and the constraints obtained in operation 91 to determine the final configuration file and the optimal configuration.

[0140] In operation 94, processor 410 outputs the optimal configuration.

[0141] Figure 10 This is a diagram illustrating an algorithm 10 for determining the values ​​of optimal compression techniques and algorithm parameters according to an embodiment.

[0142] Reference Figure 10 When Figure 9 When determining the final configuration file in operation 93, algorithm 10, which is used to determine the optimal compression technology and the values ​​of the optimal algorithm parameters, can be used.

[0143] In operation 100, processor 410 repeatedly compares the compression performance of all possible algorithm parameter values ​​for all compression techniques and determines the optimal compression technique and optimal algorithm parameter values.

[0144] Figures 8 to 10 Algorithms 8, 9, and 10 are described as examples of determining the optimal configuration according to this embodiment, and the algorithms used to determine the optimal configuration are not limited to those described above. Figures 8 to 10 Algorithms 8, 9 and 10.

[0145] Figure 11 This is a diagram used to describe the compression of bit representation according to an embodiment.

[0146] Reference Figure 11 You can refer to the above. Figure 6The described method for determining the optimal configuration is used to determine the optimal configuration for compressing neural network data with a first-bit representation 110 having a 10-bit width. For example, the optimal configuration can be determined by dividing the first-bit representation 110 sequentially from the most significant bit (MSB) into four channels 1100, 1102, 1104, and 1106 with bit widths of 2, 3, 1, and 4, and by compressing each channel via a first compression technique, a second compression technique, a third compression technique, and a fourth compression technique, respectively.

[0147] Processor 410 or 510 can use optimal configuration to compress channels 1100, 1102, 1104, 1106 representing bit 110 and channels 1110, 1112, 1114, and 1116 representing bit 111, and can output compressed channels 1130, 1132, 1134, 1136, 1140, 1142, 1144, and 1146.

[0148] Processor 410 or 510 can use an optimal configuration to compress the partitioned channels 1100, 1102, 1104, and 1106 of the first bit representation 110, and can output compressed channels 1130, 1132, 1134, and 1136. For example, processor 410 or 510 can output a first compressed channel 1130 by compressing the first channel 1100 of the first bit representation 110 using a first compression technique, output a second compressed channel 1132 by compressing the second channel 1102 of the first bit representation 110 using a second compression technique, output a third compressed channel 1134 by compressing the third channel 1104 of the first bit representation 110 using a first compression technique, and output a fourth compressed channel 1136 by compressing the fourth channel 1106 of the first bit representation 110 using a third compression technique.

[0149] Processor 410 or 510 can compress the partitioned channels 1110, 1112, 1114, and 1116 of the second bit representation 111 using optimal configuration, and output compressed channels 1140, 1142, 1144, and 1146. For example, processor 410 or 510 can output a first compressed channel 1140 by compressing the first channel 1110 of the second bit representation 111 using a first compression technique, output a second compressed channel 1142 by compressing the second channel 1112 of the second bit representation 111 using a second compression technique, output a third compressed channel 1144 by compressing the third channel 1114 of the second bit representation 111 using the first compression technique, and output a fourth compressed channel 1146 by compressing the fourth channel 1116 of the second bit representation 111 using the third compression technique.

[0150] The first bit, representing 110, and the second bit, representing 111, are divided and compressed in the same optimal configuration, but the values ​​of each bit representation can be different from each other. Even if the same compression technique (the first compression technique) is applied to the same channels 1110 and 1100 according to the values ​​of each bit representation, the values ​​of compressed channels 1140 and 1130 can still be different from each other. Compressed channels 1140 and 1130 with different values ​​can have different bit widths and different compressed channel values, can have the same bit width and different compressed channel values, or can have the same compressed channel values ​​and different bit widths.

[0151] Meanwhile, compressed channels 1130, 1132, 1134, 1136, 1140, 1142, 1144, and 1146 only represent the output produced by applying compression techniques to channels 1100, 1102, 1104, 1106, 1110, 1112, 1114, and 1116. Therefore, there are no restrictions on the format of the information included in each compressed channel. For example, depending on the compression technique used, a compressed channel may not include any information, may have an increased bit width, a decreased bit width, a constant bit width, or may include the same value as the target channel.

[0152] Various compression techniques can compress neural network data by reducing redundancy. Neural network data can include multiple redundant pieces of information. These compression techniques reduce the size of neural network data by using various methods to compress the redundant information included in the neural network data.

[0153] Multiple compression techniques can have different compression units used in each technique. For example, the compression unit used in each technique can be a symbol or a sequence of symbols.

[0154] The symbol represents the reference unit when compression is applied to the compression target. For example, the first channels 1100 and 1110 using the first compression technique can use a bit segment with a bit width of 2 as the symbol, and the second channels 1102 and 1112 using the second compression technique can use a bit segment with a bit width of 3 as the symbol.

[0155] A symbol sequence can represent a reference unit when compression is applied to a compression target. A symbol sequence can also represent a sequence of symbols arranged in several sequential compression processes. Here, a compression process can represent applying a compression technique once to a bit representation. For example, when the first channel 1100 of the first bit representation is input in a first compression process and the first channel 1110 of the second bit representation is input in a second compression process, the first compression technique can use the sequential first channels 1100 and 1110 as a symbol sequence.

[0156] Multiple compression techniques can be referenced individually based on the compression unit used in each compression technique. Compression techniques that use symbols can be called symbol-based compression techniques, and compression techniques that use sequences of symbols can be called sequence-based compression techniques.

[0157] According to this embodiment, symbol-based compression techniques and / or sequence-based compression techniques can be used, regardless of the characteristics of the input neural network data. Optimal compression ratios can be achieved by dividing the bit representation of the neural network data into multiple channels and using symbol-based compression techniques and / or sequence-based compression techniques suitable for the characteristics of each channel.

[0158] Each compression technique can have corresponding algorithm parameters, and multiple compression techniques can have different algorithm parameters. For example, the maximum sequence can be an algorithm parameter for zero-run-length encoding (Z-RLC), the matrix size can be an algorithm parameter for compressed sparse columns / rows (CSC / CSR), and the block size can be an algorithm parameter for dynamic precision reduction (DPRed). However, some compression techniques may not have corresponding algorithm parameters. For example, zero-value compression (ZVC) may not have corresponding algorithm parameters.

[0159] Processors 410 or 510 can concatenate compressed channels 1130, 1132, 1134, and 1136 and output a compressed bit representation 113. For example, processors 410 or 510 can output compressed bit representations 113 and 114.

[0160] Furthermore, it is depicted that in the compressed bit representations 113 and 114, compressed channels 1130, 1132, 1134, 1136, 1140, 1142, 1144, and 1146 are combined in the order of channels 1100, 1102, 1104, and 1106. However, the order in which compressed channels 1130, 1132, 1134, 1136, 1140, 1142, 1144, and 1146 are combined is not restricted. However, in the following description, for ease of explanation, it is assumed that the compressed channels are combined in the order of the channels.

[0161] Figure 12 This is a table used to describe various compression techniques according to embodiments.

[0162] Reference Figure 12 Various compression techniques can be used, including None, ZVC, run-length coding (RLC), Z-RLC, and DPRed.

[0163] As referenced above Figure 6 The term "None" can represent a compression technique that does not compress the channel. Because the channel is not compressed, the compressed channel can have the same values ​​as the channel before compression.

[0164] ZVC may not have corresponding algorithm parameters, and may not have a range of values ​​for the algorithm parameters.

[0165] The algorithm parameters corresponding to RLC can be the maximum sequence 2. p Furthermore, the range of algorithm parameters can be 1≤p≤32.

[0166] The algorithm parameters corresponding to Z-RLC can be the maximum sequence 2. p Furthermore, the range of algorithm parameters can be 1≤p≤32.

[0167] When compressing a channel with a DRed, the processor 410 or 510 can compress the channel by adding a bit to the most significant bit (MSB). A DRed with a bit added to the MSB is called a sparse DRed (S-DPRed). The added bit indicates whether all zeros are assigned to the block to be compressed. Because it can indicate whether all zeros are assigned, the compression ratio of sparse data can be improved.

[0168] Processors 410 or 510 can remove the zero-bit mask when compressing channels with DPRed. A DPRed with the zero-bit mask removed can be referred to as a dense DPRed (D-DPRed). Because the zero-bit mask is removed, the compression ratio of dense data can be improved.

[0169] The algorithm parameters corresponding to D-DPRed and S-DPRed can be the block size p, and the range of the algorithm parameters can be 1≤p≤8.

[0170] Figure 13 This is a diagram illustrating a method for compression based on an optimal configuration according to an embodiment.

[0171] Reference Figure 13 Processor 410 or 510 can determine the bit representation applied to the neural network data (e.g., v). t+1 and v t The optimal configuration is as follows: the number of channels is 4; the bit widths of each channel (channel 0, channel 1, channel 2, and channel 3) are 2 bits, 1 bit, 3 bits, and 4 bits, respectively; the compression techniques applied to each channel are Z-RLC, Z-RLC, D-DPRed, and None, respectively; and the algorithm parameter values ​​applied to each channel are 2. 12 2 9 And 3.

[0172] Simultaneously, the neural network device 5 can compress or decompress neural network data based on an optimal configuration predetermined through the configuration file described above. In the following text, reference will be made to... Figures 14 to 16 Describe methods for compressing or decompressing neural network data.

[0173] Figure 14 This is a diagram used to describe the encoder 140 according to an embodiment.

[0174] Reference Figure 14 The encoder 140 may include a first divider 142, a plurality of compression units 144a, 144b and 144c, and a first splicer 146. The encoder 140 may be compared with a reference... Figure 5 The constituent elements corresponding to the encoder 514 described.

[0175] The encoder 140 can divide neural network data into multiple channels based on a predetermined optimal configuration, and use the optimal configuration to compress each of the divided channels.

[0176] The first partitioner 142 can obtain an optimal configuration predetermined by the final configuration file, wherein the compression technique set for each channel is defined by dividing the neural network data into one or more channels. For example, the first partitioner 142 can be via the neural network processor 512 (see reference 142). Figure 5 From memory 520 (reference) Figure 5 The optimal configuration can be obtained, and control signals generated by the neural network processor 512 based on the optimal configuration can be obtained.

[0177] Meanwhile, the method for obtaining the optimal configuration through the first divider 142 is not limited to the method described above.

[0178] The first divider 142 can obtain a bit representation of the data used to process the neural network, and can divide the bit representation into one or more channels by using an optimal configuration.

[0179] The first divider 142 can extract information required for compression techniques used in multiple compression units 144a, 144b, and 144c from multiple bit representations. The extracted information can be sent to compression units 144a, 144b, and 144c that require the extracted information. For example, the first divider 142 can extract leading zero bits required for null suppression (NS) techniques.

[0180] Multiple compression units 144a, 144b, and 144c can be optimally configured to compress each of the divided channels and output a compressed channel. For example, multiple compression units 144a, 144b, and 144c can compress the divided channels using a compression technique applied to each channel in an optimal configuration for each compression unit, and output a compressed channel. Meanwhile, in Figure 14For ease of explanation, only three compression units 144a, 144b and 144c are shown in the encoder 140, but N (N is a positive integer) compression units may be included in the encoder 140.

[0181] Multiple compression units 144a, 144b, and 144c can perform multiple compression techniques for each compression unit. Because each compression unit performs multiple compression techniques, each compression unit can perform the appropriate compression technique even when the optimal configuration changes.

[0182] Multiple compression units 144a, 144b, and 144c can be matched one-to-one with the optimal configuration and the allocated channels. For example, the first compression unit 144a is matched one-to-one with channel 1 and performs the first compression technique, and the second compression unit 144b is matched one-to-one with channel 2 and performs the second compression technique.

[0183] The first splicer 146 can splice multiple compressed channels and output a compressed bit representation. The first splicer 146 can splice multiple compressed channels. The multiple compressed channels spliced ​​by the first splicer 146 can be referred to as spliced ​​channels. The first splicer 146 can generate a compressed bit representation 148 based on the spliced ​​channels and output the compressed bit representation.

[0184] The first splicer 146 can insert stop codes used to synchronize compressed channels output from multiple compression units 144a, 144b, and 144c into the spliced ​​channel. For example, the first splicer 146 can insert stop codes into the spliced ​​channel. A channel sequentially input to compression unit 144a, which is one of compression units 144a, 144b, and 144c, cannot be output by compression unit 144a more than a predetermined number of times. When compression unit 144a outputs a compressed channel again, the first splicer 146 can insert stop codes into the spliced ​​channel that includes the compressed channel output from compression unit 144a.

[0185] The first splicer 146 can check whether a stop code has been inserted and, based on the check result, insert a bit value indicating whether a stop code has been inserted. Even if a stop code has not been inserted, the compressed bit representation 148 can still include a code with the same stop code format. In this case, the decoder 516 (see reference) Figure 5 Ambiguity in the splicing channel can be problematic. When a bit value indicating whether a stop code has been inserted is inserted into the splicing channel, the decoder 516 can distinguish the stop code inserted through the first splicer 146.

[0186] For example, the first splicer 146 can check whether a stop code has been inserted into the splicing channel. When a stop code is detected as being inserted into the splicing channel, the first splicer 146 can add a bit value indicating that the stop code has been inserted into the splicing channel. When a stop code is detected as not being inserted into the splicing channel, the first splicer 146 can add a bit value indicating that the stop code has not been inserted into the splicing channel.

[0187] The first splicer 146 can generate a compressed bit representation 148 by inserting stop codes and bit values ​​indicating whether stop codes are inserted into the splicing channel.

[0188] Figure 15 This is a diagram used to describe the stop code insertion algorithm 15 according to an embodiment.

[0189] Reference Figure 15 , as reference Figure 14 The stop code insertion algorithm 15 can be used to insert stop codes and indicate whether a stop code has been inserted into a bit value in the splicing channel.

[0190] In operation 151, the first splicer 146 adds a stop code and indicates the bit value of the stop code to be inserted into the splicing channel when a stop code is required.

[0191] In operation 152, the first splicer 146 adds a bit value indicating that a stop code has not been inserted into the splicing channel.

[0192] For example, a stop code may have a bit width C, after which C-1 bits of bit value "0" may be selected. The bit value indicating that the stop code is inserted into the splicing channel may be a bit value "0", and the bit value indicating that the stop code is not inserted may be a bit value "1". The width of the stop code and the bit value indicating whether the stop code is inserted are for illustrative purposes only and are not limited to the above description.

[0193] Figure 16 This is a diagram used to describe the compression of the encoder according to an embodiment.

[0194] Reference Figure 16 Table 16 shows the output compressed bit representation of encoder 140 in each sequential compression process (e.g., first compression process 160 to seventh compression process 166) when a 5-bit bit representation is input to encoder 140. Here, the stop code is "10", the 2-bit channel 1 is compressed using ZVC, and the 3-bit channel 2 is compressed using a maximum sequence of 2. 2 Z-RLC compression. For ease of explanation, it is assumed that channel 1 is composed of the first compression unit 144a (refer to...). Figure 14 Compression, channel 2 is compressed by the second compression unit 144b (refer to) Figure 14 )compression.

[0195] In the first compression process 160, the compression unit 144a corresponding to channel 1 outputs compression channel "0", and the compression unit 144b corresponding to channel 2 outputs compression channel "00011". The first splicer 146 splices the compression channels and outputs the compressed bit representation "000011".

[0196] In the second to fifth compression processes 161, 162, 163 and 164, the compression unit 144b corresponding to channel 2 does not output a compressed channel. In the sixth compression process 165, the compression unit 144b corresponding to channel 2 outputs a compressed channel again, and the first splicer 146 outputs the compressed bit representation "1000001" by inserting a stop code "10" and indicating that the stop code was inserted into the splicing channel "0001" with a bit value "0".

[0197] In the second compression process 161, the compression unit 144a corresponding to channel 1 outputs a compressed channel "101", while the compression unit 144b corresponding to channel 2 does not output a compressed channel. Here, the compressed channel "101" includes a stop code that has not yet been inserted. Therefore, the first splicer 146 inserts a bit value "1" indicating that the stop code has not been inserted into the splicing channel "101", and outputs the compressed bit representation "1011".

[0198] Figure 17 This is a block diagram used to describe an encoder according to another embodiment.

[0199] Reference Figure 17 The first divider 142 may further include NS logic and a buffer. The NS logic can extract the leading zero bit required for the null (NS) technique and transmit the leading zero bit to a plurality of compression units 144a, 144b and 144c. The bit representation input to the first divider 142 can be buffered by the buffer.

[0200] The first splicer 146 may also include a multiplexer (Mux) and a stop code inserter. The Mux splices compressed channels, and the stop code inserter inserts stop codes into the spliced ​​channels.

[0201] Figure 18 This is a diagram used to describe the decoder 180 according to an embodiment.

[0202] Reference Figure 18 Decoder 516 (reference) Figure 5 ) and 180 include a second divider 182, multiple decompressors 184a, 184b and 184c and a second splicer 186.

[0203] Decoder 180 can divide the compressed data of the neural network into multiple compressed channels based on a predetermined optimal configuration, and decompress the neural network data by performing decompression techniques on each compressed channel. Here, the compressed data can represent data obtained from a reference... Figures 14 to 17 The encoder 140 describes the compressed data, and various decompression techniques can be represented by reference. Figures 14 to 17 The compression techniques described are the decompression techniques for compressed data.

[0204] Decoder 180 can be modified by applying a reference in reverse. Figures 14 to 17 The compression technology described is used to perform the decompression function.

[0205] The second divider 182 can obtain a predetermined optimal configuration for performing compression techniques on each channel by dividing the neural network data into one or more channels.

[0206] The second divider 182 can obtain a compressed bit representation 148 of the compressed data of the neural network, and can divide the compressed bit representation 148 into one or more compressed channels based on the obtained optimal configuration. For example, the second divider 182 can predict the bit width of the compressed channels constituting the compressed bit representation 148 based on the optimal configuration, and can divide the compressed bit representation 148 into one or more compressed channels based on the predicted bit width.

[0207] The optimal configuration obtained by the second divider 182 can be the same as that obtained by the encoder 140 (see reference). Figure 14 The second divider 182 can predict the bit width of the compressed channel constituting the compressed bit representation 148 based on information about how the compressed channel constituting the compressed bit representation 148 is compressed in the encoder 140.

[0208] For example, refer to again Figure 16 As can be seen, channel 1 is a 2-bit wide channel, ZVC is applied to channel 1, and the compressed channel corresponding to channel 1 is either 1 zero value or 3 non-zero values. Channel 2 is a 3-bit wide channel, Z-RLC is applied to channel 2, and the compressed channel corresponding to channel 2 is either 3 bits or 5 bits. In other words, when each channel is compressed according to the optimal configuration, the second divider 182 can know in advance the number of bits that each compressed data can have. The second divider 182 can divide the compressed bit representation 148 into one or more compressed channels by predicting the bit width of the compressed channels constituting the compressed bit representation 148 based on the number of bits that each compressed data can have.

[0209] Meanwhile, the method for predicting the bit width of the compressed channel constituting the compressed bit representation 148 by the second divider 182 is not limited to the description above. The second divider 182 may predict the bit width of the compressed channel based on at least one of the order in which the compressed channels constituting the compressed bit representation 148 are spliced, the stop code included in the compressed bit representation 148, and the operation of using zero testing.

[0210] Reference Figure 18 The second divider 182 can check the stop code included in the compressed bit representation 148. The second divider 182 can check the bit value indicating whether the stop code has been inserted, and delete the stop code according to the check result.

[0211] For example, when a stop code is detected to be inserted, the second divider 182 can delete the stop code and the bit value indicating that the stop code was inserted, and when a stop code is detected not to be inserted, the second divider 182 can delete only the bit value indicating that the stop code was not inserted.

[0212] The second divider 182 can transmit the divided compression channels to the decompression units 184a, 184b, and 184c. For example, the second divider 182 can send each compression channel to a decompression unit that performs the decompression technique corresponding to each compression channel.

[0213] Multiple decompression units 184a, 184b, and 184c can be matched one-to-one with each compression channel according to an optimal configuration. For example, the first decompression unit 184a can be matched one-to-one with the channel where channel 1 is compressed and perform a first decompression technique, and the second decompression unit 184b can be matched one-to-one with the channel where channel 2 is compressed and perform a second decompression technique.

[0214] Multiple decompression units 184a, 184b, and 184c can decompress each of the divided compression channels based on an optimal configuration and output a decompressed channel. The decompression channel can be compared with the one described above. Figure 14 The first divider 142 described divides one or more channels in the same way.

[0215] The second splicer 186 can splice decompression channels and output a decompression bit representation. The decompression bit representation can be compared with the above reference... Figure 14 The bit representation obtained by the first divider 142 described is the same.

[0216] Figure 19 This is a block diagram used to describe a decoder according to another embodiment.

[0217] Reference Figure 19 The second divider 182 may also include a buffer, a stop code detector, and multiple selectors (sel1, sel2, and sel3). NThe stop code detector can detect stop codes included in the compressed bit representation 148. Multiple selectors sel1, sel2, and sel... N The bit width of each compressed channel can be predicted sequentially, and the compressed channels can be divided into multiple compressed channels. The second splicer 186 may also include a reassembler and a buffer.

[0218] Figure 20 Table 20 is used to illustrate the statistical characteristics of neural network data according to the embodiments.

[0219] Reference Figure 20 Table 20 shows the results of analyzing the statistical properties of neural network data (activations and weights for inference, and activations, weights, and gradients for retraining) from all layers (convolutional layers, fully connected layers, and auxiliary layers for activation functions, normalization, and pooling) of seven different neural networks (LeNet-5, CifarNet1, ResNet-18, SquezeNet, MobileNet, AlexNet, and LSTM networks).

[0220] Various neural networks typically exhibit high sparsity. Redundancy used to compress neural network data can be attributed to both value sparsity and bit sparsity. Value sparsity can represent the ratio of zero-value symbols in the dataset, while bit sparsity can represent the ratio of zero bits in the binary representation.

[0221] Referring to Table 20, many neural network data sources exhibit not only high value-level sparsity, but also high bit-level sparsity in neural network data with low value-level sparsity. For example, the weights used for inference of LeNet-5 have a low value-level sparsity of 3.23% but a high bit-level sparsity of 64.32%. In Table 20, "FxP" represents floating-point numbers.

[0222] In this way, all hidden bit-level sparsity can be revealed by dividing different neural network data into multiple bit segments or multiple channels.

[0223] Therefore, the compression method according to this embodiment can generally be applied to various neural network data, regardless of the type of neural network, layer type, learning model, and type of input data.

[0224] Figure 21 This is for comparing compression ratios according to an embodiment. Figure 21 .

[0225] Reference Figure 21 The compression ratios of the network, data source, and input dataset compressed using the compression method according to this embodiment can be compared using the Shannon limit. Figure 21Each bar shown represents the average compression ratio for each network, data source, and input dataset, and the error bars represent the minimum and maximum values. In one embodiment, the data source may include activations (Inf.Acts) and weights (Inf.Weights) for inference, and activations (Re-tr.Acts), weights (Re-tr.Weights), and gradients (Re-tr.Grads) for retraining. (See reference...) Figure 21 As can be seen, by using the compression method according to this embodiment, various data can be compressed to near the Shannon limit.

[0226] Figure 22 This is a block diagram illustrating the configuration of an electronic system 2200 according to an embodiment.

[0227] Reference Figure 22 The electronic system 2200 can extract useful information by analyzing input data in real time based on a neural network, and determine the status of the device equipped with the electronic system 2200 or control the configuration of the device equipped with the electronic system 2200 based on the extracted information. For example, the electronic system 2200 can be applied to robotic devices (such as drones or advanced driver assistance systems (ADAS)), smart TVs, smartphones, medical devices, mobile devices, image display devices, measuring devices, IoT devices, and can be installed on at least one of various types of electronic devices.

[0228] Electronic system 2200 may include processor 2210, RAM 2220, neural network device 2230, memory 2240, sensor module 2250, and communication module (or transmit / receive module) 2260. Electronic system 2200 may also include input / output module, security module, and power control device. Some of the hardware components of electronic system 2200 may be mounted on at least one semiconductor chip.

[0229] Processor 2210 controls the overall operation of electronic system 2200. Processor 2210 may include a single processor core (single-core) or multiple processor cores (multi-core). Processor 2210 can process or execute programs and / or data stored in memory 2240. In some embodiments, processor 2210 can control the function of neural network device 2230 by executing programs stored in memory 2240. Processor 2210 may be implemented using a CPU, GPU, AP, etc.

[0230] RAM 2220 may temporarily store programs, data, or instructions. For example, programs and / or data stored in memory 2240 may be temporarily stored in RAM 2220 according to the control or boot code of processor 2210. RAM 2220 may be implemented as a memory (such as dynamic RAM (DRAM) or static RAM (SRAM)).

[0231] The neural network device 2230 can perform neural network operations based on received input data and generate information signals based on the execution results. The neural network may include, but is not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), feedforward neural networks (FNNs), deep belief networks, restricted Boltzmann machines, etc. The neural network device 2230 may be a dedicated hardware accelerator for neural networks or a device including a hardware accelerator. The neural network device 2230 can perform read or write operations and neural network operations.

[0232] The neural network device 2230 can perform neural network operations based on received input data and can generate information signals based on the operation results. The neural network may include, but is not limited to, CNNs, RNNs, deep belief networks, restricted Boltzmann machines, etc. The neural network device 2230 is hardware that processes the aforementioned neural network data and may correspond to a hardware accelerator specifically designed for the aforementioned neural network. The information signal may include one of various types of recognition signals (such as speech recognition signals, object recognition signals, image recognition signals, and biometric information recognition signals). For example, the neural network device 2230 can receive frame data included in a video stream as input data and generate recognition signals about objects included in an image displayed by the frame data based on the frame data. However, the inventive concept is not limited to this; the neural network device 2230 can receive various types of input data depending on the type or function of the electronic device on which the electronic system 2200 is installed, and generate recognition signals based on the input data.

[0233] Memory 2240 is a storage device for storing data, and may store an operating system (OS), various programs, and various types of data. In one embodiment, memory 2240 may store intermediate results generated during processing that performs the operation of neural network device 2230.

[0234] The memory 2240 may be DRAM, but is not limited thereto. The memory 2240 may include at least one of volatile memory and non-volatile memory. Non-volatile memory includes ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, FRAM, etc. Volatile memory includes DRAM, SRAM, SDRAM, PRAM, MRAM, RRAM, FeRAM, etc. In one embodiment, the memory 2240 may include at least one of HDD, SSD, CF, SD, Micro-SD, Mini-SD, xD, and Memory Stick.

[0235] Sensor module 2250 can collect information about the vicinity of the electronic device on which electronic system 2200 is installed. Sensor module 2250 can sense or receive signals (e.g., image signals, voice signals, magnetic signals, biosignals, touch signals, etc.) from outside the electronic device and convert the sensed or received signals into data. For this purpose, sensor module 2250 may include at least one of various types of sensing devices (e.g., microphone, imaging device, image sensor, LiDAR sensor, ultrasonic sensor, infrared sensor, biosensor, and touch sensor, etc.).

[0236] Sensor module 2250 can provide the converted data as input data to neural network device 2230. For example, sensor module 2250 may include an image sensor that can generate a video stream by capturing images of the external environment of the electronic device and sequentially provide the continuous data frames of the video stream as input data to neural network device 2230. However, this disclosure is not limited thereto, and sensor module 2250 can provide various types of image data to neural network device 2230.

[0237] The communication module 2260 may include various wired or wireless interfaces capable of communicating with external devices. For example, the communication module 2260 may include a local area network (LAN), a wireless local area network (WLAN) (such as Wi-Fi), a wireless personal area network (WPAN) (such as Bluetooth), a wireless universal serial bus (USB), ZigBee, near field communication (NFC), radio frequency identification (RFID), power line communication (PLC), or a communication interface capable of connecting to a mobile cellular network (such as third-generation (3G), fourth-generation (4G), long-term evolution (LTE), or fifth-generation (5G)).

[0238] Figure 23 This is a flowchart of a method for processing data of a neural network in a neural network device according to an embodiment. Figure 23 The method for processing neural network data shown is related to the embodiments described with reference to the above figures; therefore, even if omitted below, the description given with reference to the foregoing figures can be applied. Figure 23 The method.

[0239] In operation 2310, processor 410 may obtain one or more bit representations of data for processing the neural network.

[0240] In operation 2320, processor 410 can generate multiple candidate profiles by dividing one or more bit representations of the acquired data into one or more channels and applying compression techniques to each channel.

[0241] In operation 2330, processor 410 can determine the final profile by comparing the compression performance of each of the candidate profiles.

[0242] In operation 2340, processor 410 can determine one or more channel partitioning and compression techniques used in the determined final configuration file as the optimal configuration for compressing data from the neural network.

[0243] Embodiments of the inventive concept can be implemented as computer-readable programs and can be implemented in a general-purpose computer that executes the program using a non-transitory computer-readable recording medium. Furthermore, the structure of the data used in the embodiments of the inventive concept can be recorded on a non-transitory computer-readable recording medium by various means. The non-transitory computer-readable medium can be a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.) and an optical recording medium (e.g., CD-ROM or DVD).

[0244] It should be understood that the embodiments described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects within each embodiment should generally be considered applicable to other similar features or aspects in other embodiments. Although one or more embodiments have been described with reference to the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope defined by the claims.

Claims

1. A method for processing data from a neural network, comprising: Obtain one or more bit representations of the data used to process the neural network from the memory and / or the neural network processor; Multiple candidate profiles are generated by dividing one or more bit representations of the obtained data into one or more channels and applying compression techniques to each channel. The final configuration file is determined by comparing the compression performance of each candidate configuration file; The partitioning method and compression techniques of the one or more channels used in the final configuration file are determined as the optimal configuration for compressing the neural network data, and the neural network data is compressed using the determined optimal configuration. and The compressed neural network data is stored in the memory. The step of generating the plurality of candidate configuration files includes: generating the plurality of candidate configuration files by applying at least one of the partitioning methods and compression techniques in different ways, wherein the partitioning method is used to divide the bit representation into one or more channels, and The candidate configuration files include those that apply the following partitioning method: the number of channels used to divide the bit representation into multiple bit segments and the bit width of the bit segment corresponding to each channel are set differently from each other.

2. The method according to claim 1, further comprising: The compressed neural network data is obtained from the memory; Decompression is performed on the compressed neural network data according to the optimal configuration; and The decompressed neural network data is sent to the neural network processor for use in neural network operations.

3. The method according to claim 1, wherein, The candidate profiles include profiles with different sparsity distributions of channels based on the number of channels and the bit width of the channels.

4. The method according to claim 1, wherein, The candidate configuration files include those that apply different compression techniques to each channel.

5. The method according to any one of claims 1 to 4, wherein, Candidate configuration files include those that apply compression techniques with different algorithm parameters from each other.

6. The method according to any one of claims 1 to 4, wherein, The steps to determine the final configuration file include: Calculate the compression ratio corresponding to each candidate profile for the bit representation of the data; and The candidate profile with the highest calculated compression ratio is selected as the final profile.

7. The method according to any one of claims 1 to 4, wherein, The data for a neural network includes at least one of the network's activations, weights, and gradients.

8. The method according to any one of claims 1 to 4, wherein, Candidate profiles include profiles that apply compression techniques that do not compress the least significant bit of the one or more channels.

9. A neural network device, comprising: Memory; Neural network processor; and The encoder is configured as follows: One or more bit representations of data for processing the neural network are obtained from the memory and / or neural network processor; multiple candidate profiles are generated by dividing the one or more bit representations of the obtained data into one or more channels and applying compression techniques to each channel. The final configuration file is determined by comparing the compression performance of each of the multiple candidate configuration files; The partitioning method and one or more channel compression techniques used in the final configuration file are determined as the optimal configuration for compressing the neural network data, and the neural network data is compressed using the determined optimal configuration. And store the compressed neural network data in the memory. The step of generating the plurality of candidate configuration files includes: generating the plurality of candidate configuration files by applying at least one of the partitioning methods and compression techniques in different ways, wherein the partitioning method is used to divide the bit representation into one or more channels, and The candidate configuration files include those that apply the following partitioning method: the number of channels used to divide the bit representation into multiple bit segments and the bit width of the bit segment corresponding to each channel are set differently from each other.

10. The neural network device according to claim 9, further comprising: The decoder is configured as follows: The compressed neural network data is obtained from the memory; Decompression is performed on the compressed neural network data according to the optimal configuration; and The decompressed neural network data is sent to the neural network processor for use in neural network operations.

11. A computer-readable non-transitory recording medium that records a program for performing the method of any one of claims 1 to 8 on a computer.

12. An apparatus for processing data from a neural network, comprising: The first partitioner is configured to: obtain an optimal configuration predetermined by a final configuration file, obtain a bit representation of the data for processing the neural network from the memory and / or the neural network processor, and partition the bit representation into one or more channels by using the obtained optimal configuration, wherein the final configuration file defines a compression technique set for each channel by partitioning the data of the neural network into one or more channels; Multiple compressors are configured to compress each of the divided channels using the obtained optimal configuration and output compressed channels. and The first splicer is configured to splice compressed channels and output the compressed bit representation to the memory. The step of obtaining the optimal configuration predetermined by the final configuration file includes: generating multiple candidate configuration files by dividing the bit representation of the obtained data into one or more channels and applying compression techniques to each channel; determining the final configuration file by comparing the compression performance of each candidate configuration file; and determining the partitioning method and the compression techniques of one or more channels used in the determined final configuration file as the optimal configuration for compressing the neural network data. The step of generating the plurality of candidate configuration files includes: generating the plurality of candidate configuration files by applying at least one of the partitioning methods and compression techniques in different ways, wherein the partitioning method is used to divide the bit representation into one or more channels, and The candidate configuration files include those that apply the following partitioning method: the number of channels used to divide the bit representation into multiple bit segments and the bit width of the bit segment corresponding to each channel are set differently from each other.

13. The device according to claim 12, wherein, The multiple compressors use the compression techniques applied to each channel in the optimal configuration of each compressor to compress the divided channels.

14. The device according to claim 12 or 13, wherein, The first splicer inserts stop codes to synchronize the compression channels output from the plurality of compressors.

15. The device according to claim 14, wherein, The first splicer checks whether a stop code has been inserted and inserts a bit value indicating whether the stop code has been inserted based on the check result.

16. An apparatus for processing data from a neural network, comprising: The second partitioner is configured to: obtain an optimal configuration predetermined by a final configuration file, obtain a compressed bit representation of the compressed data of the neural network from memory, and partition the compressed bit representation into one or more compressed channels based on the obtained optimal configuration, wherein the compression technique set for each channel is defined in the final configuration file by dividing the data of the neural network into one or more channels; Multiple decompressors are configured to decompress each of the divided compression channels based on an optimal configuration and output the decompressed channels. and The second splicer is configured to splice the decompression channels and output the decompressed bit representation to the neural network processor for neural network computation. The step of obtaining the optimal configuration predetermined by the final configuration file includes: generating multiple candidate configuration files by dividing the bit representation of the obtained data into one or more channels and applying compression techniques to each channel; determining the final configuration file by comparing the compression performance of each candidate configuration file; and determining the partitioning method and the compression techniques of the one or more channels used in the determined final configuration file as the optimal configuration for compressing the neural network data. The step of generating the plurality of candidate configuration files includes: generating the plurality of candidate configuration files by applying at least one of the partitioning methods and compression techniques in different ways, wherein the partitioning method is used to divide the bit representation into one or more channels, and The candidate configuration files include those that apply the following partitioning method: the number of channels used to divide the bit representation into multiple bit segments and the bit width of the bit segment corresponding to each channel are set differently from each other.

17. The device according to claim 16, wherein, The second divider predicts the bit width of the compressed channel that constitutes each compressed bit representation based on the optimal configuration, and divides the compressed bit representation into one or more compressed channels based on the predicted bit width.

18. The device according to claim 17, wherein, The second divider checks the stop code included in the compressed bit representation.

19. The device according to claim 18, wherein, The second divider checks the bit value indicating whether a stop code has been inserted, and deletes the stop code when the bit value indicates that a stop code has been inserted.

Images