Apparatus and method for compressing neural networks

Abstract

A data processing apparatus (100) for a compressed neural network (200) is disclosed. The apparatus comprises processing circuitry (101) for operating the neural network, where the neural network comprises a plurality of processing layers (201a to 201n), where each processing layer (201a to 201n) comprises a plurality of neural network weights. The processing circuitry (101) is further configured to compress the neural network (200) by quantizing the plurality of neural network weights for each processing layer (201a to 201n) using a quantized binary symbol size and by encoding the plurality of quantized neural network weights for each processing layer (201a to 201n) to obtain a compressed neural network. The processing circuitry (101) is further configured to determine a norm for each processing layer (201a to 201n) in accordance with the plurality of neural network weights for each processing layer (201a to 201n), and to determine a respective quantized binary symbol size for each processing layer (201a to 201n) in accordance with the norm for the processing layer (201a to 201n).

Description

Technical Field

[0001] The present invention relates to data processing. More particularly, the present invention relates to apparatus and methods for operating and compressing neural networks. Background Art

[0002] For example, artificial neural networks (ANNs) in the form of convolutional neural networks (CNNs) are being implemented in an increasing number of electronic devices for a variety of different purposes, such as image or speech processing. However, ANNs are typically computationally demanding and consume a lot of energy, for example because the layers of the ANN frequently access memory and have a large memory footprint. For large ANNs with many neural network weights per layer, the memory footprint can become very large. Therefore, it is challenging to implement large ANNs on electronic devices such as smartphones, smart cameras or other types of IoT devices that have low hardware capabilities in terms of processing power, memory and energy storage. Even in cloud computing, where cloud servers have abundant processing power, memory and power resources, reducing the memory footprint of ANNs may provide advantages.

[0003] To address the issues mentioned above, there are some proposals to use compression techniques to reduce the size, i.e. the overall memory footprint of the ANN which is usually subject to error (precision) constraints, and obtain a smaller neural network with (almost) the same capabilities as the original one. Summary of the invention

[0004] The object of the present invention is to provide an apparatus and method for operating and compressing a neural network.

[0005] The above and other objects are achieved by the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the description and the drawings.

[0006] More specifically, according to a first aspect, a data processing device for operating and compressing a neural network (also referred to as a neural network model) is disclosed. The data processing device includes a processing circuit for operating the neural network, wherein the neural network includes a plurality of processing layers, wherein each processing layer includes a plurality of neural network weights, i.e., is defined by a plurality of neural network weights. The processing circuit is also used to compress the neural network, wherein, in order to compress the neural network, the processing circuit is used to quantize the plurality of neural network weights of each processing layer using a corresponding quantized binary symbol size, and encode the plurality of quantized neural network weights of each processing layer to obtain a compressed neural network. The processing circuit is also used to determine a norm for each processing layer based on the plurality of neural network weights of each processing layer, and determine a corresponding quantized binary symbol size for each processing layer based on the norm of the processing layer.

[0007] In another possible implementation, the processing circuit is used to determine the norm of the corresponding processing layer as the square root of the sum of the squares of the multiple neural network weights of the corresponding processing layer (also called the L2 norm) based on the multiple neural network weights of the corresponding processing layer.

[0008] In another possible implementation, the processing circuit is used to determine the corresponding quantized binary symbol size of each processing layer according to the norm of the corresponding processing layer, so that the corresponding quantized binary symbol size is proportional to the norm of the corresponding processing layer.

[0009] In another possible implementation, the processing circuit is used to determine the corresponding quantized binary symbol size of each processing layer as the product of the norm of the processing layer and a proportionality constant, wherein the proportionality constant is the same for all processing layers.

[0010] In another possible implementation manner, the processing circuit is configured to determine the proportionality constant according to an adjustable target quantization error.

[0011] In another possible implementation, the processing circuit is further configured to determine a quantization error (generated by quantizing the plurality of neural network weights of each processing layer using a corresponding quantization binary symbol size), and determine the proportionality constant to be a maximum proportionality constant or close to the maximum proportionality constant, wherein the determined quantization error is still less than or equal to the target quantization error. The quantization error may be a local quantization error of each processing layer, or a global quantization error generated by all processing layers.

[0012] In another possible implementation manner, the processing circuit is used to determine the proportional constant using the large-step and small-step scheme.

[0013] In another possible implementation, the processing circuit is used to encode the multiple quantized neural network weights of each processing layer using an entropy coding scheme, and the coding scheme is specifically a Huffman coding scheme, an arithmetic coding scheme and / or an asymmetric numeric system (ANS) coding scheme.

[0014] In another possible implementation, the data processing device also includes a volatile or non-volatile memory, wherein the volatile or non-volatile memory, in particular RAM, is used to store the compressed neural network, that is, for each processing layer, the multiple quantized and encoded neural network weights.

[0015] In another possible implementation, in order to operate the neural network, the processing circuit is further used to decompress the compressed neural network layer by layer. For example, the layers can be loaded from the RAM to a cache memory.

[0016] In another possible implementation, the processing circuit is further used to compress input data of the neural network.

[0017] In another possible implementation, the multiple processing layers include one or more sparse processing layers.

[0018] According to another aspect, a computer-implemented data processing method is provided, wherein the method includes the step of operating a neural network, wherein the neural network includes a plurality of processing layers, wherein each processing layer includes a plurality of neural network weights. The method also includes the step of determining a norm for each processing layer based on the plurality of neural network weights of each processing layer. In addition, the method includes the step of determining a corresponding quantized binary symbol size for each processing layer based on the norm of the corresponding processing layer. The method also includes the step of compressing the neural network by quantizing the plurality of neural network weights of each processing layer using the corresponding quantized binary symbol size and by encoding the plurality of quantized neural network weights of each processing layer to obtain a compressed neural network.

[0019] The data processing method can be executed by the data processing device described in the first aspect. In addition, other features of the data processing method described in the second aspect are directly derived from the functions of the data processing device described in the first aspect and its different implementations described above and below.

[0020] According to another aspect, a computer program or a computer program product is provided, comprising a non-transitory computer-readable storage medium carrying program code, wherein when the program code is executed by a computer or a processor, the computer or the processor executes the data processing method according to the second aspect.

[0021] Various aspects of the invention can be implemented in software and / or hardware.

[0022] The details of one or more embodiments are set forth in the accompanying drawings and the description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, wherein:

[0024] Figure 1 A schematic diagram of a data processing apparatus for operating and compressing a neural network provided by one embodiment;

[0025] Figure 2 A schematic diagram of an exemplary neural network implemented by a data processing device provided for one embodiment;

[0026] Figure 3 A diagram of a quantization range and a plurality of quantization binary symbols for uniform quantization;

[0027] Figure 4 is a graph of a uniform approximation of the error distribution in the high quantization rate region;

[0028] Figure 5 is a graph of a rate-distortion curve;

[0029] Figure 6a and Figure 6b A norm-based quantization method implemented by a data processing device is provided in one embodiment, wherein a short weight vector ( Figure 6a ) with a weight vector of greater length ( Figure 6b ) smaller quantized binary symbols to quantize;

[0030] Figure 7 shows quantization error and quantized binary symbol size during several iterations of a parameter search implemented by a data processing apparatus provided by one embodiment;

[0031] Figure 8 A schematic diagram of an optimization solution implemented by a data processing device provided in one embodiment;

[0032] Fig. 9 A flowchart of a hyperparameter search implemented by a data processing device provided in one embodiment;

[0033] Fig.10 A flow chart of a computer-implemented data processing method for operating a neural network provided for one embodiment;

[0034] Fig.11 A table showing the compression performance provided by a data processing apparatus and method provided by one embodiment compared with conventional compression schemes;

[0035] Fig.12a and Figure 12b A diagram showing the compression performance of a data processing apparatus and method provided by one embodiment compared with a traditional optimization scheme.

[0036] In the following the same reference signs refer to identical or at least functionally equivalent features. DETAILED DESCRIPTION

[0037] In the following description, reference is made to the accompanying drawings which form a part of the present invention, which show by way of illustration specific aspects of embodiments of the present invention or specific aspects in which embodiments of the present invention may be used. It should be understood that embodiments of the present invention may be used in other aspects and include structural or logical changes not depicted in the accompanying drawings. Therefore, the following detailed description should not be understood in a restrictive sense, and the scope of the present invention is defined by the appended claims.

[0038] For example, it should be understood that the disclosure related to describing a method may also be applicable to a corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, for performing the described one or more method steps (e.g., a unit that performs the one or more steps, or multiple units that each perform one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the figure. On the other hand, for example, if a specific device is described in terms of one or more units (e.g., functional units), the corresponding method may include a step to perform the function of the one or more units (e.g., a step that performs the function of the one or more units, or multiple steps that each perform the function of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the figure. In addition, it should be understood that, unless otherwise specifically noted, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0039] Figure 1 FIG. 1 is a schematic diagram of a data processing device 100 provided in accordance with an embodiment. Figure 1In the illustrated embodiment, for example, the data processing apparatus 100 is a server 100, such as a cloud server 100. However, as will be appreciated, in other embodiments, for example, the data processing apparatus 100 may be implemented as a server farm, a desktop computer, a laptop computer, a tablet computer, a smartphone, or other device having computing resources for implementing a neural network.

[0040] like Figure 1 As shown, the server 100 may include a processing circuit 101, such as one or more processors or cores 101 for processing data, and a memory 103 for storing and retrieving data. In addition, the server 100 may include a communication interface 105, for example, for exchanging data with other devices such as a smartphone 120 via a wired and / or wireless communication channel 110.

[0041] The processing circuit 101 of the server 100 can be implemented by hardware and / or software. The hardware may include a digital circuit, or both an analog circuit and a digital circuit. The digital circuit may include components such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a general-purpose processor. The memory 103 may store an executable program code, which, when executed by the processing circuit 101, enables the server 100 to perform the functions and methods described herein.

[0042] As will be further referenced below Figure 2 Described in more detail, the processing circuit 101 of the data processing device 100 (e.g., server 100) is used to implement (i.e., operate) a neural network 200, which includes a plurality of processing layers 201a to 201n. Each processing layer 201a to 201n includes a plurality of neural network weights, which are generally determined or adjusted during the training phase of the neural network 200. The plurality of neural network weights of each processing layer 201a to 201n may be stored in the memory 103 of the data processing device 100. The plurality of processing layers 201a to 201n may include any type of neural network layer defined by a plurality of neural network weights, such as a fully connected layer, a convolutional layer, a deconvolutional layer, a recursive layer, etc. As an example, in Figure 2 In the illustrated embodiment, a plurality of processing layers 201 a to 201 n of a neural network 200 implemented by the data processing device 100 is used to generate output data 205 in the form of a feature vector 205 from input data 203 in the form of an input image 203 .

[0043] Before describing the data processing apparatus 100 for compressing multiple processing layers in more detailed embodiments, some further technical background will be introduced below.

[0044] Compression of data usually consists of two stages, namely quantization and encoding. In the quantization stage, the number of unique values ​​(symbols) is reduced. As the number of unique symbols decreases, the entropy decreases, and therefore the number of bits required to represent the layer decreases. This can be a problem when dealing with neural networks, because each processing layer may have a different number of parameters (i.e., neural network weights) and may follow different distributions, which carry different entropies. In general, it may be beneficial to quantize each layer at a rate that remains close to the original layer distribution. Nevertheless, representing the symbols of the original layer at a rate lower than its original entropy introduces distortion (i.e., quantization error). Finding a technical solution to quantize the model at the lowest possible bit rate while meeting some quantization error (distortion) requirement (i.e., target quantization error threshold) is the core of the quantization optimization problem, known as the rate-distortion problem. In addition, compression schemes that quantize each layer at different rates are called hybrid bit compression schemes.

[0045] During the encoding phase, symbol statistics are collected and quantized weights are compressed using an asymptotically optimal entropy compressor (e.g., Huffman, arithmetic coding, ANS, etc.), which sets a length for each symbol inversely proportional to its probability. In addition to reducing the number of symbols, quantization can have a wide range of effects on compression because it produces unique symbol statistics (distributions) and, therefore, entropy. Therefore, different quantization schemes may produce different entropy values ​​even if the quantization rate is effectively the same. Typically, the entropy encoder compresses these quantized values ​​to their entropy limit without introducing further errors (i.e., lossless encoding). Therefore, to optimize the compression of neural networks, the goal is to choose the quantization parameter that achieves the maximum compression ratio while satisfying the quantization error constraint.

[0046] In a typical setting, optimization of quantization parameters requires finding the quantization range ([min, max]) and the number of quantization levels N (or equivalently, the quantization rate R = log2(N), where the quantization error (distortion) is measured separately at each layer) for each processing layer. This per-layer distortion approach is simpler and can be analyzed using standard rate-distortion theory. Nevertheless, it focuses on each layer independently rather than on the entire neural network, thus missing the real purpose of optimization. Moreover, such compression schemes ignore the propagation error effect that spreads the distortion to consecutive layers in addition to their own quantization error. Addressing this error propagation is complicated because the search space for parameter optimization during quantization grows exponentially with the number of processing layers of the neural network. This makes exhaustive search nearly impossible even for moderately sized neural networks.

[0047] As mentioned above, the goal of quantization is to reduce the number of symbols before (lossless) compression. It has been shown that uniform scalar quantization is (asymptotically) optimal when the quantized data is intended to be further compressed. In other words, non-uniform scalar quantization techniques do not produce better compression than simple uniform scalar quantization. This is because uniform quantization preserves the probabilistic nature of the weights and therefore helps (asymptotically) reach the entropy limit.

[0048] For variables A scalar uniform quantizer q with N quantization intervals (i.e., quantized binary symbols) U (W) The interval Evenly divide the partition so that the partition boundary is In (such as Figure 3 Then, any realization of W that falls within a quantized binary symbol j is represented by index j, and its reconstruction value c j It is usually the value W in the middle of the quantized binary symbol. The fidelity of the quantization is usually measured by a distortion metric, such as the Mean Squared Error (MSE) criterion, defined as follows:

[0049] D(N)=E|Wq U (W)| 2

[0050] It is often more convenient to analyze a uniform quantizer in terms of the quantization rate R=log2 N rather than the number of quantized binary symbols N. This rate R essentially describes the number of bits required to index a quantized binary symbol. To analyze this quantizer, a high rate regime (R>>1) is considered, where each quantized binary symbol Δ j The probability curve in is almost flat, such as Figure 2 That is, with Δ j , the quantization error distribution is approximately uniform. Therefore, let c j and Δ j are the jth quantization (reconstruction) point and the quantization interval respectively. The relationship between the quantization interval and the obtained quantization MSE is as follows:

[0051]

[0052] This relationship between the quantization rate and the distortion it produces is known. Specifically, quantization at a lower rate R results in greater distortion, and vice versa, since lower distortion requires a higher rate R. This behavior is shown in Figure 5 This is called the rate-distortion curve.

[0053] As mentioned above, after quantizing the neural network weights, they are compressed using an entropy encoder. For completeness, a brief background on entropy-implemented compression schemes is provided below.

[0054] The best compression schemes assign each symbol a bit length inversely proportional to its probability. That is, common symbols are represented with fewer bits than rare symbols. Entropy encoders are distinguished by their inherent means of capturing dependencies between symbols. A practical view is to view them as finite state automata (FSAs). In this sense, Huffman coding is the simplest and fastest encoder that utilizes a single state. That is, for each input alphabet symbol, the encoder outputs the corresponding prefix-free code from a lookup table. However, Huffman coding must assign an integer number of bits to each symbol and, therefore, stays away from the entropy limit of fractional bits per symbol that can be achieved.

[0055] Arithmetic coding can improve on this shortcoming of Huffman coding, facilitating fractional numbers of bits per symbol and thus being asymptotically optimal. However, arithmetic coding can become exponentially large in terms of the number of states, since it computes the number of all previous symbols to encode the next symbol. This involves a large number of operations, which can make the implementation cumbersome in terms of memory and latency.

[0056] J.Duda in "Asymmetric Numeral System" arXiv preprint arXiv:0902.0271, 2009 proposed a coding scheme based on Asymmetric Numeral System (ANS), which bridges the gap between Huffman coding and arithmetic coding. That is, it provides lossless compression with very high compression and decompression speeds. It helps to configure the coding table size in terms of finite state size. In addition, it exploits simple arithmetic (only shifts and additions) and has efficient hardware implementation.

[0057] The embodiments disclosed herein exploit the discovery that rotations are produced by linear operations defined by multiple processing layers 201a to 201n of the neural network 200. These rotations complicate compression optimization because they affect the quantization parameters of subsequent processing layers 201a to 201n. For mixed bit schemes, typical quantization requires a lookup for each processing layer i: (a) quantization range [min, max] i and (b) the quantization rate R i (an integer). Therefore, the traditional quantization width is given by the following equation (2):

[0058]

[0059] However, the quantization range is sensitive to rotations. Geometrically, each processing layer 201a to 201n can have its own orientation (direction), length, and size. The neural network weights of each processing layer 201a to 201n are rotated (and stretched) by its inputs, and the output of that layer rotates the weights of the successive layer, and so on. The series of rotations between layers determines the output of the model, and therefore its performance (described below). Figure 8 ). Additionally, quantization should work for all rotations in the (calibration) data. This is complicated because each input to the layer in the data will rotate it in a different way, and quantization should cater for all of these rotations.

[0060] To illustrate how rotation affects the quantization range, a simple example is provided where multiple weights of a layer are represented as a weight vector Therefore, in this case max-min = 1, and in the case of 8-bit quantization, the resulting binary symbol width = (max–min) / (2 R –1)=1 / 255. However, after rotating 45° counterclockwise, the weight vector is And we get max–min = 0 and binary symbol width = 0. Therefore, the conventional method produces different quantizations depending on the exact rotation of the weights.

[0061] In addition, the use of conventional quantization may result in the quantization error requirement being too small. More specifically, in the following, the induced quantization error in the case of cosine similarity will be shown based on formulas (1) and (2) as The order of magnitude is adjusted to O(|dim i | / ‖W i ‖ 2 ). For the relative error, the quantization error is As the norm of the weights increases, the error becomes smaller, which may be too strict and, therefore, will lead to fine quantization, which translates into poor compression ratios. Another disadvantage of traditional schemes is the focus on the quantization process to produce integer bits. This is usually not optimal in terms of compression, because other representations quantized at fractional rates may further reduce the resulting entropy, thereby achieving higher compression ratios.

[0062] Given the case where layer weights are sparse (i.e., many weight values ​​are essentially zero), using a compressed format such as compressed sparse row (CSR) can speed up computations because mathematical operations involving zeros may be skipped. However, the CSR format is generally not optimal in terms of memory because it allocates memory for values ​​and their indices without encoding them at all. Combining the embodiments disclosed herein with weight sparsity can yield significant compression ratios.

[0063] Another drawback of traditional compression schemes is the optimization time / complexity required to find a suitable technical solution. That is, in many existing techniques, parameterization must be adjusted during network training. In addition, post-training methods perform layer-by-layer optimization, which is often cumbersome and requires specialized hardware (e.g., GPU).

[0064] As mentioned above, Figure 1 The data processing apparatus 100 shown is used to operate and compress e.g. Figure 1 The neural network 200 shown has multiple processing layers 201a to 201n, wherein each processing layer 201a to 201n includes multiple neural network weights, i.e., is defined by multiple neural network weights. The processing circuit 101 of the data processing device 100 is used to compress the neural network 200, wherein, in order to compress the neural network 200, the processing circuit 101 of the data processing device 100 is used to quantize the multiple neural network weights of each processing layer 201a to 201n using a corresponding quantization binary symbol size, and encode the multiple quantized neural network weights of each processing layer 201a to 201n to obtain a compressed neural network.

[0065] The processing circuit 101 of the data processing device 100 is also used to determine (i.e., calculate) a norm for each processing layer 201a to 201n based on multiple neural network weights of each processing layer 201a to 201n, and determine a corresponding quantized binary symbol size for each processing layer 201a to 201n based on the calculated norm of the corresponding processing layer 201a to 201n.

[0066] For optimization, the embodiments disclosed herein focus on the error constraints at the output 205 of the neural network 200. This is significantly different from conventional approaches because the local quantization error in each processing layer 201a to 201n affects the resulting error at the output 205 of the neural network 200 differently, and standard rate-distortion theory may not be directly applicable to this case. The embodiments disclosed herein address quantization parameter optimization by formulating the optimization problem at hand as a single parameter optimization, which allows wrapping around multiple processing layers 201a to 201n of the neural network 200. As implemented in embodiments of the data processing device 100, deriving bounds on this single parameter search space and using a nested big-step small-step approach makes the parameter search very efficient and fast.

[0067] Embodiments of the data processing apparatus 100 implement a hybrid bit fractional quantization scheme that can efficiently search on a generalized rate-distortion curve. The embodiments disclosed herein perform rotationally invariant quantization that is both scalable (in the number of processing layers 201a to 201n) and extremely fast. The embodiments disclosed herein can be extended to full quantization, where the data processing apparatus 100 is used to compress neural network weights and input data (also called activations) of multiple processing layers 201a to 201n. In addition, the embodiments disclosed herein provide improved compression results for a neural network 200 that includes one or more sparse processing layers 201a to 201n, because the abundance of zero weight values ​​makes the network highly compressible.

[0068] The embodiments of the data processing apparatus 100 disclosed herein can achieve maximizing the neural network compression ratio under the condition of error constraints at the output 205 of the neural network 200. This can be regarded as a generalized rate-distortion problem, because the distortion is required at the output 205 of the neural network 200. In addition, the embodiments disclosed herein provide a mixed bit technology solution, in which the i-th processing layer, that is, one of the processing layers 201a to 201n, is quantized to a rate R i The embodiments disclosed herein can efficiently address an exponentially large parameter search space that scales with the number of processing layers 201a to 201n. In addition, the embodiments disclosed herein can achieve finding parameter solutions that meet time constraints as quickly as possible (ie, a few minutes or less).

[0069] As described above, the processing circuit 101 of the data processing device 100 is configured to quantize each processing layer 201a to 201n of the neural network 200 according to the norm of each processing layer 201a to 201n of the neural network 200, in particular its L2 norm (which can be regarded as a measure of the size or length of the corresponding processing layer 201a to 201n). The method is based on the following discovery: the norm (e.g. length) of the corresponding processing layer 201a to 201n is invariant to rotation. In one embodiment, for the i-th processing layer, i.e. one of the processing layers 201a to 201n, the processing circuit 101 of the data processing device 100 is configured to select the quantization binary symbol size Δ as large as possible i ∝‖W i ‖. As mentioned before, a larger quantized binary symbol size Δ i leads to a greater entropy gain and, therefore, a higher compression ratio. As will be understood and discussed in Figure 6a and Figure 6bAs schematically shown in FIG. 1 , the method of quantizing each processing layer 201 a to 201 n of the neural network 200 based on its norm, implemented by the processing circuit 101 of the data processing device 100, results in a smaller quantized binary symbol size for a smaller norm. Therefore, larger processing layers 201 a to 201 n (i.e., processing layers 201 a to 201 n with larger norms) are compressed at a higher ratio and are therefore more important for the resulting compressed size. On the other hand, their local quantization errors are also larger, and this trade-off is described in more detail below by other embodiments of the data processing device 100.

[0070] As will be appreciated, as a result of the method implemented by the processing circuit 101 of the data processing apparatus 100 of quantizing each processing layer 201a to 201n of the neural network 200 according to its norm, the quantization rate used substantially by the data processing apparatus 100 is not limited to the conventional integer rate. This enables more compressible non-integer quantization rates to be considered.

[0071] Since an error constraint is imposed at the output 205 of the neural network 200, and the quantization error of each individual processing layer 201a to 201n contributes to this global output error, an embodiment of the data processing apparatus 100 implements a parameter search that utilizes a single parameter to set a local target error ∈ t for each processing layer 201a to 201n. i and its corresponding quantized binary symbol size Δ i , rather than the traditional optimization quantization range (max-min) i and the quantization rate R i As mentioned above, the local target error ∈ i It may be beneficial to set it as large as possible to obtain a higher compression ratio. In one embodiment, the processing circuit 101 of the data processing device 100 may be used to process the data at O(|dim(W i )|) sets the local quantization error ∈ i , which is larger than the typical quantization error in the cosine distance and relative error distortion metrics by a factor of ‖W i ‖.

[0072] The processing circuit 101 of the embodiment of the data processing apparatus 100 is configured to scale the quantized binary symbol size Δ in all processing layers 201a to 201n together using a single optimization parameter during the search. i , instead of optimizing layer by layer as conventionally suggested to find the optimal quantized binary symbol size Δ for each processing layer 201a to 201n in a fast manner i Therefore, the embodiments disclosed herein keep the binary symbol widths in each layer proportional to their norms, so that the ratios between the “per-layer errors” are similar throughout the optimization process, e.g. Figure 7As shown. In other words, in one embodiment, the processing circuit 101 of the data processing device 100 is used to determine the corresponding quantized binary symbol size of each processing layer 201a to 201n according to the norm of the processing layers 201a to 201n, so that the corresponding quantized binary symbol size is proportional to the norm of the processing layers 201a to 201n. For example, the processing circuit 101 of the data processing device 100 is used to determine the corresponding quantized binary symbol size of each processing layer 201a to 201n as the product of the norm of the processing layer 201a to 201n and a proportionality constant, wherein the proportionality constant is the same for all processing layers 201a to 201n and is a single optimization parameter for determining the quantized binary symbol size of all processing layers 201a to 201n.

[0073] Even with a single optimization parameter, such as a proportionality constant between the norm of each layer 201a to 201n and its quantized binary symbol size, the search space may be quite large. To address this issue and make it computationally more feasible, in one embodiment, the processing circuit 101 of the data processing device 100 is configured to determine and / or set upper and lower bounds of the parameter search space and implement a fast-converging "big step small step" search scheme method that can find an optimized single parameter, such as The proportionality constant in the step size, where |Ω| is the size of the search space. Figure 7 An exemplary search is shown in , which shows the values ​​of the quantization error ∈ (solid line) and the quantization binary symbol size Δ (dashed line) assigned during the search in each layer for each iteration of an exemplary neural network 200 having 12 layers. Figure 7 It can be understood that the ratio of Δ and ∈ between layers, that is, Δ i / Δ j and ∈ i / ∈ j , remains the same throughout the optimization process.

[0074] From the system point of view, given a neural network weight tensor {W i}, an embodiment of the data processing device 100 can achieve a minimal (quantized and compressed) version of the neural network 200, which obtains an output as close as possible to the original, i.e., the output 205 of the non-compressed neural network 200. In order to evaluate the fidelity of the quantized neural network, the input X can be sent through the original neural network 200 and the input can be sent in parallel through the quantized neural network, such as Figure 8 As shown, and the distance, i.e. the difference between the final outputs, is measured. As will be appreciated, the input X can be viewed as calibration data or a random tensor implementation.

[0075] As the input X propagates through the original neural network 200 and its quantized version, the input X can be considered to be rotated and possibly stretched by each processing layer 201a to 201n. When the weights of the quantized layer and the original layer have similar lengths (norms), the quantization error is essentially reflected in the rotational shift, which occurs in each quantized layer. That is, each quantized layer will produce a rotational error in its output, and this error continues to propagate and accumulate on the layers until it reaches the output of the quantized neural network. It should be noted that these rotational shifts can be constructive, thereby increasing the distance, or destructive, and thereby reducing the distance in each layer, relative to the processing layers 201a to 201n of the original neural network 200.

[0076] When the output error is small enough (i.e., less than a predefined threshold), in one embodiment, the processing circuit 101 of the data processing device 100 is used to compress the neural network 200 layer by layer using an asymptotically optimal encoder such as ANS. According to the rate-distortion theory, the average length of the symbols in the i-th processing layer 201a to 201n after compression is given by the following formula (3):

[0077]

[0078] Where H(·) is the entropy function H(X) = -∑ x∈X p x logp x In other words, quantization reduces entropy by -log(Δ i ), therefore, a larger quantized binary symbol size Δ i Thus, in one embodiment, the processing circuit 101 of the data processing apparatus 100 is configured to determine a quantized version of the neural network 200 that meets a target quantization error requirement while providing the largest possible quantized binary symbol size Δ i .

[0079] As mentioned above, traditional quantization parameter optimization usually involves optimizing the quantization range [max, min] i and the quantization rate R of each layer i Even though optimizing range and rate is a good strategy for quantizing a single layer (where distortion is measured locally), as described above, optimization becomes extremely difficult when considering the entire neural network 200 and its distortion at the output 205.

[0080] Before describing other embodiments of the data processing device 100, the conventional scalar uniform quantizer will be discussed again below, in particular, the connection between the quantization binary symbol size Δ and the generated quantization error. According to the above formula (1) and the law of large numbers, it will be understood that the following relationship holds:

[0081]

[0082] Other error criteria can be determined based on this mean square error (MSE).

[0083] For example, the relative error metric (mean absolute percentage error) can be obtained by taking the square root and normalizing by the layer norm as follows:

[0084]

[0085] Or, for Δ rel :

[0086]

[0087] The cosine distance criterion can be derived from the discovery that the cosine distance is equal to the Euclidean distance of normalized vectors up to a constant. Therefore, assuming that the norm ‖q(W)‖≈‖W‖ is roughly the same, the following relationship holds:

[0088] ‖Wq(W)‖ 2 ≈2‖W‖ 2 (1-cos(∠ W,q(W) ))

[0089] Therefore, the following formula (4) holds:

[0090]

[0091] Or, for Δ cos , the following formula (5) holds:

[0092]

[0093] In the following, by way of example, the cosine distance analysis will be described in more detail. Based on the above formulas (4) and (5), it can be understood that the quantization binary symbol size Δ can be selected so that it is proportional to the norm ‖W‖ by making the local quantization error proportional to dim(W). Specifically, set Where k is the parameter to be optimized. However, as k becomes larger, the local error becomes smaller, which produces a small quantized binary symbol size Δ, which may be detrimental to compression. Therefore, in order to implement quantization and quantization error, specifically, a small fixed constant can be added Right now,

[0094]

[0095] The corresponding quantized binary symbol size is:

[0096]

[0097] The resulting fractional quantization rate of the i-th processing layer 201a to 201n can be expressed by the following equation (6):

[0098]

[0099] This fraction R i is the maximum possible Δ i This is very important for compression. However, in practice, after decoding the symbols, one of the nominal representations (e.g., fp32, fp16, int8, etc.) can be used to store the symbols.

[0100] According to formula (6), the following entropy gain can be obtained:

[0101]

[0102] As will be appreciated, since the norm of the layer is invariant to rotation, setting the quantization binary symbol size Δ to scale in the same way as the norm will result in rotation-invariant quantization. In other words, the quantization fidelity is independent of the inputs to the layer because it rotates the weights of the layer. As will be further appreciated, choosing a different k, and therefore a different scaling constant, for each layer 201a to 201n will break this rotational invariance.

[0103] As described above, the search and optimization scheme implemented by the processing circuit 101 of the data processing device 100 according to one embodiment is basically related to all processing layers 201a to 201n by using a single parameter (e.g., parameter k). Nevertheless, the search space Ω(k) may become quite large. In other words, it is possible to find the optimal k that satisfies the error constraint of the neural network 200 compression. * It may still be difficult. In order to search for the best k * According to one embodiment, the processing circuit 101 of the data processing device 100 is used to limit the actual search range of the optimal parameter. In the case of the cosine distance, the upper limit of the search range can be determined by the processing circuit 101 of the data processing device 100 in the following manner according to the following relationship:

[0104]

[0105] Among them, i * is the index of the largest layer (i.e. the layer with the largest norm) of the neural network 200. As will be appreciated, in the search scheme implemented by the processing circuit 101 of the data processing apparatus 100, this is the layer to which the error converges. This can be used to define where to stop the search. More specifically, for sufficiently large k, the error is approximately At this point, the error converges even in the largest layer (and therefore for the remaining layers which have smaller norms).

[0106] That is, when hour,

[0107] for An example value of

[0108] For the lower bound of the search range, we can use the cosine distance criterion ∈ i ≤ 1. Therefore, we focus again on the largest layer i * , it can be observed

[0109] when hour,

[0110] Therefore, in one embodiment, the processing circuit 101 of the data processing device 100 may be configured to limit the search to the following range:

[0111]

[0112] In order to further improve the search time, the processing circuit 101 of the data processing device 100 may also be used to implement a nested large-step small-step search method, which will be described in more detail below.

[0113] In the initial "big step" phase, only k Then, the smallest k that satisfies the requirement in terms of output error becomes the new upper bound, and the search area is optimized using a smaller region of k, again, this region only has to be checked This process is repeated until the search step is small enough. This allows a good technical solution to be found in relatively few iterations (for example, less than 40 iterations).

[0114] Fig. 9 The flowchart shown shows more details of the nested large-step small-step search method implemented by the processing circuit 101 of the data processing device 100. Initially, the search range and the initial step size are set to the square root of the search range size (see Fig. 9 901). Repeatedly refine this step size until the optimal solution is found (see Fig. 9 917 and 919). In iteration k, as long as k is not greater than k max (See Fig. 9 903, 905), the following quantized binary symbol size is used (see Fig. 9 907, 909, 911) in quantize each layer i relative to its norm:

[0115]

[0116] This makes the entropy of this layer smaller, which can be obtained by compression. Then, to evaluate the distortion of the output, the model is fed with calibration (or random) data (see Fig. 9 913) and measure the cosine distance at the output. If the generated model does not meet the error requirements (see Fig. 9 915), then the k parameter is increased by the step size and the model is quantized again at a higher rate (i.e., finer quantization). Otherwise, the compression ratio is stored and the search is continued in a smaller search range using a smaller search step size (see Fig. 9 919). The latter is repeated until the search step size is small enough (see Fig. 9 917). In essence, this search scheme implemented by the processing circuit 101 of the data processing device 100 according to one embodiment strives to find a solution with the smallest possible k. As described above, since the rate-distortion curve is monotonically decreasing, the smallest k that meets the target quantization error requirement produces the maximum compression ratio.

[0117] Fig.10 For manipulating and compressing neural networks (e.g. Figure 2 1 . The method 1000 includes: step 1001, operating the neural network 200, wherein, as described above, the neural network 200 includes a plurality of processing layers 201a to 201n, wherein each processing layer 201a to 201n includes a plurality of neural network weights. In addition, the method 1000 includes: step 1003, determining the norm of each processing layer 201a to 201n based on the plurality of neural network weights of the processing layer 201a to 201n. The method 1000 also includes: step 1005, determining the corresponding quantized binary symbol size of each processing layer 201a to 201n based on the norm of the processing layer 201a to 201n. As described above, in one embodiment, this is performed in a manner that the total error of the neural network output 205 is less than the desired target error. In addition, the method 1000 includes: step 1007, compressing the neural network 200 by quantizing multiple neural network weights of each processing layer 201a to 201n using a quantization binary symbol size and encoding the multiple quantized neural network weights of each processing layer 201a to 201n to obtain a compressed neural network.

[0118] The compression performance provided by embodiments of the data processing apparatus 100 has been evaluated for various neural network models of different sizes using the cosine distance criterion (where, for example, the quantization must achieve a cosine similarity < 0.005). As a comparison baseline, the Mindspore version 1.6 converter (tested on x86) was used, which has been highly optimized compared to other state-of-the-art solutions. Fig.11 As can be seen from the table shown, the implementation provided by the embodiment of the data processing device 100 can converge quickly (within a few seconds) even for large neural network models such as Resnet50. The compression ratio provided by the embodiment of the data processing device 100 is significant and superior, even the current state-of-the-art technology solution of Mindspore.

[0119] In order to evaluate the distance of the solution provided by the data processing apparatus 100 according to an embodiment from the actual optimal solution, the compression performance provided by the data processing apparatus 100 according to an embodiment is compared with the compression performance provided by a Multi-Objective Bayesian Optimization (MOBO) solution, which is computationally very complex and requires a long running time (several hours to several days on multiple GPUs). Fig.12a and Figure 12b It can be seen that the compression ratio achieved by the data processing device 100 according to one embodiment is very similar to the compression ratio provided by the MOBO technical solution obtained after running for one week.

[0120] Those skilled in the art will understand that the "blocks" ("units") in the various figures (methods and devices) represent or describe the functions of an embodiment of the present invention (and not necessarily independent "units" in hardware or software), and therefore equally describe the functions or features of the device embodiments and the method embodiments (units are equivalent to steps).

[0121] In the present application, several embodiments are provided, and it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the above device embodiments are only exemplary. For example, the unit division is only a logical function division and can be other divisions in the actual implementation.

[0122] For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be achieved through some interfaces. Indirect coupling or communication connection between devices or units may be achieved through electronic, mechanical or other forms.

[0123] The units described as discrete parts may be physically separated or not, and the parts shown as units may be physical units or not, and may be located in one location or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] In addition, the functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Claims

1. A data processing device (100), characterized in that: include: A processing circuit (101) for operating a neural network (200), wherein the neural network (200) comprises a plurality of processing layers (201a to 201n), wherein each processing layer (201a to 201n) comprises a plurality of neural network weights; The processing circuit (101) is further used to compress the neural network (200), wherein, in order to compress the neural network (200), the processing circuit (101) is used to quantize the multiple neural network weights of each processing layer (201a to 201n) using a quantization binary symbol size, and encode the multiple quantized neural network weights of each processing layer (201a to 201n) to obtain a compressed neural network, The processing circuit (101) is further used to determine a norm for each processing layer (201a to 201n) according to the multiple neural network weights of each processing layer (201a to 201n), and determine a corresponding quantized binary symbol size of each processing layer (201a to 201n) according to the norm of the processing layer (201a to 201n).

2. The data processing device (100) according to claim 1, characterized in that: The processing circuit (101) is used to determine the norm of the corresponding processing layer (201a to 201n) as the square root of the sum of the squares of the multiple neural network weights of the corresponding processing layer (201a to 201n) based on the multiple neural network weights of the corresponding processing layer (201a to 201n).

3. The data processing device (100) according to claim 1 or 2, characterized in that: The processing circuit (101) is used to determine the corresponding quantized binary symbol size of each processing layer (201a to 201n) according to the norm of the processing layer (201a to 201n), so that the corresponding quantized binary symbol size is proportional to the norm of the processing layer (201a to 201n).

4. The data processing device (100) according to claim 3, characterized in that: The processing circuit (101) is used to determine the corresponding quantized binary symbol size of each processing layer (201a to 201n) as the product of the norm of the processing layer (201a to 201n) and a proportionality constant, wherein the proportionality constant is the same for all processing layers (201a to 201n).

5. The data processing device (100) according to claim 4, characterized in that: The processing circuit (101) is used to determine the proportionality constant according to a target quantization error.

6. The data processing device (100) according to claim 5, characterized in that: The processing circuit (101) is further used to determine a quantization error, wherein the processing circuit (101) is used to determine the proportionality constant as a maximum proportionality constant, wherein the quantization error is less than or equal to the target quantization error.

7. The data processing device (100) according to claim 6, characterized in that: The processing circuit (101) is used to determine the proportionality constant using a large-step small-step scheme.

8. The data processing device (100) according to any one of the preceding claims, characterized in that The processing circuit (101) is used to encode the plurality of quantized neural network weights of each processing layer (201a to 201n) using an entropy coding scheme.

9. The data processing device (100) according to claim 8, characterized in that: The entropy coding scheme is based on a Huffman coding scheme, an arithmetic coding scheme and / or an asymmetric numeral system (ANS) coding scheme.

10. The data processing device (100) according to any one of the preceding claims, characterized in that The data processing device (101) further comprises a volatile or non-volatile memory (103), wherein the volatile or non-volatile memory (103) is used to store the compressed neural network.

11. The data processing device (100) according to claim 9 or 10, characterized in that: In order to operate the neural network (200), the processing circuit (101) is also used to decompress the compressed neural network layer by layer.

12. The data processing device (100) according to any one of the preceding claims, characterized in that The processing circuit (101) is also used to compress input data of the neural network (200).

13. The data processing device (100) according to any one of the preceding claims, characterized in that The plurality of processing layers (201a to 201n) include one or more sparse processing layers (201a to 201n).

14. A computer-implemented data processing method (1000), characterized in that: include: Operating (1001) a neural network (200), wherein the neural network (200) comprises a plurality of processing layers (201a to 201n), wherein each processing layer (201a to 201n) comprises a plurality of neural network weights; determining (1003) a norm for each processing layer (201a to 201n) based on the plurality of neural network weights of each processing layer (201a to 201n); Determining (1005) a corresponding quantized binary symbol size for each of said processing layers (201a to 201n) based on said norm of said processing layers (201a to 201n); The neural network (200) is compressed (1007) by quantizing the plurality of neural network weights of each processing layer (201a to 201n) using the quantized binary symbol size, and by encoding the plurality of quantized neural network weights of each processing layer (201a to 201n) to obtain a compressed neural network.

15. A computer program product, characterized in that A computer-readable storage medium is included for storing program code, which, when executed by a computer or a processor, causes the computer or the processor to perform the method (1000) according to claim 14.

Images