Systems and methods for encoding / decoding deep neural networks

Abstract

The present disclosure relates to a method including quantizing parameters of an input tensor, the quantizing using a codebook whose size is obtained according to a distortion value determined between at least one tensor and a quantized version of the at least one tensor. The present disclosure also relates to a method for quantizing parameters of an input tensor using a PDF-based initialization bounded according to at least one first PDF coefficient, the first PDF coefficient being selected from among several candidate boundary PDF coefficients according to a resulting entropy. The present disclosure also relates to corresponding signals; bitstreams, storage media, and encoder and / or decoder devices.

Description

[Technical Field]

[0001] A domain technical field of one or more embodiments of the present disclosure relates to a data processing technology domain, such as data compression and / or data decompression. For example, at least some embodiments relate to data compression / decompression involving large amounts of data, such as compression and / or decompression of at least a portion of an audio stream and / or a video stream, or compression and / or decompression of data associated with deep learning techniques, such as at least some parameters of a deep neural network (DNN).

[0002] At least some embodiments relate to improving compression efficiency compared to developing video compression systems such as HEVC (HEVC is High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2, as described in "ITU-T H.265 Telecommunication standardization sector of ITU (10 / 2014), series H: audiovisual and multimedia systems, infrastructure of audiovisual services—coding of moving video, High efficiency video coding, Recommendation ITU-T H.265"), or compared to evolving video compression systems such as VVC (Versatile Video Coding, a new standard being developed by the JVET Joint Video Experts Team).

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction, including spatial vector prediction and / or motion vector prediction, and transformation to exploit spatiotemporal redundancy in video content. Intra- or inter-prediction is generally used to exploit correlation within or between frames, whereby the difference between an original image and a predicted image, often called a prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by an inverse process corresponding to entropy coding, quantization, transformation, and prediction.

[0004] At least some embodiments are directed to improving compression efficiency compared to existing systems for compression of deep neural networks (DNNs), such as some compression standards or draft standards, such as the upcoming neural network standard ISO / MPEG7 for multimedia content description and analysis currently being developed by the International Organization for Standardization.

[0005] Typically, the encoding process involves quantizing the parameters of the DNN and entropy coding to obtain compressed data. To reconstruct the data, the compressed data is decoded, and the decoding process includes entropy decoding and dequantization. Summary of the Invention

[0006] The principles of the present invention make it possible to overcome at least one drawback of some known compression and / or decompression methods by proposing a method comprising encoding data in at least one bitstream, the data being one or more parameters of at least one tensor of at least one layer or sublayer of at least one deep neural network. It is noted that the tensor of network parameters associated with a layer can include network parameters of the type such as weights and / or biases (for example, in terms of "Bach normalized" tensors).

[0007] According to an embodiment, there is provided a method for encoding at least one tensor associated with at least one layer of a deep neural network in a bitstream, wherein encoding the at least one tensor includes obtaining a size of a codebook for quantizing parameters of the at least one tensor, the size being obtained according to a distortion value determined between the at least one tensor and a quantized version of the at least one tensor, and quantizing the parameters using a codebook having the obtained size.

[0008] Another embodiment provides a device for encoding at least one tensor associated with a layer of at least one deep neural network in a bitstream, the device comprising: at least one processor configured to obtain a size of a codebook for quantizing parameters of the at least one tensor, the size being obtained according to a distortion value determined between the at least one tensor and a quantized version of the at least one tensor; and quantizing the parameters using a codebook having the obtained size.

[0009] Another embodiment provides a method for decoding at least one tensor associated with a layer of at least one deep neural network from a bitstream, where decoding the at least one tensor includes decoding information representing a type of codebook from the bitstream and dequantizing parameters of the at least one tensor using the codebook.

[0010] According to another embodiment, there is provided an apparatus for decoding at least one tensor associated with a layer of at least one deep neural network from a bitstream, the apparatus comprising at least one processor configured to decode information indicative of a type of codebook from the bitstream and to dequantize parameters of the at least one tensor using the codebook.

[0011] One aspect of the present disclosure relates to a device comprising at least one processor adapted to quantize parameters of an input tensor, the quantization using a PDF-based initialization bounded according to at least one first PDF coefficient, the first PDF coefficient being selected from among several candidate boundary PDF coefficients according to a resulting entropy.

[0012] One aspect of the present disclosure relates to a method that includes quantizing parameters of an input tensor, the quantizing using a PDF-based initialization bounded according to at least one first PDF coefficient, the first PDF coefficient being selected from among several candidate boundary PDF coefficients according to a resulting entropy.

[0013] According to some embodiments of the present disclosure, the quantizing uses codebook-based quantization, and the codebook size is selected from several candidate codebook sizes according to an error between the input tensor and a quantized tensor obtained from the input tensor using the candidate codebook sizes.

[0014] According to another aspect, an apparatus is provided, the apparatus comprising a processor, the processor can be configured to encode at least one tensor of at least one layer of at least one deep neural network in at least one bitstream and / or decode a bitstream representing at least one tensor of at least one layer of at least one deep neural network by performing any of the methods described above.

[0015] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any of the decoding embodiments and at least one of (i) an antenna configured to receive data including the input data, (ii) a band limiter configured to limit the received signal to a frequency band including the input data, or (iii) a display configured to display an output representing the video block.

[0016] According to yet another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described decoding embodiments or variations.

[0017] According to yet another general aspect of at least one embodiment, a signal is provided that includes data representing at least one tensor of at least one layer or sub-layer of at least one deep neural network, generated according to any of the described decoding embodiments or variations.

[0018] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described decoding embodiments or variations.

[0019] According to another general form of at least one embodiment, a computer program product is provided that includes instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0020] According to another general aspect of at least one embodiment, there is provided a computer-readable non-transitory program storage device tangibly embodying a program of instructions executable by a computer to perform at least one of the disclosed methods in any of the embodiments.

[0021] According to another general aspect of at least one embodiment, a computer-readable storage medium is provided that includes instructions that, when executed by a computer, cause the computer to perform at least one of the disclosed methods in any of those embodiments.

[0022] Although not explicitly stated, the devices of the present disclosure may be adapted to perform the methods of the present disclosure in any of their embodiments.

[0023] Although not explicitly stated, the present embodiments relating to methods or corresponding signals, devices, and computer-readable storage media may be employed in any combination or subcombination. [Brief explanation of the drawings]

[0024] [Figure 1] Here are some common standard encoding methods: [Figure 2] A common standard decoding method is shown below. [Figure 3] 1 illustrates an exemplary processor device in which the described embodiments may be implemented. [Figure 4] 1 illustrates a DNN encoding scheme using at least some embodiments of the encoding method of the present disclosure. [Figure 5] 1 illustrates a DNN decoding scheme using at least some embodiments of the decoding method of the present disclosure. [Figure 6] 1 illustrates an example of a method for quantizing parameters of tensors of a DNN layer according to an embodiment. [Figure 7] 1 illustrates an example of a method for quantizing parameters of tensors of a DNN layer according to another embodiment. [Figure 8] 1 illustrates an example of a method for encoding parameters of a tensor of a DNN layer according to an embodiment. [Figure 9] 1 illustrates an example of a method for decoding a DNN according to an embodiment. [Figure 10] 1 illustrates an example of a portion of a bitstream containing data representing tensors of at least one layer of a deep neural network generated in accordance with one embodiment.

[0025] It should be noted that the drawings illustrate example embodiments, and that the embodiments of the present disclosure are not limited to the illustrated embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0026] Many technical fields may involve the processing of large amounts of data by computer means. Such processing may involve data compression and / or data decompression, for example for storing or transforming at least some of such data. Examples of compression and / or decompression of streams containing large amounts of data can be found in the video processing technical field or in technical fields involving deep learning techniques.

[0027] For purposes of illustration and not limitation, embodiments of the present disclosure will be described in detail in connection with deep neural networks (DNNs). However, it should be apparent that the present disclosure may also be applied to the compression / decompression of other large amounts of data, such as in the video processing technology field. For example, the present disclosure may be applied to the compression / decompression of tensors obtained from at least one image by a deep learning algorithm.

[0028] Deep neural networks (DNNs) represent the state of the art in performance across diverse domains such as multimedia processing, computer vision, speech recognition, natural language processing, etc. However, this performance can be computationally expensive, as DNNs tend to have a large number of parameters, often in the millions, and sometimes even billions.

[0029] This can lead to, for example, prohibitively high inference complexity. Simply put, inference is the deployment of a DNN once it has been trained to process input data, for example in terms of its classification. Inference complexity can be defined as the computational cost of applying a trained DNN to the input data to be inferred.

[0030] Therefore, this high inference complexity can be a significant challenge in using DNNs in environments involving electronic devices with limited hardware and / or software resources, e.g., mobile or embedded devices with resource limitations such as battery size, limited computational power, and memory capacity.

[0031] A deep neural network is composed of several layers. Each layer is associated with a set of parameters, which can be obtained, for example, during training of the DNN. These parameters (such as weights and / or biases) are stored as a multidimensional array. For simplicity, the term "matrix" may be used below to refer to a set of parameters (e.g., tensor parameters of a given layer). However, it should be understood that some embodiments of the methods disclosed herein can also be applied to tensor parameters of more than two dimensions, such as a 2D convolutional layer, which typically includes 4D tensor parameters. The large number of parameters in a DNN may require a large bandwidth for deploying the DNN (or a solution including a DNN) in a distributed environment.

[0032] At least some embodiments of the present disclosure apply to compressing and / or decompressing (decoding) at least some parameters of at least one DNN (e.g., a pre-trained DNN). Indeed, compression can facilitate transmission and / or storage of at least one DNN. More precisely, at least some embodiments of the present disclosure apply to compressing parameters of at least one tensor associated with at least one layer of at least one deep neural network. The network parameters can be, for example, compressed one tensor at a time.

[0033] In some embodiments of the present disclosure, the layers (or sublayers) can be of different types. For example, in some embodiments, all of at least one layer can be convolutional layers, or full connector layers, or at least one layer can comprise at least one convolutional layer and / or at least one full connector layer.

[0034] 4 and 5 illustrate, at a high level, a general process for encoding / decoding parameters of at least one tensor of at least one DNN, respectively, that may be used in at least some embodiments of the present disclosure.

[0035] 4, the method 400 may include obtaining 410 (or, in other words, acquiring) parameters for the tensor to be compressed. Obtaining may be performed, for example, by retrieving the parameters for at least one tensor from a storage unit or by receiving the parameters from a data source via a communications interface.

[0036] As shown in Figure 4, compressing at least some parameters of the neural network has the following advantages: quantizing 430 the parameters (such as weights and biases) of the neural network; That is, it may involve lossless entropy coding 440 of the quantized information.

[0037] Such a process allows the parameters to be represented using fewer bits.

[0038] In some embodiments, compressing 400 can further include reducing 420 the number of parameters of the neural network by exploiting redundancy inherent in the neural network prior to quantizing 430. Thus, reducing 420 provides at least one tensor of reduced dimension relative to the dimension of the tensor associated with the layer and the input to the reducing step. For example, the parameters of the tensor of at least one layer of the DNN can be decomposed or made sparse in reducing 420.

[0039] The resulting tensors, typically comprising floating-point values, are quantized and entropy coded into the final bitstream that is transmitted to the receiver.

[0040] This reducing 420 is optional and therefore may be omitted in some embodiments.

[0041] Quantizing a tensor can involve approximating the values ​​of the tensor with integer-like values ​​that require fewer bits than the input tensor (e.g., floating-point values).

[0042] Depending on the DNN compression solution, different types of quantization can be performed, for example by using uniform quantization, as found in some DNN compression solutions, or by using non-uniform quantization, such as codebook-based quantization, as found in some compression standards, notably ISO / MPEG7, and some upcoming standards for neural networks for multimedia content description and analysis, hereafter simply denoted MPEG NNR.

[0043] With uniform quantization, a floating-point "step size" can be specified, and all floating-point values ​​in a tensor can be represented as multiples of the step size. An approximation of the tensor can be reconstructed at the decoder by simply multiplying the integer value by the "step size."

[0044] A codebook is a set of values ​​(such as integer or floating-point values) that a layer's parameters can have after quantization. The index can be derived from the codebook value assigned to the original tensor's parameters by quantization. To reconstruct an approximation of the original tensor, a decoding device uses the index to look up the corresponding floating-point value from the codebook. Codebook-based quantization is discussed in more detail below.

[0045] 4, the parameter input to the quantization may be of floating-point type, while the output of the quantization may include one or more tensors of exponents of integer type. Optionally, the quantization may also output a codebook involving the exponents. Indeed, in some embodiments, the codebook may be omitted in the output of the quantization, for example, in embodiments in which the codebook is pre-fixed by the quantization.

[0046] 4, at least some of the output of quantization 430 is used as input to lossless entropy coding 440. The information input to entropy coding (such as information about a codebook and / or integer tensors for quantized tensors) may, in some embodiments, be split into header information and a payload containing an exponent.

[0047] Other factors (such as the shape of the original tensor, i.e., symbol counts) may also be input to entropy coding 440 .

[0048] If the encoding method 400 can encode several layers, after encoding the parameters associated with a layer, the method can be repeated for each layer on the DNN until the end of encoding the parameters of the last layer to be encoded (450).

[0049] FIG. 5 illustrates a decoding method 500 that can be used to decode a bitstream obtained by the encoding method 400 described above. At the decoder, as illustrated by FIG. 5, the encoding method 500 can include some inverse operations (compared to those at the encoder side). For example, the decoding method 500 can include analysis / entropy decoding 510 of input bins to derive quantized forms of the parameters. Inverse quantization 520 can then be applied to derive final values ​​for the parameters. Matrix decomposition / sparsification of tensors at the encoder typically avoids the need for an inverse process at the decoder. For example, parameters set to zero in the parameter reduction stage (reduction 420) can remain zero after inverse quantization at the decoder.

[0050] As shown in Figure 5, the output of parsing and decoding 510 a bitstream corresponding to a layer of a DNN can consist of metadata and quantized parameters. For example, when codebook-based quantization is performed on the encoder side, the output includes the codebook and corresponding exponents. For example, both the codebook and the tensor of exponents can be computed using K-means algorithms to derive a codebook of K values, shown as cluster centers and tensors of exponents, which can have values ​​in the integer range [0...K-1].

[0051] The decoding method may further include performing inverse quantization 520 using the decoded information (such as the exponents and the codebook).

[0052] When several layers can be decoded by the decoding method 500, the method 500 can be performed repeatedly until the parameters of the last layer are coded (550).

[0053] 4, at least some of the elements output by quantization 430 are used as inputs in performing entropy encoding 440, and the way in which quantization is performed should conform to the way in which entropy encoding is performed, and vice versa. This should be true, for example, whether the quantization is uniform quantization or codebook-based quantization (as used in some compression solutions such as the upcoming standard MPEG-NNR).

[0054] However, this is not necessarily the case. For example, codebook-based quantization can have some drawbacks. For example, codebook quantization can often be more efficient with larger tensors than with smaller tensors. One reason for this is that when the tensor is small, the codebook overhead can be significant compared to the size of the tensor.

[0055] At least some embodiments of the present disclosure help address this problem.

[0056] More precisely, according to a first aspect of the present disclosure, at least some embodiments propose to use uniform quantization for smaller tensors and a codebook for larger tensors.

[0057] This disclosure also proposes an exemplary format (hereinafter also referred to as a "unified format") adapted to be used for several types of quantization, in particular a format that targets both codebook and uniform quantization.

[0058] Entropy coding is a lossless data compression method that works by relying on the fact that any data can be compressed if some data symbols are more likely to occur than others. Examples of entropy coding methods include Huffman coding and arithmetic coding.

[0059] The entropy of the quantized information directly affects the efficiency of arithmetic coding. If the quantized information has high entropy (e.g., high randomness), arithmetic coding algorithms cannot compress the data efficiently. For example, if symbols appear in the data with the same frequency (high randomness, high entropy), compression will not be efficient. However, if some symbols appear more frequently than others, compression will be more efficient by using fewer bits for the more frequent symbols.

[0060] According to a second aspect of the present disclosure, at least some embodiments propose a method to facilitate improving the efficiency of arithmetic coding performed during entropy coding (e.g., conditional arithmetic coding and / or adaptive arithmetic coding) by minimizing the entropy of quantized information.

[0061] The quantized information may be optimized (or at least improved) based on at least one first Mean Square Error (MSE) value (which controls distortion), for example by minimizing the Mean Square Error value while attempting to minimize (or at least lower) the entropy of the quantized information (e.g., by using pdf boundary initialization in a K-means algorithm).

[0062] The present disclosure relates to several aspects. Some aspects, such as the first and second aspects introduced above, may be implemented in the same embodiment and / or separately (some embodiments implement both aspects, while some embodiments implement only one of the aspects). For example, in some embodiments, a binary search for symbol counting may be performed without optimizing the pdf coefficients according to MSE. However, an embodiment that combines a binary search for symbol counting with optimizing the pdf according to MSE may often help obtain better rate / distortion results than when quantization only seeks to minimize the error (MSE) between the quantized tensor and the original tensor. Also, in some embodiments of the present disclosure, quantization as described herein, such as codebook quantization (as associated with the second aspect), and conditional coding and / or adaptive arithmetic coding may be combined.

[0063] It should be noted that the embodiments of the methods of the present disclosure detailed herein can be implemented in many compression solutions and are not limited to any particular standard, and even at least some of the present disclosure may be applied in the context of several compression standards, such as several draft standards being developed by ISO / MPEG7.

[0064] Binary search for "best" symbol count The term symbol count, which collectively refers to the total number of possible symbols that may appear during entropy coding, is the codebook size in the case of codebook quantization.

[0065] In some compression frameworks, the parameters input to the quantization stage can sometimes include a "qBitt" value that indicates the number of bits for representing each symbol in the codebook; thus, the codebook size is 2 qBit is equal to.

[0066] However, in embodiments in which arithmetic coding (or conditional arithmetic coding and / or adaptive arithmetic coding) is used after quantization, the codebook size need not be a power of 2. In accordance with at least some embodiments of the present disclosure, the codebook size (not necessarily a power of 2) can be obtained (or determined) to a first (specified) precision.

[0067] More precisely, according to some embodiments of the present disclosure, instead of specifying a "qBit" value, a first distortion value between the original tensor and the quantized tensor, for example, a desired maximum distortion value such as a desired maximum mean square error (MSE) value (hereinafter referred to as MaxMSE), can be specified in the encoder, and then the codebook size can be obtained according to this first MSE value, for example, by using a binary search.

[0068] In some embodiments of the present disclosure, a binary search can be performed across a range of codebook sizes, for example, a range of codebook sizes from 4 to 4096 (corresponding to 2 to 12 "qBits").

[0069] Obviously, depending on the embodiment, the minimum and maximum values ​​of the ranges may vary, as may the minimum and maximum values ​​of "qBit", and the numerical values ​​(e.g., 4 or 4096 in the case of codebook sizes) are merely exemplary values. For example, in some embodiments, qBit values ​​(and corresponding codebook sizes) between 2 and 20 may be used.

[0070] Depending on the embodiment, a binary search can be applied to monotonically increasing or decreasing values. In fact, as the symbol count (or codebook size) increases, the MSE decreases monotonically. Therefore, a binary search can be used to obtain the best (smallest) symbol count for a given MSE value (maxMSE).

[0071] FIG. 6 illustrates an example of a method 600 for quantizing parameters of tensors of a DNN layer according to an embodiment. In 601, a codebook size for quantizing parameters of at least one tensor of a DNN is obtained. As described above, the codebook size is obtained according to a distortion value determined between at least one tensor and a quantized version of the at least one tensor. In a variant, the codebook size is obtained using a binary search over a codebook size range. In 602, the parameters of the tensor are quantized using a codebook having the size obtained in 601. The embodiment described in conjunction with FIG. 6 may be implemented in the method for encoding a DNN described in conjunction with FIG. 4.

[0072] Reducing entropy quantization using pdf-based initialization Some quantization solutions can be based on pdf-based initialization, such as pdf-based initialization of k-means clusters. For example, using bounds on the pdf function can help control how uniformly spaced the initial k-means clusters (or, more precisely, the k-means cluster center values ​​used for k-means initialization) are from each other. If the initialization is entirely pdf-based, more initial clusters are assigned to more frequent symbols. While this can help improve the accuracy of quantization (i.e., reduce distortion), it can also move the cluster populations (i.e., the number of symbols in each cluster) closer together, which ultimately reduces entropy and therefore increases the size of the bitstream (i.e., in terms of rate-distortion, considering MSE and network inference error rate as distortion and the size of the compressed model as rate). Therefore, if you want to transmit the model in a constant amount of time, the size will be proportional to the bitrate.

[0073] According to at least some embodiments of the present disclosure, "pdf coefficients" are defined that indicate how PDF-based initialization is applied to quantization (e.g., k-means quantization). Depending on the embodiment of the present disclosure, the format of the pdf coefficients may vary.

[0074] For example, in some embodiments, the "pdf coefficient" may be a number ranging from 0.0 to 1.0. Thus, in such embodiments, the "pdf coefficient" represents how much PDF-based initialization we want to apply. A pdf coefficient value of "0" means we do not want to use any PDF-based initialization. In this case, the k-means algorithm is uniformly initialized. A pdf coefficient value of 1.0 means we want to use the PDF feature with its maximum effect. (For example, in the following formula, the lower limit is 0 and the upper limit is twice the mean value.)

[0075] In some embodiments, an exemplary bounded PDF function used to initialize K-means can be defined as follows: BoundedPDF=Clip(PDF, LowerBound, UpperBound) where: LowerBound=(1.0-pdfFactor) * Avg(PDF) UpperBound=(1.0+pdfFactor) * Avg(PDF) Here, PDF is a probability density function, which can give the likelihood of a random variable taking on a specified value.

[0076] As described above, in accordance with at least some embodiments of the present disclosure, one of the inputs to the quantization algorithm can be a first distortion value (such as a desired maximum MSE value, MaxMSE). The quantization algorithm can use, for example, a binary search algorithm on the encoder side to find the minimum number of symbols (i.e., codebook size) required to quantize a tensor while keeping the distortion (or error) below a specified maximum. (MSE <maxMSE)

[0077] In at least some embodiments of the present disclosure, during quantization at a specified symbol count (or codebook size), the PDF coefficients can be varied to improve the entropy of the quantized information. For example, if the values ​​of the PDF coefficients can be defined in a range between a first PDF coefficient value and a second PDF value, the values ​​of the PDF coefficients can be varied from the first value to the second value in fixed and / or variable increments, and the entropy of the quantized information can be calculated by a computer to select, for each value of the PDF coefficient, the corresponding PDF coefficient with the lowest entropy.

[0078] According to a first example, the value of the PDF coefficient is changed incrementally (e.g., in increments of 0.1) within a range from the minimum PDF coefficient value to the maximum PDF coefficient value (e.g., 0 to 1), and the entropy of the quantized information for each value of the PDF coefficient can be calculated. According to a second example, the PDF coefficient is changed incrementally within a range from the maximum PDF coefficient value to the minimum PDF coefficient value (e.g., 1 to 0) in increments of 0.1, and the entropy of the quantized information for each PDF coefficient can be calculated. Thereby, the PDF coefficient corresponding to the lowest entropy in the first and / or second examples is determined.

[0079] In at least some embodiments of the present disclosure, two nested loops can be performed, the first optimizing for MSE (such as a binary search to find the best symbol count given a maxMSE value) and the second optimizing for entropy (searching for the pdf coefficients that result in the lowest entropy).

[0080] In at least some embodiments of the present disclosure, a first loop that facilitates optimizing MSE may be performed, but a second loop that facilitates optimizing entropy may not be performed, or vice versa.

[0081] For example, optimization of codebook size relative to MSE can be performed in embodiments where a codebook quantization initialization method other than boundary PDF quantization is used, and boundary PDF quantization using PDF coefficients that reduce entropy can be performed with a codebook size determined from the input q bits, as described above.

[0082] 7 illustrates an example of a method 700 for quantizing parameters of a tensor of a DNN layer according to the above-described embodiments. At 701, a pdf coefficient is selected from among several candidate boundary pdf coefficients based on entropy obtained for the quantized parameters from each one of the several candidate boundary pdf coefficients, as described above. At 702, the parameters of the tensor are quantized using a pdf-based initialization bounded according to the selected pdf coefficient. The embodiment described in conjunction with FIG. 7 may be implemented in the method for encoding a DNN described in conjunction with FIG. 4.

[0083] In some embodiments, the methods illustrated in Figures 6 and 7, respectively, can be combined, as described above.

[0084] Integrated Codebook Information At least some of the embodiments of the present disclosure also propose an exemplary codebook format ("unified codebook information") that is adapted to be used for several types of quantization, and that can cover, among other things, both codebook quantization and uniform quantization.

[0085] In at least some embodiments of the present disclosure, the integrated codebook information can be associated with an integer array (hereinafter referred to as codebook information).

[0086] The integrated codebook information may include first information indicating a type of codebook. For example, in an exemplary format in which the integrated codebook information is an integer array, the type of codebook may be specified by a first integer in the array. Examples of codebook types are listed below.

[0087] Some components in the joint codebook information may be determined by the codebook type. For example, in the above exemplary format, the component following the first integer in the joint codebook information may be determined by the codebook type.

[0088] For some codebook types (such as types "1" and "2" below), the actual quantized tensors can be omitted in the bitstream (i.e., no exponents are coded in the bitstream) because the codebook information contains all the information needed to reconstruct the tensors.

[0089] Examples of codebook types and associated integrated codebook information are provided below: Of course, the designation and / or numbering of the types is exemplary only and cannot be considered limiting of the present disclosure.

[0090] Codebook type 1: The integer values ​​of the tensor components are always the same. This is a rare case that sometimes occurs in bias tensors (e.g., as seen in some convolutional layers of the image classification neural network "ResNet50" studied in MPEG-NNR). Codebook info: [1, intVal] where intVal is the integer value for all components of the tensor.

[0091] Codebook type 2: The floating-point values ​​of the tensor components are always approximately the same. This means that if a tensor is uniformly quantized, all the resulting integer values ​​will be the same. This is a rare case that sometimes occurs in bias tensors (e.g., "ResNet50"). Codebook info: [2, rangeInt, symCount, offset] The floating-point value floatVal used for the components of the tensor can be calculated as follows: step=float(rangeInt) / (symCount-1) floatVal=offset * step "symCount" is the symbol count during quantization obtained by the quantization algorithm (binary search based on maxMSE). The values ​​of "rangeInt" and "offset" are calculated from the original tensor using, for example, the arithmetic functions ceil and floor as follows: rangeInt=ceil(max(tensor)-floor(min(tensor)) step=float(rangeInt) / (symCount-1) offset=round(min(tensor) / step

[0092] Codebook Type 3: Uniform quantization of tensors. Codebook Info:[3, rangeIntrangelnt, symCount, offset] For each integer component 'q' in the quantized tensor, the corresponding floating-point component 'r' in the reconstructed tensor can be calculated as follows: step=float(rangeInt) / (symCount-1) r=(q+offset) * step Here, rangeInt, symCount, and offset have the same meaning as above.

[0093] Codebook Type 0: Codebook quantization. Codebook Info:[0, cbRangeInt, cbSymCount, cbOffset, cbInt0, cbInt1, ..., cbInt N-1 ] In this case, cbInt0 to cbInt in the array "Codebook info" N-1 The components of rcbFloat are the N components of the codebook quantized to integer values. First, e.g., at the decoder side, we reconstruct the floating-point codebook and then use it to reconstruct (i.e., look up) the tensor components. i is the integer cbInt in the codebook information. i Assuming that the floating-point reconstructed codebook elements corresponding to , the codebook can be reconstructed as follows: cbStep=float(cbRangelnt) / (cbSymCount-1) rcbFloat i =(cbInt i +Offset) * cbStep recCodebook=[rcbFlat0, rcbFloat1, ..., rcbFloat N-1 ] Here, recCodebook represents the codebook reconstructed from the codebook information. For each integer component "q" in the quantized tensor (i.e., in the "exponent"), the corresponding floating-point component "r" in the reconstructed tensor is: r=recCodebook[q] On the encoding side, the original codebook after K-means quantization is: floatCodebook=[cbFloat0, cbFloat1, ..., cbFloat N-1 ] The values ​​of "cbRangeInt", "cbSymCount", and "cbOffset" can be set after codebook quantization of the tensor as follows: cbRangeInt=ceil(max(floatCodebook))-floor(min(floatCodebook)) cbSymCount=max(MinCbSymCount,symCount 2 ) cbStep=float(cbRangeInt) / (cbSymCount-1) cbOffset=round(min(floatCodebook) / step) cbInt i =round(cbFloat i / cbStep)-cbOffset where cblnt i is the original floating-point codebook cbFloat i is the quantized integer value corresponding to the i-th component in where MinCbSymCount represents the maximum codebook size, which may vary depending on the implementation, and is 2 20 or 2 12 (=4096).

[0094] As a non-limiting example, some embodiments may be applied to non-standardized technologies, but some embodiments may be used in the context of standards for DNN compression / decompression, such as the upcoming standard ISO / MPEG7, which concerns neural network compressed representations for multimedia content description and analysis, hereafter more simply referred to as MPEG NNR.

[0095] Some embodiments of the present disclosure may include transmitting / receiving signaling information between an encoder and a decoder. This signaling information is presented in this disclosure in conjunction with an exemplary, non-limiting syntax. For ease of explanation, this exemplary syntax is based on the syntax used in the exemplary MPEG NRR draft standard (N19225 - Working Draft 4 of Compression of neural networks for multimedia content description and analysis, International Organization for Standardization ISO / IEC JTC1 / SC29 / WG11, April 2020). Differences from this exemplary MPEG NRR syntax are underlined in the syntax table.

[0096] The following syntax is merely an exemplary syntax that does not limit the present disclosure. For example, the number of bits used for syntax elements is an exemplary embodiment. For ease of understanding, in the exemplary syntax, the following identifiers and clauses according to embodiments of the present disclosure are numbered with section and table numbers that are consistent with the current exemplary working draft of MPEG-NNR.

[0097] According to the exemplary syntax detailed herein, the bitstream can be divided into units that represent one tensor. The parameters contained in the header of each unit, i.e., nnr_compressed_data_unit_header.

[0098] As shown below, an exemplary syntax is detailed in conjunction with the current version of the MPEG NRR draft standard, which specifies parsing of the codebook_size codebook component stored as a float32 value.

[0099] [Table 1]

[0100] In accordance with at least some embodiments of the present disclosure, using the exemplary syntax introduced above, we propose to revise the definition of nnr_compressed_data_unit_header() { so that it is adapted to correspond to the codebook mechanism (or format) proposed by the present principles. The new parts are underlined.

[0101] [Table 2]

[0102] [Table 3]

[0103] In the above example syntax, all signed and unsigned integer values ​​in this table use a variable number of bytes. In some example implementations, the functions defined below are used to serialize / deserialize these integer values ​​to / from the byte stream. Other variable-length methods of serialization / deserialization may be used in some embodiments.

[0104] Example functions for serializing / deserializing these integer values ​​to / from a byte stream: The following functions can be used to serialize signed / unsigned integers into a byte stream, where the number of bytes used in the byte stream can be determined by the integer value:

[0105] [Table 4] The following functions deserialize signed / unsigned integers from a byte stream when decoding a bitstream.

[0106] [Table 5]

[0107] 8 illustrates an example method 800 for encoding a DNN according to one embodiment. At 810, parameters of the tensors of the DNN layers are quantized using a codebook and coded in a bitstream.

[0108] When quantizing the parameters, a codebook having the type defined above is used. As described above, at 820, information representing the type of codebook used to quantize the tensor parameters is encoded in the bitstream to signal to the decoder the codebook type used.

[0109] 9 illustrates an example method 900 for decoding a DNN according to one embodiment. At 910, information representing a codebook type is decoded from an input bitstream containing data representing at least one tensor of a layer of the DNN. At 920, parameters of the tensor are decoded from the bitstream and dequantized using a codebook of the type indicated by the decoded information, as described above.

[0110] The embodiments of the encoding method 800 and the decoding method 900 described in relation to Figures 8 and 9, respectively, can be implemented in the method for encoding a DNN described in relation to Figure 4 and the method for decoding a DNN described in relation to Figure 5.

[0111] In some embodiments, the aspects described above and in relation to Figures 6, 7, and 8 may be combined.

[0112] 10 illustrates an example of a portion of a bitstream STR_100 including data representing tensors of at least one layer of a deep neural network generated according to an embodiment. The bitstream may be generated, for example, according to any one of the above embodiments. The portion illustrated in FIG. 10 includes data representing a tensor (STR_101) and information representing the type of codebook used to quantize the parameters of the tensor (STR_102).

[0113] Further embodiments and information This application describes various aspects, including tools, features, embodiments, models, techniques, and the like. Many of these aspects are described with specificity, often in a definitive manner, to at least indicate their individual characteristics. However, this is for clarity of description and does not limit the applicability or scope of the aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. These aspects can also be combined and interchanged with aspects described in previous applications.

[0114] The aspects described and embraced in this application can be implemented in many different forms. While Figures 1, 2, and 3 below provide some embodiments, other embodiments are also embraced, and the discussion of Figures 1, 2, and 3 is not intended to limit the breadth of implementations. At least one of the aspects generally relates to encoding and decoding (e.g., video encoding and video decoding, and / or encoding and decoding at least some parameters of at least some layers of a DNN), and at least one other aspect generally relates to transmitting a generated or encoded bitstream. Aspects such as these can be implemented as a method, an apparatus, or a computer-readable storage medium having stored thereon instructions for encoding or decoding data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0115] In this application, the terms "reconstructed" and "decoded" can be used interchangeably, the terms "pixel" and "sample" can be used synonymously, and the terms "image," "picture," and "frame" can be used interchangeably. Typically, although not necessarily, the term "reconstructed" is used on the encoder side, while the term "decoded" is used on the decoder side.

[0116] Various methods and other aspects described herein can be used to modify modules of the encoder 100 and decoder 200, such as the intra-prediction module, entropy coding module, and / or decoding module (160, 260, 145, 230), as shown in Figures 1 and 2. Additionally, aspects of the present disclosure are not limited to VVC or HEVC, but may also be applied to other standards and recommendations, whether existing or developed in the future, as well as extensions of any such standards and recommendations (including VVC and HEVC).

[0117] Furthermore, this aspect is not limited to VVC or HEVC, or even to video data, but may be applied to an encoder or decoder adapted to encode or decode, respectively, at least one tensor of at least one layer of a neural network that may be used in many technical fields other than video (of course, in such an embodiment, some modules, such as intra prediction module 160, may be optional).

[0118] Unless otherwise specified or technically excluded, the aspects described in this application can be used individually or in any combination.

[0119] In this application, various numerical values ​​are used (e.g., pdf coefficient ranges, or increments for changing pdf coefficients, or maximum codebook sizes used in some computations), and specific values ​​are provided for illustrative purposes, and the described aspects are not limited to these specific values.

[0120] 1 shows an encoder 100. While variations of this encoder 100 are contemplated, for purposes of clarity, the encoder 100 is described below without necessarily describing all anticipated variations.

[0121] Before being encoded, the sequence may undergo pre-encoding processing (101), such as applying a color transformation to the input color picture in the case of video conversion (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or remapping the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Pre-encoding processing may also include binarization, such as the exemplary binarization detailed above in connection with CABAC.

[0122] Metadata can be associated with the preprocessing and added to the bitstream.

[0123] In encoder 100, a picture is coded by encoder elements as described below. The picture to be coded is divided into units, e.g., CUs (102), and processed. Each unit is coded, e.g., using either intra mode or inter mode. If a unit is coded in intra mode, it undergoes intra prediction (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to code the unit in intra mode or inter mode, and indicates the intra / inter decision, e.g., by a prediction mode flag. A prediction residual is calculated (110), e.g., by subtracting the prediction block from the original image block.

[0124] The prediction residual is then transformed (125) and quantized (130).

[0125] The quantized transform coefficients as well as syntax elements such as motion vectors are entropy coded to output a bitstream (145).

[0126] In the encoder, the transform can be skipped and quantization can be applied directly to the untransformed residual signal. In the encoder, both the transform and quantization can be ignored, i.e., the residual is coded directly without applying either the transform or quantization process.

[0127] In the encoder, the coded block is decoded to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150), and the prediction residual is decoded. For example, for a video sequence, an image block is reconstructed by combining the decoded prediction residual with the prediction block (155). An in-loop filter (165) is applied to the reconstructed picture to reduce coding artifacts, for example, by performing deblocking / sample adaptive offset (SAO) filtering. The filtered image is stored in a reference picture buffer (180).

[0128] Figure 2 shows a block diagram of a decoder 200. In the decoder 200, the bitstream is decoded by decoder elements as described below. The decoder 200 typically performs a decoding pass that is almost the reverse of the encoding pass, as described in Figure 1. The encoder 100 also typically performs decoding as part of encoding data.

[0129] Specifically, the input of decoder 200 includes a bitstream, which may be generated by encoder 100. The bitstream is first entropy decoded (230) to obtain coded information such as transform coefficients, motion vectors, etc.

[0130] For a video bitstream, picture partition information indicates how a picture is partitioned, allowing a decoder to separate the picture according to the decoded picture partition information (235). To decode the prediction residual, the transform coefficients are inverse quantized (240) and inverse transformed (250). The decoded prediction residual is combined with the prediction block (255) to reconstruct an image block. The prediction block can be obtained (270) from intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0131] The decoded elements (such as picture or layer parameters) may further undergo post-decoding processing (285), such as, in the case of a decoded image, an inverse color conversion (e.g., YCbCr 4:2:0 to RGB 4:4:4 conversion) or an inverse remapping that reverses the remapping process performed in pre-encoding processing (101). Post-decoding processing can use metadata derived in pre-encoding processing and signaled in the bitstream.

[0132] FIG. 3 illustrates a block diagram of an example system in which various aspects and embodiments may be implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 1000, singly or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0133] The system 1000 includes at least one processor 1010 configured to execute instructions loaded thereon, for example, to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile and / or non-volatile storage device). The system 1000 includes a storage device(s) 1040, which may include non-volatile and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random-Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. The storage device(s) 1040 may include, by way of non-limiting example, internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0134] The system 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, an encoded or decoded data stream (e.g., a video stream and / or a stream representing at least one parameter of at least one tensor of at least one layer of at least one DNN), which may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device that performs encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0135] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the performance of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, data representing at least one parameter of at least one tensor of at least one layer of at least one DNN, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and operational logic.

[0136] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM is used as working memory for coding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding is a new standard being developed by the Joint Video Experts Team of JVET).

[0137] As shown in block 1130, inputs to the elements of system 1000 can be provided through various input devices. Such input devices include, but are not limited to, (i) a Radio Frequency (RF) section that receives RF signals transmitted throughout a broadcast, for example, by a broadcaster; (ii) a Component (COMP) input terminal (or set of COMP input terminals); (iii) a Universal Serial Bus (USB) input terminal; and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Further examples, not shown in FIG. 3, include composited video.

[0138] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error correction device, and a demultiplexer. The RF section can include, for example, a tuner that performs a variety of these functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and filtering again to the desired frequency band an RF signal transmitted over a wired (e.g., cable) medium. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0139] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices over USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented in a separate interface IC or within processor 1010, as desired. For example, processor 1010, working in combination with memory and storage elements, provides the demodulated, error corrected, and demultiplexed stream to various processing elements, including encoder / decoder 1030, to process the data stream as desired for presentation on an output device.

[0140] The various elements of system 1000 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between each other using suitable connection devices 1140, for example, internal buses such as those known in the art, including Inter-IC (I2C) buses, wiring, and printed circuit boards.

[0141] System 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit data and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented in a wired and / or wireless medium, for example.

[0142] In various embodiments, data is streamed or otherwise provided to system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received over communication channel 1060 and communication interface 1050, which are adapted for Wi-Fi communication. Communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable over-the-top communication, such as streaming applications. In other embodiments, streaming data is provided to system 1000 using a set-top box that delivers data over an HDMI connection in input block 1130. In still other embodiments, streaming data is provided to system 1000 using an RF connection in input block 1130. As noted above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0143] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be for a television, a tablet, a laptop, a mobile phone, or another device. Display 1100 can also be integrated with other components (e.g., as found in a smartphone) or separate (e.g., an external monitor for a laptop). Other peripheral devices 1120, in various example embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (DVR, as an abbreviation for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of system 1000. For example, a disc player serves to play the output of the system 1000 .

[0144] In various embodiments, control signals are communicated between system 1000 and display 1100, speaker 1110, or other peripheral device 1120 using AV or other signaling, including communication protocols such as Consumer Electronics Control (CEC), which allow for inter-device control with or without user intervention. Output devices may be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 using communication channel 1060 via communication interface 1050. Display 1100 and speaker 1110 may be integrated into a single unit with the other components of system 1000 in an electronic device such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0145] Alternatively, the display 1100 and speakers 1110 may be separate from one or more of the other components, for example, if the RF portion of the input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0146] Embodiments may be implemented by computer software implemented by the processor 1010, by hardware, or by a combination of hardware and software. As a non-limiting example, embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technology environment, and may be implemented using any suitable data storage technology, such as, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and rib-bubble memory. The processor 1010 may be of any type suitable for the technology environment, and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0147] In various implementations, this involves decoding. As used herein, "decoding" can encompass all or some of the processes performed on a received encoded sequence to generate a final output suitable for, for example, a display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as, for example, entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by decoders in various implementations described herein.

[0148] As a further example, in one embodiment, "decoding" refers to entropy decoding only, in another embodiment, "decoding" refers to differential decoding only, and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer to a subset of operations specifically or to the broader decoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0149] In various implementations, this involves encoding. Similar to the above discussion regarding "decoding," "encoding," as used herein, can encompass all or part of the processes performed on an input video sequence to, for example, generate an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as, for example, partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by the encoders of various implementations described herein.

[0150] As a further example, in one embodiment, "encoding" refers to entropy encoding only, in another embodiment, "encoding" refers to differential encoding only, and in yet another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process as a whole will become clear based on the context of the specific description and will be readily apparent to those skilled in the art.

[0151] It should be noted that syntax element, as used herein, is a descriptive term and therefore does not preclude the use of other syntax element names.

[0152] Where a figure is presented as a flow diagram, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow diagram of the corresponding method / process.

[0153] Various embodiments refer to parametric models or rate-distortion optimization. Specifically, during the encoding process, the balance or trade-off between rate and distortion is typically considered, often subject to computational complexity constraints. This can be measured through a Rate Distortion Optimization (RDO) metric or other such measures, such as Least Mean Square (LMS), Mean of Absolute Errors (MAE), etc. Rate-distortion optimization is typically formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all coding options, including all possible modes or coding parameter values, with a thorough evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Also, faster approaches may be used to reduce coding complexity, especially with the calculation of approximate distortion based on a prediction or prediction residual signal rather than a reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some of the possible encoding choices and full distortion for others. Other approaches evaluate only a subset of the possible encoding choices. More generally, many approaches employ any of a variety of techniques to perform optimization, but the optimization does not necessarily involve a complete evaluation of both the coding cost and the associated distortion.

[0154] Implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of an implementation form (e.g., discussed only as a method), the implementation of the discussed feature may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. A method may be implemented in, for example, a processor, which refers to a general processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0155] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that at least some of the features, structures, characteristics, etc. described in connection with the embodiment are included in at least some of the embodiments. Thus, the appearances of the phrases "in one embodiment" or "an embodiment" or "in one implementation" or "in an implementation" appearing in various places throughout this specification, as well as any other variations, are not necessarily all referring to the same embodiment.

[0156] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0157] Also, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0158] Additionally, this application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from memory). Furthermore, "receiving" typically involves some form of activity, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0159] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to refer to and encompass selection of only the first listed alternative (A), or selection of only the second listed alternative (B), or selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass selection of only the first listed alternative (A), or selection of only the second listed alternative (B), or selection of only the third listed alternative (C), or selection of only the first and second listed alternatives (A and B), or selection of only the first and third listed alternatives (A and C), or selection of only the second and third listed alternatives (B and C), or selection of all three alternatives (A, B, and C). This can be expanded to include as many of the items listed as would be apparent to one skilled in this and related arts.

[0160] Also, as used herein, the term "signaling" refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals at least one of multiple transforms, coding modes, or coding flags. Thus, in some embodiments, the same parameters are used on both the encoder and decoder sides. Thus, for example, an encoder can transmit a particular parameter to a decoder (explicit signaling) so that the decoder can use the same particular parameter. Conversely, if the decoder already has that particular parameter as well as other parameters, it can use signaling instead (implicit signaling) to allow the decoder to know and select that particular parameter. By avoiding transmitting any actual functionality, bit savings are realized in various embodiments. It should be understood that signaling can be achieved in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a decoder. Although the above refers to the verb form of the word "signal," the word "signal" may also be used as a noun herein.

[0161] As will be appreciated by those skilled in the art, implementations can produce a variety of signals formatted to carry information that can be stored or transmitted, for example. Information can include, for example, instructions to perform a method or data produced by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

[0162] Several embodiments are described. Features of these embodiments may be provided alone or in any combination, across various claim categories and types. Also, embodiments may include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types: A process or device for encoding and decoding using deep neural network compression of a pre-trained deep neural network. A process or device that performs deep neural network compression by encoding and decoding at least one layer of a pre-trained deep neural network. A process or device that performs deep neural network compression of a pre-trained deep neural network with one or more layers by encoding and decoding using information inserted into a bitstream representing parameters. A process or device that performs deep neural network compression of deep neural networks by encoding and decoding using information inserted into a bitstream representing parameters. A bitstream or signal containing one or more of the syntax elements described, or variations thereof. A bitstream or signal including syntax carrying information generated according to any of the described embodiments. · Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments. A method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments. Inserting syntax elements into the signaling that allow the decoder to determine the coding mode in a manner that corresponds to the method used by the encoder. · Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof. An electronic device, such as a television, a set-top box, a mobile phone, a tablet, etc., that performs a conversion method according to any of the described embodiments. An electronic device, such as a television, set-top box, mobile phone, tablet, etc., that makes the transformation method determination and displays the resulting image (e.g., using a monitor, screen, etc. type of display) according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that selects, bandlimits, or tunes (e.g., using a tuner) a channel to receive a signal containing the encoded image and performs a conversion method according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing coded images over the air and performs a conversion method.

Claims

1. 1. A method comprising: obtaining a codebook including a codebook size for quantizing parameters of a tensor associated with at least one layer of a deep neural network, the codebook size being obtained according to a distortion value determined between the tensor and a quantized version of the tensor; quantizing the parameters of the tensor using the obtained codebook to represent the parameters with at least the determined size.

2. encoding the quantized parameters in a bitstream to be transmitted; The method of claim 1 , further comprising: transmitting the quantized coded parameters in the bitstream to a decoder.

3. The method of claim 1 , further comprising obtaining the codebook size from a binary search over a range of codebook sizes.

4. quantizing based on a PDF-based initialization bounded according to the first PDF coefficient; The method of claim 1 , further comprising: selecting from candidate boundary pdf coefficients, the candidate boundary pdf coefficients being based on entropy obtained from candidate quantized parameters.

5. The method of claim 1 , further comprising encoding information representing a codebook type corresponding to the codebook.

6. 1. An apparatus comprising one or more processors, the one or more processors comprising: obtaining a codebook including a codebook size for quantizing parameters of a tensor associated with at least one layer of a deep neural network, the codebook size being obtained according to a distortion value determined between the tensor and a quantized version of the tensor; quantizing the parameters of the tensor using the obtained codebook to represent the parameters with at least a determined size.

7. the one or more processors: encoding the quantized parameters in a bitstream to be transmitted; The apparatus of claim 6 , further configured to: and transmitting the quantized coded parameters in the bitstream to a decoder.

8. The apparatus of claim 6 , wherein the one or more processors are further configured to obtain the codebook size from a binary search over a range of codebook sizes.

9. the one or more processors: quantizing based on a PDF-based initialization bounded according to the first PDF coefficient; 7. The apparatus of claim 6, further configured to: select, from the candidate boundary PDF coefficients, the candidate boundary PDF coefficients based on entropy obtained from candidate quantized parameters.

10. The apparatus of claim 6 , wherein the one or more processors are further configured to encode information representing a codebook type corresponding to the codebook.

11. receiving an encoded bitstream, the encoded bitstream including quantized parameters of a tensor associated with at least one layer of a deep neural network, the encoded bitstream including a codebook including a codebook size obtained according to a distortion value determined between the tensor and a quantized version of the tensor; decoding the codebook from the bitstream; and performing inverse quantization of the parameters of the tensor using the codebook.

12. analyzing the input bins to derive quantized parameters; dequantizing the quantized parameters to derive final parameter values; The method of claim 11 , further comprising: inversely transforming the final parameter values.

13. The method of claim 11 , further comprising dequantizing based on a pdf-based initialization bounded according to the first pdf coefficient.

14. 1. An apparatus comprising one or more processors, the one or more processors comprising: receiving an encoded bitstream, the encoded bitstream including quantized parameters of a tensor associated with at least one layer of a deep neural network, the encoded bitstream including a codebook including a codebook size obtained according to a distortion value determined between the tensor and a quantized version of the tensor; decoding the codebook from the bitstream; and performing dequantization of the parameters of the tensor using the codebook.

15. 15. The apparatus of claim 14, wherein the one or more processors are further configured to: analyze input bins to derive quantized parameters; dequantize the quantized parameters to derive final parameter values; and inverse transform the final parameter values.

Images