System and method for encoding / decoding deep neural networks

Abstract

We propose a method that involves encoding data in at least one bitstream. [Solution] This disclosure relates to a method comprising quantizing the parameters of an input tensor, wherein the quantization uses a codebook whose size is obtained according to a distortion value determined between at least one tensor and a quantized version of said at least one tensor. This disclosure also relates to a method for quantizing the parameters of an input tensor using initialization based on a PDF bounded according to at least one first PDF coefficient, wherein the first PDF coefficient is selected from several candidate bounding PDF coefficients according to the resulting entropy. This disclosure also relates to corresponding signals; bitstreams, storage media, and encoder and / or decoder devices.

Description

[Technical Field]

[0001] The domain art of one or more embodiments of this disclosure relates to the domain of data processing technologies, such as data compression and / or data decompression. For example, at least some embodiments relate to data compression / decompression associated with large amounts of data, such as compression and / or decompression of at least a portion of audio streams and / or video streams, or data compression and / or decompression associated with deep learning techniques, such as at least some parameters of a deep neural network (DNN).

[0002] At least some embodiments relate to improving compression efficiency compared to video compression systems under development such as HEVC (HEVC is a high-efficiency video coding, also known as H.265 and MPEG-H Part 2, as described in "ITU-T H.265 Telecommunication standardization sector of ITU (10 / 2014), series H: audiovisual and multimedia systems, infrastructure of audiovisual services - coding of moving video, High efficiency video coding, Recommendation ITU-T H.265"), or compared to video compression systems under deployment such as VVC (Variable-purpose video coding, a new standard being developed by the JVET Integrated Video Experts Team).

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transformation, including spatial vector prediction and / or motion vector prediction, to leverage spatiotemporal redundancy in video content. Generally, intra-prediction or inter-prediction is used to leverage intra-frame or inter-frame correlation, thereby transforming, quantizing, and entropicating the difference between the original and predicted images, often called prediction error or prediction residual. To reconstruct the video, the compressed data is decoded through the reverse process corresponding to entropicating, quantization, transformation, and prediction.

[0004] At least some embodiments relate to improving compression efficiency compared to existing systems for deep neural network (DNN) compression, such as several compression standards or draft standards, including the upcoming ISO / MPEG7 neural network standard for multimedia content description and analysis currently being developed by the International Organization for Standardization.

[0005] Typically, the encoding process involves quantizing the DNN parameters, entropy coding, and obtaining compressed data. To reconstruct the data, the compressed data is decoded, and the decoding process involves entropy decoding and inverse quantization. [Overview of the project]

[0006] The principles of the present invention make it possible to overcome at least one drawback of several known compression and / or decompression methods by proposing a method comprising encoding data in at least one bitstream, wherein the data is one or more parameters of at least one tensor of at least one layer or sublayer of at least one deep neural network. It should be noted that the tensor of network parameters associated with a layer can include network parameters of the kind such as weights and / or biases (for example, with respect to a "Bach standardization" tensor).

[0007] One embodiment provides a method for encoding at least one tensor associated with at least one layer of a deep neural network in a bitstream. Encoding the at least one tensor involves obtaining the size of a codebook for quantizing the parameters of the at least one tensor, the size of which is obtained according to a distortion value determined between the at least one tensor and a quantized version of the at least one tensor, and quantizing the parameters using a codebook having the obtained size.

[0008] Another embodiment provides a device for encoding at least one tensor associated with at least one layer of a deep neural network in a bitstream, the device comprising at least one processor configured to obtain the size of a codebook for quantizing the parameters of the at least one tensor, the size of which is obtained according to a distortion value determined between the at least one tensor and a quantized version of the at least one tensor, and the parameters are quantized using a codebook having the obtained size.

[0009] Another embodiment provides a method for decoding at least one tensor associated with at least one layer of a deep neural network from a bitstream, wherein decoding the at least one tensor includes decoding information representing the type of codebook from the bitstream and using the codebook to dequantize the parameters of the at least one tensor.

[0010] Another embodiment provides a device for decoding from a bitstream at least one tensor associated with at least one layer of a deep neural network, the device comprising at least one processor configured to decode from the bitstream information representing the type of codebook and to dequantize the parameters of said at least one tensor using the codebook.

[0011] One aspect of the present disclosure relates to a device comprising at least one processor adapted to quantize the parameters of an input tensor, wherein the quantization is performed using initialization based on a PDF bounded by at least one first PDF coefficient, the first PDF coefficient being selected from several candidate bounding PDF coefficients according to the resulting entropy.

[0012] One aspect of the present disclosure relates to a method comprising quantizing the parameters of an input tensor, wherein the quantization is performed using initialization based on a PDF bounded by at least one first PDF coefficient, the first PDF coefficient being selected from several candidate bounding PDF coefficients according to the resulting entropy.

[0013] In some embodiments of the present disclosure, quantization is performed using codebook-based quantization, where the codebook size is obtained from several candidate codebook sizes according to the error between the input tensor and the quantized tensor obtained from the input tensor, using the candidate codebook size.

[0014] In another embodiment, the apparatus is provided, comprising a processor. The processor can be configured to encode at least one tensor of at least one layer of at least one deep neural network in at least one bitstream, and / or decode a bitstream representing at least one tensor of at least one layer of at least one deep neural network by performing one of the methods described herein.

[0015] Another generalized aspect of at least one embodiment provides a device comprising: an apparatus according to any of the decoding embodiments; and at least one of the following: (i) an antenna configured to receive data including input data; (ii) a band limiter configured to limit the received signal to a frequency band including the input data; or (iii) a display configured to display an output representing a video block.

[0016] Another generalized aspect of at least one embodiment provides a non-temporary computer-readable medium containing data content generated according to any of the decrypting embodiments or variations described.

[0017] A further generalized aspect of at least one embodiment provides a signal containing data representing at least one tensor of at least one layer or sublayer of at least one deep neural network, generated according to any of the decoded embodiments or variations described.

[0018] According to another generic aspect of at least one embodiment, the bitstream is formatted to include data content generated according to any of the decrypting embodiments or variations described.

[0019] According to another general form of at least one embodiment, there is provided a computer program product including instructions that cause a computer to perform any of the described decoding embodiments or variants when executed by the computer.

[0020] According to another general aspect of at least one embodiment, there is provided a computer-readable non-transitory program storage device tangibly embodying a program of instructions executable by a computer to perform at least one of the methods of the present disclosure in any of those embodiments.

[0021] According to another general aspect of at least one embodiment, there is provided a computer-readable storage medium including instructions that cause a computer to perform at least one of the methods of the present disclosure in any of those embodiments when executed by the computer.

[0022] Although not explicitly described, the device of the present disclosure can be adapted to perform the method of the present disclosure in any of those embodiments.

[0023] Although not explicitly described, any combination or sub-combination of the embodiments related to the method or corresponding signals, devices, and computer-readable storage media can be adopted.

Brief Description of the Drawings

[0024] [Figure 1] Shows a general standard encoding method. [Figure 2] Shows a general standard decoding method. [Figure 3] Shows a typical processor device in which the described embodiments can be implemented. [Figure 4] Shows a DNN encoding formula using at least some embodiments of the encoding method of the present disclosure. [Figure 5] Shows a DNN decoding method using at least some embodiments of the decoding method of the present disclosure. [Figure 6] This document presents an example of a method for quantizing the parameters of a tensor in a DNN layer according to one embodiment. [Figure 7] An example of a method for quantizing the tensor parameters of a DNN layer according to another embodiment is shown. [Figure 8] An example of a method for encoding the parameters of a DNN layer tensor according to one embodiment is shown. [Figure 9] An example of a method for decoding a DNN according to a certain embodiment is shown. [Figure 10] An example of a portion of a bitstream containing data representing the tensor of at least one layer of a deep neural network generated according to a certain embodiment is shown.

[0025] Please note that the drawings illustrate examples of embodiments, and the embodiments of this disclosure are not limited to those shown. [Modes for carrying out the invention]

[0026] Many technical fields involve the computer processing of large amounts of data. Such processing may involve data compression and / or decompression for the storage or transformation of at least a portion of such data. Examples of compressing and / or decompressing streams containing large amounts of data can be found in the field of video processing or in the field of deep learning techniques.

[0027] For illustrative purposes only, rather than limiting them, embodiments of this disclosure will be described in detail in connection with deep neural networks (DNNs). However, it is clear that this disclosure can also be applied to the compression / decompression of other large amounts of data, such as in the video processing technology field. For example, this disclosure can be applied to the compression / decompression of tensors obtained from at least one image by a deep learning algorithm.

[0028] Deep neural networks (DNNs) demonstrate a remarkable performance in diverse fields such as multimedia processing, computer vision, speech recognition, and natural language processing. However, this performance is undeniable, as DNNs often tend to have millions, sometimes billions, of parameters, potentially leading to enormous computational costs.

[0029] This can, for example, lead to excessively high inference complexity. Simply put, inference is, for example, the deployment of a DNN at the point in time when it was trained to process the input data, from a classification perspective. Inference complexity can be defined as the computational cost of applying the trained DNN to the input data being inferred.

[0030] Therefore, this high inference complexity can be a significant challenge when using DNNs in environments with electronic devices that have limited hardware and / or software resources, such as mobile or embedded devices with resource limitations including battery size, limited computing power, and memory capacity.

[0031] A deep neural network consists of several layers. Each layer is associated with a set of parameters that can be obtained, for example, during training of a DNN. These parameters (such as weights and / or biases) are stored as multidimensional arrays. For simplicity, the term “matrix” may be used below to refer to a set of parameters (e.g., the parameters of a tensor in a given layer). However, it should be understood that some embodiments of the methods of this disclosure can also be applied to the parameters of tensors of more than two dimensions, such as a 2D convolutional layer containing parameters for a 4D tensor. The enormous number of parameters in a DNN may require high bandwidth for deploying a DNN (or a solution containing a DNN) in a distributed environment.

[0032] At least some embodiments of this disclosure relate to the compression and / or decompression (decoding) of at least some parameters of at least one DNN (e.g., a pre-trained DNN). Indeed, compression can facilitate the transmission and / or storage of at least one DNN. More precisely, at least some embodiments of this disclosure relate to the compression of parameters of at least one tensor associated with at least one layer of at least one deep neural network. Network parameters can be, for example, a single tensor that is compressed at once.

[0033] Depending on the embodiments of this disclosure, the layers (or sublayers) may be of different types. For example, in some embodiments, at least one layer may be entirely a convolutional layer or a complete connector layer, or at least one layer may comprise at least one convolutional layer and / or at least one complete connector layer.

[0034] Figures 4 and 5 show, at a high level, a general process for encoding / decoding the parameters of at least one tensor of at least one DNN, which can be used in at least some embodiments of this disclosure.

[0035] In an exemplary embodiment of the compression method 400 shown in Figure 4, the method 400 may include obtaining (or in other words, retrieving) the parameters of the tensor to be compressed. Retrieval can be done, for example, by taking the parameters of at least one tensor from a storage unit, or by receiving the parameters from a data source via a communication interface.

[0036] As shown in Figure 4, compressing at least some parameters of the neural network requires, Quantizing the parameters of a neural network (such as weights and biases) 430, In other words, it can include lossless entropic coding 440 of quantized information.

[0037] This process makes it possible to represent parameters with fewer bits.

[0038] In some embodiments, compression 400 may further include a 420 step prior to quantization 430, which reduces the number of parameters in the neural network by taking advantage of the redundancy inherent in neural networks. Thus, reduction 420 provides at least one tensor with a reduced dimension compared to the dimension of the tensor associated with the layer, and an input to the reduction step. For example, the parameters of the tensor in at least one layer of the DNN can be decomposed or sparsed in reduction 420.

[0039] Typically, the resulting tensor, which contains floating-point values, is quantized, entropic-coded, and then converted into the final bitstream sent to the receiver.

[0040] This reduction of 420 is optional and can therefore be omitted in some embodiments.

[0041] Tensor quantization can involve approximating the values ​​of a tensor with integer-like values ​​that require fewer bits than the input tensor (e.g., floating-point values).

[0042] Depending on the DNN compression solution, different types of quantization can be performed. For example, quantization can be performed by using uniform quantization, or by using non-uniform quantization such as codebook-based quantization, as seen in some DNN compression solutions, and especially in some compression standards such as ISO / MPEG7, which is a coming standard for neural networks for describing and analyzing multimedia content, hereafter more simply referred to as MPEG NNR.

[0043] In uniform quantization, a floating-point "step size" can be defined, and all floating-point values ​​in a tensor can be represented as multiples of this step size. The approximate value of the tensor can be reconstructed in the decoder simply by multiplying the integer value by the "step size".

[0044] A codebook is the set of values ​​(such as integer or floating-point values) that a layer's parameters can hold after quantization. The exponents can be derived from the codebook values ​​assigned to the parameters of the original tensor through quantization. To reconstruct an approximation of the original tensor, the decoding device uses the indices to look up the corresponding floating-point values ​​from the codebook. Codebook-based quantization will be discussed in more detail later.

[0045] As shown in Figure 4, the parameter input to quantization may be of decimal type, but the output of quantization may include one or more tensors of integer exponents. Optionally, quantization may also output a codebook involving the exponents. Indeed, in some embodiments, for example, in embodiments where the codebook is fixed beforehand by quantization, the codebook can be omitted from the output of quantization.

[0046] As shown in Figure 4, at least some of the outputs of quantization 430 are used as input when performing lossless entropic coding 440. The information input for entropic coding (such as information about the codebook and / or integer-type tensors in the case of a quantized tensor) may, in some embodiments, be divided into header information and a payload containing exponents.

[0047] Other elements (such as the shape of the original tensor, i.e., the symbol count) can also be input to entropic coding 440.

[0048] If several layers can be encoded by encoding method 400, the method can be repeated layer by layer for the DNN after encoding the parameters associated with the layers, until the encoding of the parameters of the last layer to be encoded is completed (450).

[0049] Figure 5 shows a decoding method 500 that can be used to decode a bitstream obtained by the encoding method 400 described above. In the decoder, as shown in Figure 5, the encoding method 500 can include several inverse operations (compared to operations on the encoder side). For example, the decoding method 500 can include an analysis / entropy decoding of the input bins 510 to extract the quantized form of the parameters. Then, inverse quantization 520 can be applied to derive the final values ​​of the parameters. Matrix decomposition / sparsification of tensors in the encoder usually does not require an inverse process in the decoder. For example, a parameter set to zero in the parameter reduction stage (reduction 420) may remain zero after inverse quantization in the decoder.

[0050] As shown in Figure 5, the 510 output, which analyzes and decodes the bitstream corresponding to the layers of the DNN, can consist of metadata and quantized parameters. For example, when the encoder performs a codebook-based quantum, the output includes the codebook and the corresponding exponent. Both the exponent codebook and tensor can be obtained by computer using the K-means method, which derives a codebook of K values, which are shown as cluster centers and tensors of the exponent, and which can have values ​​belonging to the integer range [0...K-1].

[0051] The decoding method may further include performing inverse quantization 520 using the decoded information (such as the exponent and codebook).

[0052] When several layers can be decoded by the decoding method 500, the method 500 can be repeated until the parameters of the last layer are encoded (550).

[0053] As shown in Figure 4, at least some of the elements output by quantization 430 are used as input when entropy coding 440 is performed, and the method by which quantization is performed should conform to the method by which entropy coding is performed, and vice versa. This should be true, for example, whether the quantization is uniform quantization or codebook-based quantization (as the quantization used in some compression solutions such as the upcoming standard MPEG-NNR).

[0054] However, this is not always the case. For example, codebook-based quantization may have some drawbacks. For instance, codebook quantization may often be more efficient with larger tensors than with smaller tensors. One reason for this is that when tensors are small, the overhead of the codebook can become significant compared to the size of the tensor.

[0055] At least some embodiments of this disclosure help address this problem.

[0056] More precisely, in a first aspect of this disclosure, at least some embodiments propose using uniform quantization for smaller tensors and codebooks for larger tensors.

[0057] This disclosure also proposes exemplary formats adapted for use in several types of quantization (hereinafter referred to as “integrated formats”), in particular a format that covers both codebooks and uniform quantization.

[0058] Entropic coding is a lossless data compression method that works well on the fact that any data can be compressed if there is a data symbol that is more likely to occur than other data symbols. Examples of entropic coding methods include Huffman coding and arithmetic coding.

[0059] The entropy of quantized information directly affects the efficiency of arithmetic coding. If the quantized information has high entropy (e.g., high randomness), arithmetic coding algorithms cannot efficiently compress the data. For example, if symbols appear in the data with the same frequency (high randomness, high entropy), compression is inefficient. However, if some symbols appear more frequently than others, compression becomes more efficient by using fewer bits for the more frequent symbols.

[0060] In a second aspect of this disclosure, at least some embodiments propose methods for improving the efficiency of arithmetic coding performed during entropic coding (e.g., conditional arithmetic coding and / or adaptive arithmetic coding) by minimizing the entropy of quantized information.

[0061] Quantized information can be optimized (or at least improved) based on at least one first Mean Square Error (MSE) value (to control distortion), for example, by minimizing the Mean Square Error value while attempting to minimize (or at least reduce) the entropy of the quantized information (for example, by using PDF boundary initialization of the K-means algorithm).

[0062] This disclosure relates to several embodiments. Some embodiments, such as the first and second embodiments described above, may be implemented in the same embodiment and / or separately (some embodiments implement both embodiments, while some embodiments implement only one of the embodiments). For example, in some embodiments, a binary search for symbol counting can be performed without optimizing the pdf coefficients according to the MSE. However, embodiments that combine binary search for symbol counting with optimizing the pdf according to the MSE can often help obtain better rate / distortion results than when quantization only attempts to minimize the error (MSE) between the quantized tensor and the original tensor. Also, some embodiments of this disclosure may combine quantization as described herein, e.g., codebook quantization (as associated with the second embodiment) and conditional coding and / or adaptive arithmetic coding.

[0063] It should be noted that embodiments of the methods of the Disclosure described herein in detail may be implemented in many compression solutions and are not limited to any particular standard, and that at least some of the embodiments of the Disclosure may even be applicable to the background of several compression standards, such as some of the draft standards being developed by ISO / MPEG7.

[0064] Binary search for "best" symbol count The term symbol count, which represents the total number of all possible symbols that may appear during entropic coding, is equivalent to the codebook size in the case of codebook quantization.

[0065] Depending on the compression framework, the parameters input to the quantization stage may occasionally include a "qBitt" value indicating the number of bits used to represent each symbol in the codebook, and therefore the codebook size may be 2 qBit It is equal to.

[0066] However, in embodiments where arithmetic coding (or conditional arithmetic coding and / or adaptive arithmetic coding) is used after quantization, the codebook size does not have to be a power of 2. According to at least some embodiments of this disclosure, the codebook size (which is not necessarily a power of 2) can be obtained (or determined) with a first (specified) precision.

[0067] More precisely, according to some embodiments of the present disclosure, instead of specifying a “qBit” value, a first distortion value between the original tensor and the quantized tensor, such as a desired maximum mean squared error (MSE) value (hereinafter referred to as MaxMSE), can be specified in the encoder. Thereafter, the codebook size can be obtained according to this first MSE value, for example by using binary search.

[0068] In some embodiments of this disclosure, a binary search can be performed across a codebook size range, for example, a range of 4 to 4096 codebook sizes (corresponding to 2 to 12 "qBits").

[0069] Clearly, depending on the embodiment, the minimum and maximum values ​​of the range may vary depending on the embodiment, as may the minimum and maximum values ​​of "qBit", and the numerical values ​​(e.g., 4 or 4096 for the codebook size) are merely illustrative values. For example, in some embodiments, qBits values ​​from 2 to 20 (and the corresponding codebook size) can be used.

[0070] Depending on the embodiment, binary search can be applied to values ​​that increase or decrease monotonically. In fact, as the symbol count (or codebook size) increases, the MSE decreases monotonically. Therefore, binary search can be used to obtain the best (minimum) symbol count for a given MSE value (maxMSE).

[0071] Figure 6 shows an example of a method 600 for quantizing the parameters of a tensor in a DNN layer according to one embodiment. In 601, the size of the codebook for quantizing the parameters of at least one tensor in the DNN is obtained. As described above, the size of the codebook is obtained according to a distortion value determined between at least one tensor and the quantized version of that at least one tensor. In a variant, the size of the codebook is obtained using binary search across the entire codebook size range. In 602, the tensor parameters are quantized using the codebook having the size obtained in 601. The embodiment described in relation to Figure 6 may be implemented in the method for encoding a DNN described in relation to Figure 4.

[0072] Entropy quantization reduction using PDF-based initialization Some quantization solutions can be based on PDF-based initialization, such as PDF-based initialization of K-means clusters. For example, using boundaries on the PDF function can help control how uniformly the initial K-means clusters (more precisely, the center values ​​of the K-means clusters used for K-means initialization) are spaced apart from one another. If the initialization is entirely PDF-based, more initial clusters will be assigned to more frequent symbols. This can help improve the accuracy of quantization (i.e., reduce distortion), but it can also bring the cluster populations (i.e., the number of symbols in each cluster) closer together, which ultimately degrades entropy and thus increases the size of the bitstream (i.e., increases the rate, in terms of rate distortion, considering MSE and network inference error rate as distortion, and the size of the compressed model as the rate). Therefore, if you want to send the model in a constant time amount, the size will be proportional to the bitrate.

[0073] According to at least some embodiments of this disclosure, a “pdf coefficient” is defined that indicates how PDF-based initialization is applied to quantization (e.g., K-means quantization). Depending on the embodiment of this disclosure, the format of the pdf coefficient may vary.

[0074] For example, in some embodiments, the "pdf coefficient" may be a number in the range of 0.0 to 1.0. Therefore, in such embodiments, the "pdf coefficient" represents how much PDF-based initialization is desired. A value of "0" for the pdf coefficient means that no PDF-based initialization is desired at all. In this case, the K-means algorithm is initialized uniformly. A value of 1.0 for the pdf coefficient means that the PDF feature with the greatest effect is desired to be used. (For example, in the following equation, the lower bound is 0 and the upper bound is twice the mean).

[0075] In some embodiments, the exemplary boundary PDF function used for initializing the K-means algorithm can be defined as follows: BoundedPDF=Clip(PDF, LowerBound, UpperBound) Here, LowerBound=(1.0-pdfFactor) * Avg (PDF) UpperBound=(1.0+pdfFactor) * Avg (PDF) Here, PDF is a probability density function, which can give the likelihood of a random variable taking a given value.

[0076] As described above, according to at least some embodiments of this disclosure, one of the inputs to the quantization algorithm may be a first distortion value (such as a desired maximum MSE value, MaxMSE). The quantization algorithm may, for example, use a binary search algorithm on the encoder side to find the minimum number of symbols (i.e., codebook size) required to quantize the tensor while keeping the distortion (or error) below a specified maximum value. (MSE <maxMSE)

[0077] In at least some embodiments of this disclosure, the pdf coefficient can be varied to improve the entropy of the quantized information while quantizing at a specified symbol count (or codebook size). For example, if the value of the pdf coefficient can be defined within a range between a first pdf coefficient value and a second pdf value, the value of the pdf coefficient can be varied from the first value to the second value in constant and / or variable increments, and for each value of the pdf coefficient, the entropy of the quantized information can be determined by a computer to select the pdf coefficient corresponding to the lowest entropy case.

[0078] According to the first example, the value of the PDF coefficient can be gradually increased (e.g., in increments of 0.1) within the range of a minimum PDF coefficient value to a maximum PDF coefficient value (e.g., 0 to 1), and the entropy of the quantized information can be calculated for each value of the PDF coefficient. According to the second example, the PDF coefficient can be gradually decreased using increments of 0.1 within the range of a maximum PDF coefficient value to a minimum PDF coefficient value (e.g., 1 to 0), and the entropy of the quantized information can be calculated for each PDF coefficient. This determines the PDF coefficient corresponding to the lowest entropy in the first and / or second example.

[0079] In at least some embodiments of the present disclosure, two nested loops can be performed, the first loop prompting optimization for MSE (such as a binary search to find the best symbol count given a maxMSE value), and the second loop prompting optimization for entropy (investigating the pdf coefficient that yields the lowest entropy).

[0080] In at least some embodiments of the present disclosure, a first loop can be implemented that encourages optimization of the MSE, but a second loop that encourages optimization of entropy cannot, or vice versa.

[0081] For example, codebook size optimization for MSE can be performed in embodiments where a codebook quantization initialization method other than boundary PDF quantization is used, and boundary PDF quantization using PDF coefficients to reduce entropy can be performed with a codebook size determined from the input q bits, as described above.

[0082] Figure 7 shows an example of method 700 for quantizing the parameters of a DNN layer tensor according to the embodiment described above. In 701, as described above, the pdf coefficients are selected from several candidate boundary pdf coefficients based on the entropy obtained for the quantized parameters from each of several candidate boundary pdf coefficients. In 702, the tensor parameters are quantized using initialization based on pdf bounded according to the selected pdf coefficients. The embodiment described in relation to Figure 7 can be implemented in the method for encoding a DNN described in relation to Figure 4.

[0083] In one embodiment, the methods shown in Figures 6 and 7, respectively, can be combined as described above.

[0084] Integrated Codebook Information In at least some embodiments of the present disclosure, we also propose exemplary codebook formats ("Integrated Codebook Information") adapted for use with several types of quantization, and which, in particular, can cover both codebook quantization and uniform quantization.

[0085] In at least some embodiments of this disclosure, integrated codebook information can be associated with an integer array (hereinafter referred to as codebook information).

[0086] Integrated codebook information may include a first piece of information representing the codebook type. For example, in the exemplary format where integrated codebook information is an integer array, the first integer in the array can specify the codebook type. Examples of codebook types are given below.

[0087] Some components in the integrated codebook information can be determined by the codebook type. For example, in the example format above, the component following the first integer in the integrated codebook information can be determined by the codebook type.

[0088] In some codebook types (such as types "1" and "2" below), the codebook information contains all the information necessary to reconstruct the tensor, so the actual quantized tensor can be omitted in the bitstream (i.e., there are no indices encoded in the bitstream).

[0089] Examples of codebook types and associated integrated codebook information are shown below. Naturally, the type notations and / or numbering are illustrative and should not be considered as limiting the scope of this disclosure.

[0090] Codebook Type 1: The integer values ​​of the tensor components are always the same. This is a rare case that sometimes occurs in bias tensors (for example, as seen in some convolutional layers of the image classification neural network "ResNet50" studied in MPEG-NNR). Codebook information: [1, intVal] Here, intVal is the integer value for all components of the tensor.

[0091] Codebook Type 2: The floating-point values ​​of tensor components are always approximately the same. This means that if uniform quantization of a tensor is performed, all the resulting integer values ​​will be the same. This is a rare case that sometimes occurs with biased tensors (e.g., "ResNet50"). Codebook information: [2, rangeInt, symCount, offset] The floating-point value floatVal used for the components of a tensor can be calculated as follows: step=float(rangeInt) / (symCount-1) floatVal=offset * shop "symCount" is the symbol count obtained during quantization by the quantization algorithm (binary search based on maxMSE). The values ​​of "rangeInt" and "offset" are calculated from the original tensor using, for example, the arithmetic functions ceil and floor, as follows: rangeInt=ceil(max(tensor)-floor(min(tensor)) step=float(rangeInt) / (symCount-1) offset = round(min(tensor) / step)

[0092] Codebook Type 3: Uniform quantization of tensors. Codebook Info:[3, rangeIntrangelnt, symCount, offset] For each integer component “q” in the quantized tensor, the corresponding floating-point component “r” in the reconstructed tensor can be calculated as follows. step = float(rangeInt) / (symCount - 1) r = (q + offset) * step Here, rangeInt, symCount, and offset have the same meanings as above.

[0093] Codebook Type0: Codebook quantization. Codebook Info: [0, cbRangeInt, cbSymCount, cbOffset, cbInt0, cbInt1,..., cbInt N-1 In this case, the components of cbInt0 to cbInt in the array “Codebook info” are the N components of the codebook quantized to integer values. First, for example, on the decoder side, the floating-point codebook is reconstructed, and thereby used to reconstruct (i.e., look up) the tensor components. Assuming that rcbFloat N-1 is the floating-point reconstructed codebook component corresponding to the integer cbInt i in the codebook information, the codebook can be reconstructed as follows. i cbStep = float(cbRangelnt) / (cbSymCount - 1) cbStep = float(cbRangelnt) / (cbSymCount - 1) rcbFloat i = (cbInt i + Offset) * cbStep recCodebook = [rcbFlat0, rcbFloat1,..., rcbFloat N-1 Here, recCodebook represents the codebook reconstructed from the codebook information. ​​For each integer component "q" in the quantized tensor (i.e., in the "exponents"), the corresponding floating-point component "r" in the reconstructed tensor is as follows: r = recCodebook[q] On the encoding side, the original codebook after K-means quantization is as follows: floatCodebook=[cbFloat0, cbFloat1, ..., cbFloat N-1 ] The values ​​of "cbRangeInt", "cbSymCount", and "cbOffset" can be set after quantization of the tensor codebook, as follows: cbRangeInt=ceil(max(floatCodebook))-floor(min(floatCodebook)) cbSymCount=max(MinCbSymCount,symCount 2 ) cbStep=float(cbRangeInt) / (cbSymCount-1) cbOffset=round(min(floatCodebook) / step) cbInt i =round(cbFloat i / cbStep)-cbOffset Here, cblnt i This is the original floating-point codebook cbFloat i This is the quantized integer value corresponding to the i-th component in [the expression]. Here, MinCbSymCount represents the maximum codebook size, which may vary depending on the embodiment. 20 or 2 12 (=4096) is equal to this.

[0094] As a non-limiting example, some embodiments can be applied to non-standardized technologies, and some embodiments can be used in terms of standards for DNN compression / decompression, such as the upcoming ISO / MPEG7 standard, relating to neural network compressed representations for multimedia content description and analysis, hereafter referred to more simply as MPEG NNR.

[0095] Some embodiments of this disclosure may include transmitting / receiving signaling information between an encoder and a decoder. This signaling information is presented in this disclosure in conjunction with an exemplary, non-limiting syntax. For convenience of explanation, this exemplary syntax is based on the syntax used in the exemplary MPEG NNR draft standard (N19225 - Working Draft 4 of Compression of neural networks for multimedia content description and analysis) in the International Organization for Standardization (ISO / IEC JTC1 / SC29 / WG11, April 2020), and differences from this exemplary MPEG NNR syntax are underlined in the syntax table.

[0096] The following syntax is merely illustrative and does not limit the present disclosure. For example, the number of bits used for syntax elements is representative of the exemplary embodiment. For ease of understanding, in the exemplary syntax, the following identifiers and sections in embodiments of the present disclosure are numbered in sections and tables that are consistent with the current exemplary working draft of MPEG-NNR.

[0097] According to the exemplary syntax detailed herein, a bitstream can be divided into units representing a single tensor. Each unit has a header, i.e., parameters contained in nnr_compressed_data_unit_header.

[0098] The following details exemplary syntax linked to the current version of the MPEG NRR draft standard, which specifies the parsing of the codebook_size codebook component, which is stored as a float32 value.

[0099] [Table 1]

[0100] According to at least some embodiments of this disclosure, we propose revising the definition of nnr_compressed_data_unit_header(){ to be adapted to the codebook mechanism (or format) proposed by this principle, using the exemplary syntax introduced above. The new portion is underlined.

[0101] [Table 2]

[0102] [Table 3]

[0103] In the example syntax above, all signed and unsigned integer values ​​in this table use a variable number of bytes. In some exemplary implementations, these integer values ​​are serialized to / deserialized from a byte stream using the functions defined below. Other variable-length methods may be used for serialization / deserialization in some embodiments.

[0104] Exemplary functions for serializing / deserializing these integer values ​​into / from a byte stream. The following functions can be used to serialize signed / unsigned integers into byte streams. The number of bytes used in the byte stream can be determined by the integer value.

[0105] [Table 4] The following function deserializes signed / unsigned integers from a byte stream when decoding a bitstream.

[0106] [Table 5]

[0107] Figure 8 shows an example of a method 800 for encoding a DNN according to one embodiment. In 810, the parameters of the DNN layer tensor are quantized using a codebook and encoded in a bitstream.

[0108] When quantizing parameters, a codebook of the type defined above is used. As described above, in 820, information representing the type of codebook used to quantize the tensor parameters is encoded in the bitstream in order to signal the decoder the type of codebook to be used.

[0109] Figure 9 shows an example of a method 900 for decoding a DNN according to one embodiment. In 910, information representing the type of codebook is decoded from an input bitstream containing data representing at least one tensor of a layer of the DNN. In 920, as described above, the parameters of the tensor are decoded from the bitstream and dequantized using a codebook of the type indicated by the decoded information.

[0110] The embodiments of the encoding method 800 and decoding method 900 described in relation to Figures 8 and 9 can be implemented in the method for encoding a DNN described in relation to Figure 4 and the method for decoding a DNN described in relation to Figure 5.

[0111] In some embodiments, the embodiments described above and in relation to Figures 6, 7, and 8 can be combined.

[0112] Figure 10 shows an example of a portion of the bitstream STR_100 containing data representing a tensor of at least one layer of a deep neural network generated according to one embodiment. The bitstream is generated, for example, according to one of the embodiments described above. The portion shown in Figure 10 includes data representing a tensor (STR_101) and information (STR_102) representing the type of codebook used to quantize the parameters of the tensor.

[0113] Further embodiments and information This application describes various embodiments, including tools, features, embodiments, models, and methods. Many of these embodiments are described in detail, often in a way that sounds restrictive, in order to at least indicate their individual characteristics. However, this is for clarity and not to limit the application or scope of those embodiments. In fact, all of the different embodiments can be combined and interchanged to provide further embodiments. These embodiments can also be combined and interchanged with embodiments described in previous applications.

[0114] The embodiments described herein, in which the possibility is accepted, can be implemented in many different forms. Figures 1, 2, and 3 below provide some embodiments, but other embodiments are also in which the possibility is accepted, and the considerations in Figures 1, 2, and 3 do not limit the breadth of implementation forms. At least one of the embodiments generally relates to encoding and decoding (e.g., video encoding and video decoding, and / or encoding and decoding of at least some parameters of at least some layers of a DNN), and at least one other embodiment generally relates to transmitting a generated or encoded bitstream. These and other embodiments can be implemented as a computer-readable storage medium that stores itself instructions requesting to encode or decode data according to any of the described methods, and / or a computer-readable storage medium that stores a bitstream generated according to any of the described methods.

[0115] In this application, the terms “reconstructed” and “decoded” are interchangeable, the terms “pixel” and “sample” are also interchangeable, and the terms “image,” “picture,” and “frame” are also interchangeable. Typically, though not always, the term “reconstructed” is used on the encoder side, while the term “decoded” is used on the decoder side.

[0116] Modules of the encoder 100 and decoder 200, such as the intra-prediction module, entropic coding module, and / or decoding module (160, 260, 145, 230) shown in Figures 1 and 2, can be modified using various methods and embodiments described in this application. Furthermore, embodiments of this disclosure are not limited to VVC or HEVC, but can be applied to other standards and recommendations, whether existing or to be developed in the future, and to any extensions of such standards and recommendations (including VVC and HEVC).

[0117] Furthermore, this embodiment is not limited to VVC or HEVC, or even video data, but can be applied to encoders or decoders adapted to encode or decode at least one tensor of at least one layer of a neural network that can be used in many art fields other than video (of course, in such embodiments, some modules, such as the intra-prediction module 160, may be optional).

[0118] Unless otherwise specified or technically excluded, the embodiments described in this application can be used individually or in combination.

[0119] This application uses various numerical values ​​(for example, the range of the PDF coefficient, or the increments when changing the PDF coefficient, or the maximum codebook size used for some computing). The specific values ​​are for illustrative purposes only, and the embodiments described are not limited to these specific values.

[0120] Figure 1 shows encoder 100. While variations of encoder 100 are intended, for clarity, encoder 100 will be described below without necessarily explaining all expected variations.

[0121] Before encoding, the sequence may undergo pre-encoding processing (101), such as applying a color conversion to the input color picture in the case of video conversion (e.g., conversion from RGB4:4:4 to YCbCr4:2:0), or remapping the input picture components to obtain a signal distribution more resilient to compression (e.g., using histogram equalization of one of the color components). Pre-encoding processing may also include binarization, as exemplified binarization described in detail above in relation to CABAC.

[0122] Metadata can be associated with preprocessing and added to the bitstream.

[0123] In encoder 100, the picture is encoded by encoder elements as described below. The encoded picture is divided into units, for example, CUs (102), and processed. Each unit is encoded, for example, using either intra-mode or inter-mode. If a unit is encoded in intra-mode, it performs intra-prediction (160). In inter-mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides whether to use intra-mode or inter-mode to encode the unit (105), and indicates the intra / inter decision, for example, by a prediction mode flag. For example, the prediction residual is calculated by subtracting the prediction block from the original image block (110).

[0124] As a result, the predicted residual is transformed (125) and quantized (130).

[0125] Syntax elements such as motion vectors are entropicated along with the quantized transformation coefficients, and the bitstream is output (145).

[0126] In an encoder, the conversion can be skipped, and quantization can be applied directly to the unconverted residual signal. In an encoder, both conversion and quantization can be ignored; that is, the residual is coded directly without applying either the conversion or quantization process.

[0127] The encoder decodes the encoded blocks to provide a reference for further predictions. The quantized transformation coefficients are inversely quantized (140), inversely transformed (150), and the prediction residuals are decoded. For example, in the case of a video sequence, the image blocks are reconstructed by combining the decoded prediction residuals with the prediction blocks (155). For example, an in-loop filter (165) is applied to the reconstructed picture to reduce encoding artifacts by performing unblocking / SAO (Sample Adaptive Offset) filtering. The filtered image is stored in a reference picture buffer (180).

[0128] Figure 2 shows a block diagram of the decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The decoder 200 typically performs a decoding path that is almost the reverse of the encoding path, as shown in Figure 1. The encoder 100 also typically performs decoding as part of encoding the data.

[0129] Specifically, the input to decoder 200 includes a bitstream, which may be generated by encoder 100. The bitstream is first entropi-decoded to obtain encoded information such as conversion coefficients and motion vectors (230).

[0130] In the case of a video bitstream, picture segmentation information indicates how the picture is segmented. This allows the decoder to segment the picture according to the decoded picture segmentation information (235). To decode the prediction residuals, the transformation coefficients are inversely quantized (240) and inversely transformed (250). The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image blocks. Prediction blocks can be obtained from intra-predictions (260) or motion-compensated predictions (i.e., inter-predictions) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0131] Decoded elements (such as picture or layer parameters) may undergo further post-decoded processing (285), for example, in the case of a decoded image, inverse color transformation (e.g., conversion from YCbCr4:2:0 to RGB4:4:4), or inverse remapping, which is the reverse of the remapping process performed in pre-encoding processing (101). Post-decoded processing may use metadata derived in pre-encoding processing and signaled in the bitstream.

[0132] Figure 3 shows a block diagram of an example of a system in which various aspects and embodiments are implemented. System 1000 can be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of System 1000 can be embodied individually or in combination in one integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of System 1000 are distributed across multiple ICs and / or separate components. In various embodiments, System 1000 is communicably connected to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, System 1000 is configured to implement one or more of the aspects described herein.

[0133] System 1000 includes, for example, at least one processor 1010 configured to execute instructions loaded therein to implement various embodiments described herein. Processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile storage device and / or a non-volatile storage device). System 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, such as electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives, and / or optical disk drives. The storage device 1040 may, in non-limiting examples, include internal storage devices, removable storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0134] System 1000 includes, for example, an encoder / decoder module 1030 configured to process data to provide an encoded or decoded data stream (such as a video stream and / or stream representing at least one parameter of at least one tensor of at least one layer of at least one DNN), the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that can be included in a device that performs encoding and / or decoding functions. As is known, the device may include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 1030 may be implemented as a separate element of System 1000 or may be incorporated into the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0135] Program code loaded into the processor 1010 or encoder / decoder 1030 to perform various embodiments described herein can be stored in the storage device 1040 and subsequently loaded into memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 can store one or more of the various items at the time of performing the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or a portion of decoded video, data representing at least one parameter of at least one tensor of at least one layer of at least one DNN, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and operational logic.

[0136] In some embodiments, the internal memory of the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and to provide working memory for processing required during encoding or decoding. However, in other embodiments, memory outside the processing device (for example, the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, high-speed external dynamic volatile memory such as RAM is used as working memory for coding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, and MPEG-2 is also known as ISO / IEC 13818, with 13818-1 also known as H.222 and 13818-2 also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, and is also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding is a new standard being developed by JVET's Joint Video Experts Team).

[0137] As shown in block 1130, inputs can be provided to the elements of system 1000 through various input devices. Such input devices include, but are not limited to, (i) a radio frequency (RF) section for receiving RF signals transmitted over the entire broadcast by a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples include composite video, which is not shown in Figure 3.

[0138] In various embodiments, the input device of block 1130 has a corresponding input processing element as known in the Art. For example, the RF portion can be associated with an element suitable for (i) selecting a desired frequency (also said to be selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band, which in certain embodiments may be called a channel (e.g.), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF portion of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various functions, such as down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, down-converting, and again filtering the RF signal transmitted over a wired (e.g., cable) medium to a desired frequency band. In various embodiments, the order of the above (and other) elements may be rearranged, some of these elements may be removed, and / or other elements performing similar or different functions may be added. Adding elements may include inserting elements between existing elements, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF section includes an antenna.

[0139] Additionally, the USB and / or HDMI terminals may include their respective interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be performed, for example, in a separate input processing IC or within the processor 1010, as needed. Similarly, aspects of USB or HDMI interface processing may be performed, for example, in a separate interface IC or within the processor 1010, as needed. For example, the processor 1010, working in combination with memory and storage elements, and various processing elements, including the encoder / decoder 1030, may provide a demodulated, error-corrected, and demultiplexed stream, and process this data stream as needed for presentation on an output device.

[0140] Various elements of system 1000 can be provided within an integrated housing, where the various elements are interconnected using an internal bus known in the art, such as a suitable connection device 1140, including an Inter-IC (I2C) bus, wiring, and a printed circuit board, and can transmit data between them.

[0141] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may, but is not limited to, include a transceiver configured to transmit and receive data on the communication channel 1060. The communication interface 1050 may, but is not limited to, include a modem or a network card, and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0142] In various embodiments, data is streamed to system 1000 or provided in another manner using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE stands for Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received on a communication channel 1060 and communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to an external network, including the Internet, to enable over-the-top communication such as streaming applications. In other embodiments, streaming data is provided to system 1000 using a set-top box that distributes data over the HDMI connection of input block 1130. Still other embodiments provide streaming data to system 1000 using the RF connection of input block 1130. As shown above, in various embodiments, data is provided in a non-streaming manner. Additionally, in various embodiments, wireless networks other than Wi-Fi, e.g., cellular networks or Bluetooth networks, are used.

[0143] System 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripheral devices 1120. In various embodiments, the display 1100 includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 may be for a television, tablet, laptop, mobile phone, or other device. The display 1100 may also be integrated with other components (e.g., as seen in a smartphone) or separate (e.g., an external monitor for a laptop). Other peripheral devices 1120, in various examples of embodiments, include one or more of a standalone digital video disc (or digital multi-purpose disc) (DVR, abbreviated as both terms), a disc player, a stereo system, and / or a lighting system. In various embodiments, one or more peripheral devices 1120 are used to provide functionality based on the output of System 1000. For example, the disc player performs the function of playing back the output of System 1000.

[0144] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripheral devices 1120 using signaling such as AV. A communication protocol such as Consumer Electronics Control (CEC) is linked, enabling inter-device control with or without user intervention. Output devices may be communicably connected to the system 1000 via dedicated connections through their respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to the system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speaker 1110 may be integrated into a single unit with other components of the system 1000 in an electronic device such as a television. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0145] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 can, alternatively, be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal can be provided via a dedicated output connection, such as an HDMI port, a USB port, or a COMP output.

[0146] The embodiments can be implemented by computer software implemented by processor 1010, by hardware, or by a combination of hardware and software. In a non-limiting example, the embodiments can be implemented by one or more integrated circuits. Memory 1020 can be any type appropriate for the technical environment and, in a non-limiting example, can be implemented using any suitable data storage technology such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and live-bubble memory. Processor 1010 can be any type appropriate for the technical environment and, in a non-limiting example, can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0147] Various implementations involve decoding. As used in this application, “decoding” can encompass all or part of a process performed on a received encoded sequence to produce, for example, a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively, processes performed by the decoders of the various implementations described in this application.

[0148] As further examples, in one embodiment, “decoding” refers only to entropy decoding; in another embodiment, “decoding” refers only to differential decoding; and in yet another embodiment, “decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to an operational subset or to a broader decoding process as a whole will become clear from the context of the specific explanation and will be well understood by those skilled in the art.

[0149] Various implementations involve encoding. Similar to the above considerations regarding "decoding," "encoding" as used in this application can encompass all or part of a process performed on an input video sequence to generate an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as splitting, differential coding, transformation, quantization, and entropy coding. In various embodiments, such a process also, or alternatively, includes processes performed by the encoders of the various implementations described in this application.

[0150] As further examples, in one embodiment, “encoding” refers only to entropy coding; in another embodiment, “encoding” refers only to differential coding; and in yet another embodiment, “encoding” refers to a combination of differential coding and entropy coding. Whether the phrase “encoding process” is intended to refer specifically to a working subset or to a broader encoding process as a whole will become clear from the context of the specific explanation and should be readily apparent to those skilled in the art.

[0151] Please note that the syntax elements used herein are descriptive terms. Therefore, the syntax elements do not preclude the use of other syntax element names.

[0152] When a diagram is presented as a flowchart, it should be understood that the diagram also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that the diagram also provides a flowchart of the corresponding method / process.

[0153] In various embodiments, parametric models or rate distortion optimization are referred to. Specifically, during the coding process, the balance or trade-off between rate and distortion is usually considered to be constrained by computational complexity. This can be measured through the Rate Distortion Optimization (RDO) metric, or through such measurements as Least Mean Square (LMS) or Mean of Absolute Errors (MAE). Rate distortion optimization is usually formulated to minimize a rate distortion function, which is a weighted sum of rate and distortion. There are different methods for solving rate distortion optimization problems. For example, these methods may be based on extensive testing of all coding options, including all possible modes or coding parameter values, and involve a complete evaluation of their coding costs and the associated distortions of the reconstructed signals after coding and decoding. Also, faster methods may be used to reduce coding complexity, in particular, along with the calculation of approximate distortions based on the predicted or predicted residual signal rather than the reconstructed signal. A mixture of these two methods can also be used, for example, by using approximate distortion for only a subset of the possible coding options and full distortion for the others. Other methods evaluate only a subset of the possible coding options. More generally, many methods employ any of the various techniques to perform optimization, but the optimization is not necessarily a full evaluation of both the coding cost and the associated distortion.

[0154] The implementations and embodiments described herein can be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if an implementation of an embodiment is considered only in the context of an implementation of an embodiment (for example, only as a method), the implementation of an embodiment of an embodiment considered can be implemented in other forms (for example, apparatus or programs). Apparatus can be implemented, for example, in appropriate hardware, software, and firmware. Methods can be implemented, for example, in a processor, where a processor is a general processing device including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, mobile phones, and portable / mobile information terminals ("Personal Digital Assistants, PDAs"), which facilitate the communication of information between end users.

[0155] When we refer to "one embodiment" or "a certain embodiment," or "one implementation" or "a certain implementation," or other variations thereof, we mean that certain features, structures, characteristics, etc., described in relation to that embodiment are included in at least one embodiment. Therefore, when the phrases "in one embodiment" or "in a certain embodiment," or "in one implementation" or "in one implementation," as well as any other variations, appear in various places throughout this specification, they do not necessarily all refer to the same embodiment.

[0156] In addition, this application may refer to "determining" various types of information. Determining information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or retrieving information from memory.

[0157] Furthermore, this application may refer to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0158] Additionally, this application may refer to "receiving" various types of information. Receiving is intended to be a broad term, similar to "accessing." Receiving information may include, for example, one or more of the following: accessing information, or retrieving information (e.g., from memory). Furthermore, "receiving" is usually involved in some way when performing tasks such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0159] For example, in the cases of "A / B", "A and / or B)", and "at least one of A and B", please understand that the use of any of the following " / ", "and / or", and "at least one of" is intended to encompass the selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As further examples, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such expressions are intended to encompass the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to the number of items described, as will be obvious to those skilled in the art in the relevant and related fields.

[0160] Furthermore, as used herein, the term "signaling" means indicating something to a corresponding decoder, among many others. For example, in certain embodiments, an encoder signals at least one of several transformations, encoding modes, or encoding flags. Thus, in certain embodiments, the same parameter is used on both the encoder and decoder sides. Therefore, for example, an encoder can transmit a specific parameter to a decoder (explicit signaling) so that the decoder can use the same specific parameter. Conversely, if the decoder already has other parameters along with that specific parameter, signaling can be used without transmission (implicit signaling) to allow the decoder to know and select that specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual function. It should be understood that signaling can be achieved in various ways. For example, in various embodiments, information is signaled to the decoder using one or more syntax elements, flags, etc. The above concerns the verb form of the word "signal," but the word "signal" may also be used as a noun in this specification.

[0161] As will be apparent to those skilled in the art, various implementations can generate a variety of signals formatted to carry information that can be stored or transmitted. This information may include, for example, instructions prompting a method to perform a specific action, or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of one of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored in a processor-readable medium.

[0162] Several embodiments are described below. Features of these embodiments can be provided individually or in any combination across various claims and types. Furthermore, embodiments can include, individually or in any combination, one or more of the following features, devices, or aspects across various claims and types. A process or device that performs encoding and decoding using deep neural network compression of a pre-trained deep neural network. A process or device for performing deep neural network compression by encoding and decoding at least one layer of a pre-trained deep neural network. A process or device that performs deep neural network compression of a pre-trained deep neural network having one or more layers by encoding and decoding information inserted into a bitstream representing parameters. A process or device that performs deep neural network compression of a deep neural network by encoding and decoding information inserted into a bitstream representing parameters. A bitstream or signal containing one or more of the described syntax elements, or variations thereof. A bitstream or signal containing a syntax that carries information generated in accordance with any of the embodiments described. • Create and / or transmit and / or receive and / or decode by any of the embodiments described. • A method, process, apparatus, medium for storing instructions, medium for storing data, or signal, according to any of the embodiments described. - Insert syntax elements into the signaling that allow the decoder to determine the encoding mode in a manner corresponding to the method used by the encoder. • Create and / or transmit and / or receive and / or decode a bitstream or signal containing one or more of the described syntax elements or their variations. An electronic device such as a television, set-top box, mobile phone, or tablet that performs a conversion method according to any of the embodiments described. An electronic device such as a television, set-top box, mobile phone, or tablet that determines a conversion method according to any of the embodiments described and displays the resulting image (for example, using a display of a type such as a monitor or screen). A television, set-top box, mobile phone, tablet, or other electronic device that selects, band-limits, or tunes (e.g., using a tuner) a channel to receive a signal containing an encoded image, and performs the conversion method according to any of the embodiments described. A TV, set-top box, mobile phone, tablet, or other electronic device that receives and converts signals containing encoded images throughout the entire broadcast.

Claims

1. A method comprising encoding at least one tensor associated with at least one layer of a deep neural network in a bitstream, Encoding the aforementioned at least one tensor is Obtaining the size of a codebook for quantizing the parameters of the at least one tensor, wherein the size is obtained according to a distortion value determined between the at least one tensor and the quantized version of the at least one tensor. A method comprising quantizing the parameters using a codebook having the obtained size.

2. The method according to claim 1, wherein the size of the codebook is obtained from a binary search over the entire range of codebook sizes.

3. The method according to claims 1 and 2, wherein the quantization uses initialization based on a pdf bounded according to at least one first pdf coefficient.

4. The method according to claim 3, wherein the first pdf coefficient is selected from among several candidate boundary pdf coefficients based on the entropy obtained for the quantized parameter from each of several candidate boundary pdf coefficients.

5. The method according to any one of claims 1 to 4, further comprising encoding information representing the type of the codebook.

6. A device for encoding at least one tensor associated with at least one layer of a deep neural network in a bitstream, wherein the device comprises at least one processor, and the at least one processor is - Obtaining the size of the codebook for quantizing the parameters of the at least one tensor, wherein the size is obtained according to the strain value determined between the at least one tensor and the quantized version of the at least one tensor. - A device configured to quantize the parameters using a codebook having the acquired size.

7. The apparatus according to claim 6, wherein the size of the codebook is obtained from a binary search over the entire codebook size range.

8. The apparatus according to claims 6 and 7, wherein the quantization uses initialization based on a pdf bounded according to at least one first pdf coefficient.

9. The apparatus according to claim 8, wherein the first pdf coefficient is selected from among several candidate boundary pdf coefficients based on the entropy obtained for the quantized parameter from each of several candidate boundary pdf coefficients.

10. The apparatus according to any one of claims 6 to 9, further comprising encoding information representing the type of the codebook.

11. A signal including a bitstream, which includes data representing at least one first tensor of at least one layer of the deep neural network encoded according to the method of any one of claims 1 to 5.

12. The signal according to claim 11, wherein the data includes information representing the type of the codebook.

13. A computer-readable medium comprising the signal described in claim 11 or 12.

14. A method comprising decoding at least one tensor associated with at least one layer of a deep neural network from a bitstream, wherein decoding the at least one tensor is - Decoding information representing the type of codebook from the bitstream, A method comprising: - Inverse quantizing the parameters of the at least one tensor using the codebook.

15. An apparatus comprising at least one processor configured to decode at least one tensor associated with at least one layer of a deep neural network from a bitstream, wherein decoding the at least one tensor is - Decoding information representing the type of codebook from the bitstream, - An apparatus comprising dequantizing the parameters of the at least one tensor using the codebook.

16. A computer-readable storage medium storing instructions for causing one or more processors to perform the method described in any one of claims 1 to 5 or 14.

17. A computer program product that, when executed by one or more processors, includes instructions causing one or more processors to perform the method of any one of claims 1 to 5 or 14.

Images