Systems and methods for encoding / decoding deep neural networks

Abstract

The present disclosure relates to a method, in response to determining that at least one first tensor of at least one layer of at least one deep neural network is decomposed into a second tensor and a third tensor whose parameters are encoded in a bitstream, decoding a size of at least one of the second tensor and the third tensor from the bitstream, and decoding at least one of the second tensor and the third tensor from the bitstream based on the decoded size. Corresponding apparatuses, encoding methods, signals, bitstreams, storage media, and encoder and / or decoder devices are also provided.

Description

[Technical Field]

[0001] A domain technical field of one or more embodiments of the present disclosure relates to a data processing technology domain, such as data compression and / or data decompression. For example, at least some embodiments relate to data compression / decompression involving large amounts of data, such as compression and / or decompression of at least a portion of an audio and / or video stream, or compression and / or decompression of data associated with deep learning techniques, such as at least some parameters of a deep neural network (DNN).

[0002] At least some embodiments relate to improving compression efficiency over existing video compression systems such as HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2, as described in "ITU-T H.265 Telecommunication standardization sector of ITU (10 / 2014), series H: audiovisual and multimedia systems, infrastructure of audiovisual services—coding of moving video, High efficiency video coding, Recommendation ITU-T H.265"), or over evolving video compression systems such as VVC (Versatile Video Coding, a new standard being developed by the JVET Joint Video Experts Team).

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction, including spatial vector prediction and / or motion vector prediction, and transformation to exploit spatiotemporal redundancy in video content. Intra- or inter-prediction is generally used to exploit correlation within or between frames, whereby the difference between an original image and a predicted image, often referred to as a prediction error or prediction residual, is transformed, quantized, and entropy coded. To recover the video, the compressed data is decoded by an inverse process corresponding to entropy coding, quantization, transformation, and prediction.

[0004] At least some embodiments relate to improving compression efficiency compared to existing systems for compression of deep neural networks (DNNs), such as some compression standards or draft standards, such as ISO / MPEG7, the upcoming neural network standard for multimedia content description and analysis currently being developed by the International Organization for Standardization.

[0005] Typically, the encoding process involves quantizing and entropy coding the parameters of the DNN to obtain compressed data. To restore the data, the compressed data is decoded, and the decoding process includes entropy decoding and dequantization. Summary of the Invention

[0006] The principles of the present invention make it possible to overcome at least one of the drawbacks of some known compression and / or decompression methods by proposing a method and apparatus for encoding or decoding data in at least one bitstream, which are one or more parameters of at least one tensor of at least one layer or sublayer of at least one deep neural network. It is noted that the parameter tensors associated with a layer may include weights and / or biases, although for brevity they may be referred to simply as "weights" hereinafter.

[0007] According to an embodiment, there is provided a method for decoding at least one first tensor of at least one layer of at least one deep neural network, the method including, in response to determining that the at least one first tensor is decomposed into a second tensor and a third tensor whose parameters are encoded in the bitstream, decoding a size of at least one of the second tensor and the third tensor from the bitstream, and decoding at least one of the second tensor and the third tensor from the bitstream based on the decoded size.

[0008] Another embodiment provides an apparatus for decoding at least one first tensor of at least one layer of at least one deep neural network, comprising one or more processors configured to determine that at least one first tensor of at least one layer of the at least one deep neural network is decomposed into a second tensor and a third tensor whose parameters are encoded in a bitstream, decode a size of at least one of the second tensor and the third tensor from the bitstream, and decode at least one of the second tensor and the third tensor from the bitstream based on the decoded size.

[0009] Another embodiment provides a method that includes encoding data representing at least one first tensor of at least one layer of a deep neural network in a bitstream, the method including, in response to determining that the at least one first tensor is decomposed into a second tensor and a third tensor, encoding a size of at least one of the second tensor and the third tensor and encoding a parameter representing at least one of the second tensor and the third tensor.

[0010] According to yet another aspect, an apparatus is provided for encoding data representing at least one first tensor of at least one layer of a deep neural network in a bitstream, the apparatus comprising one or more processors, the one or more processors configured to determine that the at least one first tensor is decomposed into a second tensor and a third tensor, and in response to the determination, encode a size of at least one of the second tensor and the third tensor, and encode at least one of the second tensor and the third tensor.

[0011] One or more embodiments also provide a computer program product including instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding or decoding method according to any of the previously described embodiments. One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding data according to the previously described methods. One or more embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the previously described methods. One or more embodiments also provide methods and apparatus for transmitting or receiving a bitstream generated according to the methods described above.

[0012] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any one of the decoding embodiments and at least one of (i) an antenna configured to receive a signal including input data, (ii) a band limiter configured to limit the received signal including the input data to a frequency band, or (iii) a display configured to display an output representing the video block.

[0013] Although not explicitly described, the devices of the present disclosure may be adapted to perform the methods of the present disclosure in any of their embodiments.

[0014] Although not explicitly described, the present embodiments relating to methods or corresponding signals, devices, and computer-readable storage media may be employed in any combination or subcombination. [Brief explanation of the drawings]

[0015] [Figure 1] Here are some common standard encoding methods: [Figure 2] A common standard decoding method is shown below. [Figure 3] 1 illustrates an exemplary processor arrangement in which the described embodiments may be implemented. [Figure 4] 1 illustrates a DNN-wide encoding architecture that employs at least some embodiments of the encoding method of the present disclosure. [Figure 5] 1 illustrates a DNN-wide decoding architecture that employs at least some embodiments of the encoding method of the present disclosure. [Figure 6] 1 illustrates an example of a method for decoding tensors of a DNN encoded in a bitstream, according to an embodiment of the present disclosure. [Figure 7] 1 illustrates an example of a method for encoding tensors of a DNN in a bitstream, according to an embodiment of the present disclosure. [Figure 8] 1 illustrates an example of a portion of a bitstream containing data representing a first tensor of at least one layer of a deep neural network, according to an embodiment.

[0016] It should be noted that the drawings illustrate example embodiments, and that the embodiments of the present disclosure are not limited to the illustrated embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0017] Many technical fields may involve the processing of large amounts of data by computer means. Such processing may involve data compression and / or data decompression, for example for storing or transforming at least some of such data. Examples of compression and / or decompression of streams containing large amounts of data can be found in the video processing technical field or in technical fields involving deep learning techniques.

[0018] For illustrative purposes and not for limitation, the following detailed description of embodiments of the present disclosure will be directed to a deep neural network (DNN). However, it is clear that the present disclosure can also be applied to other large-volume data compression / decompression fields, such as those in the video processing technology field. For example, the present disclosure can be applied to the compression / decompression of tensors obtained from at least one image by a deep learning algorithm.

[0019] Deep neural networks (DNNs) represent the state of the art in performance across diverse domains such as multimedia processing, computer vision, speech recognition, natural language processing, etc. However, this performance can be computationally expensive, as DNNs tend to have a large number of parameters, often in the millions, and sometimes even billions.

[0020] This can lead to, for example, prohibitively high inference complexity. Simply put, inference is the deployment of a DNN once it has been trained to process input data, e.g., in terms of its classification. Inference complexity is sometimes defined as the computational cost of applying a trained DNN to the input data being inferred. Inference complexity is sometimes defined as the computational cost of applying a trained DNN to the test data being inferred.

[0021] This high inference complexity can therefore pose a significant challenge in using DNNs in environments involving electronic devices with limited hardware and / or software resources, for example, in environments involving mobile or embedded devices with resource constraints such as battery size, limited computing power, and memory capacity.

[0022] A deep neural network is composed of several layers. Each layer is associated with a set of parameters, which can be obtained, for example, during training of the DNN. These parameters (such as weights and / or biases) are stored as a multidimensional array (also referred to herein as a "tensor"). Hereinafter, for simplicity's sake, the term "matrix" may be used to refer to a set of parameters (e.g., parameters of a given layer). However, it should be understood that some embodiments of the methods disclosed herein can also be applied to parameters of tensors with dimensions greater than two, such as a 2D convolutional layer, which typically includes parameters of a 4D tensor. The large number of parameters in a DNN may require a large bandwidth for the deployment of the DNN (or a solution including a DNN) in a distributed environment.

[0023] At least some embodiments of the present disclosure apply to compressing and / or decompressing (decoding) at least some parameters of at least one DNN (e.g., a pre-trained DNN). In practice, the compression can facilitate transmission and / or storage of the parameters of the at least one DNN. More precisely, at least some embodiments of the present disclosure apply to compressing parameters of at least one tensor associated with at least one layer of at least one deep neural network.

[0024] Depending on the embodiment of the present disclosure, the layers (or sublayers) can be of different types. For example, in some embodiments, all of the at least one layer can be convolutional layers or full connector layers, or the at least one layer can be composed of at least one convolutional layer and / or at least one full connector layer.

[0025] Some embodiments of the present disclosure more specifically relate to compression solutions that include or may at least include decompression of at least one tensor, for example, to improve compression efficiency, and / or decoding solutions that include or may at least include restoration of at least one tensor. The at least one decomposed tensor may be, for example, at least one tensor of one or more layers to be compressed of one or more DNNs. The at least one restored tensor may be, for example, at least one tensor of the same form as the at least one decomposed tensor of one or more layers to be compressed of one or more DNNs.

[0026] According to some embodiments of the present disclosure, the decomposition of the tensor can be obtained using Low Rank (LR) and / or Low Displacement Rank (LDR) techniques.

[0027] Tensor decomposition is used to compress a large tensor weight, producing at least two smaller tensors that can be further compressed, quantized, entropy coded, and stored or transmitted in a bitstream.

[0028] As non-limiting examples, some embodiments may be applicable in non-standardized technologies, while others may be used in the context of standards for DNN compression / decompression, such as the upcoming ISO / MPEG7 standard relating to the compressed representation of neural networks in multimedia content description and analysis, hereinafter more simply referred to as MPEG NNR.

[0029] At least some embodiments of the present disclosure also propose a syntax mechanism along with a syntax structure for decomposing a tensor and recovering a tensor from multiple decoded tensors.

[0030] More precisely, in some embodiments, tensor decomposition can be used to compress large tensor weights to produce at least two smaller tensors (which can also be input to quantization, with at least some of the output of the quantization being, for example, encoding).

[0031] For example, in an exemplary embodiment where the decomposition is based on lower-order (LR), the lower-order approximation can be expressed as a product of the original matrix of weights.

[0032]

number

[0033] At the decoder, when using this compression technique, two options are possible: Any device / implementation of the inference of the decoded model supports such a decomposition, in which case the tensors G and H can be output as is. Or the original graph in its original tensor form is required by the inference engine, in which case reconstruction must be performed (e.g., by a decoder).

[0034] At least some embodiments of the present disclosure provide a syntax to allow for conditions as well as mechanisms for restoring a tensor in its original form. In fact, the inventors have wisely realized that no solution has yet been proposed for restoring the original form of a tensor from the tensors obtained by decomposition of the tensor, e.g., at the decoder at the time of model restoration.

[0035] For example, in the current specification draft of MPEG NNR, the current assumption is that the tensors output by the decoder result from decomposing the original tensors.

[0036] This can be a problem when the original tensor form is needed by an inference engine such as the one described above.

[0037] At least some embodiments of the disclosed invention help address this problem. It is worth noting that embodiments of the disclosed methods described herein can be implemented in many compression solutions and are not limited to any particular standard, even though at least some of the embodiments may be applied in the context of several compression standards, such as, for example, several draft standards being developed by ISO / MPEG7.

[0038] As can be seen in equation (1) above, in an exemplary embodiment where the tensor W is a 2D matrix, G and H T After decoding, we need to perform a matrix multiplication of the two matrices to get the original matrix form.

[0039] In the exemplary embodiments described above, LR decomposition can be used on the original tensors. However, the decomposition performed may vary depending on the embodiment or on the tensors of one or more DNNs. For example, for tensors in convolutional layers or depthwise convolutional layers, the tensors can be restructured into two-dimensional matrices, which allows for LR / LDR methods.

[0040] This disclosure describes the syntax and processes necessary to allow the tensors to be recovered.

[0041] In particular, for example, in some embodiments of the present disclosure adapted to reconstruct an original tensor from one or more tensor units, a mechanism is proposed to perform the reconstruction, involving a tensor buffer that holds pre-reconstructed tensors, such as the pre-decoded G and / or H matrices (in the exemplary use case introduced above in connection with equation (1)) to perform the reconstruction of W.

[0042] In the following, we introduce a decoded tensor buffer (DTB) that can contain multiple already decoded tensors in memory. For example, in the previous exemplary embodiment, decoded tensors G and H are added to the buffer, and they are the first of two tensors (G and H) to be decoded for a given layer. More precisely, for a given layer, when tensor H (respectively tensor G) has not yet been decoded, decoded tensor G (respectively decoded tensor H) is added to the buffer.

[0043] This allows the decryption of the corresponding tensor in the same layer to trigger the restoration of the tensor with its original shape, freeing up the memory occupied by the tensor stored in the DTB.

[0044] In at least some embodiments of the present disclosure, when an original tensor is decomposed into several tensors, some of the resulting tensors can be coded and decoded separately (in other words, independently of the others).

[0045] 4 and 5 illustrate, at a high level, general processes for encoding / decoding parameters of at least one tensor in at least one layer of at least one DNN that can be used in at least some embodiments of the present disclosure. The method of FIG. 4 can be performed, for example, in an encoding device (i.e., an encoder), and the method of FIG. 5 can be performed, for example, in a decoding device (i.e., a decoder).

[0046] 4, in an encoder, the method can include obtaining 401 (or acquiring) parameters of tensors associated with a layer to be compressed (also referred to herein as "original tensors"), for example, by retrieving the parameters of at least one tensor from a storage unit or by receiving the parameters from a data source via a communications interface.

[0047] In some embodiments, each resulting tensor can be decomposed.

[0048] In other embodiments, the decomposition may be performed conditionally, as in the embodiment of FIG. 4 . In fact, as one example, the decomposition may not be applicable. Tensor decomposition cannot be performed on biases that are 1D arrays, for example. In some embodiments, other factors (such as the encoding cost of the original tensor) may also be considered to determine whether the decomposition is applied to a tensor. For example, a mode may be associated with a tensor and / or with a layer of the tensor or with one or more layers that comprise the layer of the tensor (403). At least one first value of the mode may represent the decomposition to be performed on the tensor, if applicable, and / or at least one second value of the mode may represent the tensor to be processed without the decomposition being applied.

[0049] In the example embodiment of FIG. 4, the method may include testing whether a decomposition is applicable to an input tensor (402), and if the tensor can be decomposed (402), testing whether a decomposition mode (e.g., a first value of the mode) is selected (403).

[0050] If a decomposition mode is selected by the encoder (404), the method may include decomposing the tensor and encoding the resulting tensor (e.g., encoding tensor G (405) and encoding tensor H (407)).

[0051] In the example use case of FIG. 4, if decomposition is not chosen (no decomposition mode is selected) or is not applicable, the input tensor can be directly encoded (406).

[0052] The output of the encoding is used to construct a bitstream.

[0053] This process can be repeated 408 for several input tensors, for example, for all tensors in the model to be quantized and / or encoded.

[0054] In some embodiments, the method may further include reducing the number of parameters (i.e., weights or biases) of the neural network prior to encoding by exploiting redundancy inherent in the neural network. For example, the original tensors of the parameters of at least one layer of the DNN or tensors resulting from the decomposition of the original tensors of the parameters of at least one layer of the DNN may be sparse. This reduction is optional, and in some embodiments, and / or for some tensors in some layers, this reduction may be omitted.

[0055] The encoding may include quantizing parameters (such as weights and biases) of at least one tensor (e.g., a tensor output by decomposition of a tensor in a layer of a neural network, or the tensor itself if no decomposition occurs), and representing the quantized information using a smaller number of bits by lossless entropy coding of the quantized information.

[0056] In some embodiments, when several layers of a DNN are to be coded, this method can be repeated for each layer until the end of coding the parameters of the last layer to be coded.

[0057] In some embodiments, this tensor can be coded in parallel or interleaved between other tensors in other layers, rather than having to code tensors in the same layer sequentially, as in the embodiment shown in Figure 4. In practice, the coding may be based on tensor-level units, where weights and biases of the same layer are contained in the same or different tensor units (e.g., as in the upcoming MPEG NRR draft).

[0058] FIG. 5 illustrates the corresponding processing performed on the decoding side for a tensor decoded from a bitstream, e.g., obtained by the encoding method already described with reference to FIG. 4. In the exemplary embodiment of FIG. 5, the tensor is first parsed and identified (501), e.g., using its unit header and / or layer parameter set. For example, in the exemplary syntax presented in more detail below, the associated high-level syntax referenced by _lps_layer_parameter_set_id in the unit header can be used to point to the correct layer parameter set. According to FIG. 5, the tensor payload can be decoded (502). In the exemplary embodiment of FIG. 5 where the decomposition is of LR or LDR type, if the decoded tensor is of type TENSOR_G or TENSOR_H (503) (denoted as G and H, respectively, in the figure), steps 505-508 can be performed (see below). Otherwise, if it is not the last decoded tensor (509), the next tensor can be accessed from the bitstream. If the current tensor is of type TENSOR_G or TENSOR_H, the corresponding tensor belonging to the same layer is looked up in the decoded tensor buffer. This can be done by looking for a tensor associated with an identifier (such as a reference identifier "ref_id") that specifies the same layer as the current tensor. In the example syntax provided below, such a reference identifier (e.g., "ref_id") may be a syntax element from a tensor unit header that maps a tensor unit to a uniquely identifiable data structure defined by a topology storage format, e.g., ONNX or NNEF.

[0059] If there is a corresponding tensor, it is fetched (505) from the DTB, and both the current tensor and the fetched (507) tensor are used to reconstruct (508) the tensor in the form of the original tensor (i.e., the tensor obtained in step 401). It is worth noting that in many embodiments, the reconstructed tensor, while having the same dimensions as the original tensor, will be different from the original tensor.

[0060] If the corresponding tensor is not in the DTB, the method 500 may include storing the current tensor in the DTB for subsequent use (506).

[0061] In either case (in other words, after decoding the current tensor unit) (step 509), the method may further include checking 509 whether the current tensor is the last in the bitstream, and if so, outputting the model; otherwise, accessing the next tensor unit.

[0062] At the decoder, decoding may involve some inverse operations (relative to the encoder-side operations), as shown in Figure 5. For example, the decoding method may involve parsing / entropy decoding 510 of the input bins to extract the metadata and / or quantized forms of the parameters, so that inverse quantization 520 can be applied to derive the final values ​​of the tensor's parameters.

[0063] When several tensors (e.g., of several layers) are to be decoded, method 500 can be performed until all of the several tensors have been decoded.

[0064] Some embodiments of the present disclosure may include transmitting / receiving signaling information between an encoder and a decoder. This signaling information is presented in this disclosure in conjunction with an exemplary, non-limiting syntax. For ease of explanation, this exemplary syntax is primarily based on the syntax used in the exemplary MPEG NNR draft standard (N19225 - Working Draft 4 of Compression of neural networks for multimedia content description and analysis, International Organization for Standardization ISO / IEC JTC1 / SC29 / WG11, April 2020).

[0065] The following syntax is merely an example syntax that does not limit the present disclosure. For example, the number of bits used for syntax elements is an example embodiment. For ease of understanding, in the example syntax, the following identifiers and clauses according to embodiments of the present disclosure are numbered with section and table numbers that are consistent with the current example working draft of MPEG-NNR.

[0066] In this example syntax, the following tensor operations can be added: Returns the matrix product of array_name_1 and array_name_2: MatrixProd(array_name_1,array_name_2[]).

[0067] TensorReshape(array_name[],tensor_dimension[]) returns the tensor array_name[] reshaped with the specified tensor_dimension[] without changing the data.

[0068] We also provide definitions for the following terms:

[0069] Bin: A string of 1-bit bins.

[0070] Binarization: A set of bin sequences for all possible values ​​of a syntax element.

[0071] Binarization process: A process that uniquely maps all possible values ​​of a syntax element to a set of bin sequences.

[0072] Bin sequence: An intermediate binary representation of the value of a syntax element from the binarization of the syntax element.

[0073] Bitstream: A sequence of bits that constitutes a representation of coded units and associated data that form one or more coded neural network models. Decoded Tensor Buffer (DTB): A buffer that holds the decoded tensors / units as reference.

[0074] When decoding at least a portion of the bitstream (referred to simply as the "decoding process"), the following conditions may apply, using an exemplary detailed syntax: Information needed to decode an NNR unit of an NNR bitstream can be signaled as part of the NNR bitstream. If such information is not part of the NNR bitstream, it can be provided to the decoding process by other means (e.g., out-of-band topology information or parameters that are needed for decoding but are not signaled or carried in the NNR bitstream). The decoding process may start with an NNR unit of type NNR_STR (see table below). Reception of an NNR_STR unit allows the decoder to reset its internal state and prepare to receive an NNR bitstream. The presence and cardinality of preceding NNR units may be specified in several subclauses and / or annexes. The buffer DTB is set to empty (DTB fullness is set equal to 0) at the start of the decoding process.

[0075] In the exemplary syntax detailed herein, the following table refers to the different unit types: NNR_STR, as mentioned above, specifies the starting unit of an NNR bitstream.

[0076] [Table 1]

[0077] According to some embodiments of the present disclosure, we propose to specify the NNR tensor type in the case of tensor decomposition. For example, in conjunction with the exemplary MPEG NNR draft standard, the following exemplary syntax can be used:

[0078] 6.2 NNR Decomposition Identifier Using the exemplary syntax detailed herein, the table can specify the NNR tensor type for tensor decomposition.

[0079] [Table 2]

[0080] Tensor Output On the decoder side, after decoding the unit header (e.g., nnr_compressed_data_unit_header with exemplary syntax) and compressed payload, tensor processing can be performed once per NNR compressed payload.

[0081] In the detailed exemplary embodiment, the output of the processing of the tensor can then be specified as follows: -If output_original_graph is equal to 0, or lps_tensor_decomposition_flag is equal to 0, or nnr_decomposition_tensor_type is equal to "TENSOR_OTHER", the tensor at that time is output. Otherwise, if there is no tensor with an identifier (e.g., ref_id) that specifies the same layer in the DTB, add the current tensor to the DTB. No tensor is output. - Otherwise (the current nnr_decomposition_tensor_type specifies a tensor of type "TENSOR_G" or "TENSOR_H" and there is a tensor with an identifier (e.g., ref_id) specifying the same layer in the DTB), call decompress tensor in its original shape as specified above, passing both the current tensor of type "TENSOR_G" or "TENSOR_H", respectively, and its corresponding tensor in the DTB of type "TENSOR_H" or "TENSOR_G", respectively. The latter is removed from the DTB. The restored tensor is output.

[0082] Recovering a tensor in its "original" form After decoding all of the tensors resulting from the decomposition of the original tensor, a reconstruction of the tensor having the shape of the original tensor can be achieved.

[0083] For example, in an embodiment where the original tensor is decomposed into tensor G and tensor H, as described above, if there is a corresponding tensor in the DTB (e.g., a tensor with an identifier (e.g., ref_id) specifying the same layer), then recovery can be performed after decoding the tensor (e.g., a tensor of type "TENSOR_G" or "TENSOR_H").

[0084] In the exemplary syntax described herein, the inputs to this reconstruction may include: - tensor_h[] of type 'TENSOR_H' - an array tensor_dimensions_h[] corresponding to the dimensions of tensor_h[] as defined by its decoded syntax tensor_dimensions from the corresponding nnr_compressed_data_unit_header - a tensor tensor_g[] of type 'TENSOR_G'. - an array tensor_dimensions_g[] corresponding to the dimensions of tensor_g[] defined by the decoded syntax tensor_dimensions - The values ​​tensor_reconstruction_mode and tensor_reconstruction_additional_info from the layer parameter set of the tensor header that was just decoded.

[0085] In the described exemplary embodiment, the output of this reconstruction is a tensor array_w now having the same shape as the original tensor (also referred to herein as the original shape). According to some embodiments of the present disclosure, the tensor array_w now can be computed by considering the reconstruction mode of the tensor. In the exemplary syntax detailed herein, the following table can be used to specify the reconstruction mode of a tensor in the bitstream:

[0086] [Table 3]

[0087] The tensor array_w can then be computed as follows:

[0088] [Table 4]

[0089] Regarding the high level syntax, in the exemplary draft standard MPEG-NNR, some elements may be added to some tables of the exemplary draft standard as follows:

[0090] [Table 5] where: For the compression unit in use, nnr_layer_parameter_set_id specifies, for example, the value of lps_layer_parameter_set_id. The value of unit_layer_parameter_set_id may, for example, be in the range 0 to 63 inclusive. For tensor decomposition, nnr_decomposition_tensor_type specifies the tensor type, e.g., as defined above.

[0091] [Table 6] where: The mps_model_parameter_set_id provides an identifier to the MPS for reference by other syntax elements. The value of mps_model_parameter_set_id may be set in the inclusive range of 0 to 15. decomposition_flag equal to 1 specifies that tensor decomposition has been applied to at least one tensor in at least one layer of the model. output_original_graph equal to 1 specifies that when tensor decomposition is used, the decoder will output tensor weights in their original shape. mps_max_dec_tensor_buffering_minus1+1 specifies the maximum required size of the decoded tensor buffer for the NNR model, in tensor storage buffer units. The value of mps_max_dec_tensor_buffering_minus1 can be set in the range 0 to 63.

[0092] [Table 7] where: lps_model_parameter_set_id specifies the mps_model_parameter_set_id of the active LPS. The value of lps_model_parameter_set_id can be set in the inclusive range of 0 to 15. lps_layer_parameter_set_id provides an identifier for the LPS for reference by other syntax elements. The value of lps_layer_parameter_set_id can be set in the range 0 to 63 inclusive. lps_tensor_decomposition_flag equal to 1 specifies that tensor decomposition is used for this layer. tensor_reconstruction_mode specifies the mode used to reconstruct a tensor in its original shape from a decomposed and decoded tensor, as defined above. tensor_reconstruction_additional_info_counts specifies the number of parameters that may be needed to reconstruct a decomposed tensor. tensor_reconstruction_additional_info[i] specifies the parameter array that may be needed to reconstruct the decomposed tensor (e.g., for a depthwise convolutional layer, tensor_reconstruction_additional_info_counts can be set to 1, and tensor_reconstruction__additional_info[0] specifies the convolution kernel size).

[0093] Variations: Several exemplary embodiments have been detailed above. The present disclosure also encompasses many variations of the above-described embodiments. For example, some embodiments of the present disclosure may relate to the following variations:

[0094] First variant: version without output_original_graph A first variant allows the variable output_original_graph (introduced above) to be omitted. In this variant, the reconstruction depends on the topology_storage_format variable. Thereby, for the exemplary NNR table introduced above, the table "NNR Model Parameter Set Payload Syntax" can be modified (since the variable output_original_graph is not needed).

[0095] In conjunction with the MPEG NNR standard and in the exemplary syntax detailed herein, this leads to the following table:

[0096] [Table 8]

[0097] For the restoration method the following applies: Tensor Output This process can be called once per NNR compressed payload after decoding the unit header nnr_compressed_data_unit_header and the compressed payload.

[0098] The output of the tensor can then be specified as follows: -If lps_tensor_decomposition_flag is equal to 0, or nnr_decomposition_tensor_type is equal to "TENSOR_OTHER", or topology_storage_format specifies a topology corresponding to inference using a decomposition matrix, the tensor at that time is output. - Otherwise, if there is no tensor with the same "ref_id" in the DTB, add the current tensor to the DTB with that "ref_id". No tensor is output. - Otherwise (the current nnr_decomposition_tensor_type specifies a tensor of type "TENSOR_G" or "TENSOR_H" and there is a tensor with the same "ref_id" in the DTB), call tensor decomposition in its original shape as specified above, passing the current tensor of type "TENSOR_G" or "TENSOR_H", respectively, and also its corresponding tensor in the DTB of type "TENSOR_H" or "TENSOR_G", respectively. The latter is removed from the DTB. The restored tensor is output.

[0099] Using the example syntax detailed herein, the topology_storage_format variable may be defined as shown below (eg, in section 8.3.2.3.4):

[0100] 8.3.2.3.4 NNR Topology Unit Header Semantics topology_storage_format specifies the format of the stored neural network topology information, as specified below.

[0101] [Table 9]

[0102] Second variant: Decomposition implementation map In a second variant, information about the performance of the decomposition process can be provided in the signaling, for example, such information can represent a mapping between different mean squared error (MSE) values ​​between the decomposed tensors and their original versions and the resulting neural network (NN) inference accuracy.

[0103] The resulting accuracies can be provided separately for different aspects or characteristics of the NN's output. In the case of a classifier NN, each MSE value (e.g., threshold) can be mapped to a separate accuracy for each class in addition to an overall accuracy that considers all classes. In some embodiments, the classes can be ordered based on the neural network output order, i.e., the order specified during training.

[0104] In conjunction with the MPEG NNR standard, the following table may be used in the exemplary syntax detailed herein:

[0105] [Table 10] Here, Decomposition_performance_map() can be defined, for example, as follows:

[0106] [Table 11] where: decomposition_performance_map() specifies the mapping between different mean squared error (MSE) thresholds between a decomposed tensor and its original version and the resulting NN inference accuracy. The resulting accuracies are given separately for different aspects or characteristics of the NN's output. For classifier NNs, each MSE threshold is mapped to a separate accuracy per class in addition to an overall accuracy that considers all classes. The classes are ordered based on the neural network output order, i.e., the order specified during training. ·count_thresholds specifies the number of decomposition MSE thresholds. Decomposition_threshold specifies the MSE threshold array that is applied to derive the rank of the different weight tensors. nn_accuracy specifies the overall accuracy of the NN (e.g., classification accuracy by considering all classes). ·nn_reduction_ratio[i] specifies the ratio between the total number of parameters after tensor decomposition of the entire model and the number of parameters in the original model. ·count_classes specifies the number of classes that will be given different accuracies for each decomposition threshold. · nn_class_accuracy specifies the accuracy array for a particular class when a particular decomposition threshold is applied.

[0107] Third variant: Example: Restoration in case of units of type NNR_PT_BLOCK For ease of explanation, this third variant will be detailed using an exemplary MPEG NNR draft standard (N19225-Working Draft 4 of Compression of neural networks for multimedia content description and analysis, as per the International Organization for Standardization ISO / IEC JTC1 / SC29 / WG11, April 2020). In such an exemplary syntax, there may be multiple parameters within a unit, such as weight tensor parameters, bias parameters, batch norm parameters, etc., which may be optionally decomposed. In terms of the NNR_PT_BLOCK unit, the output tensor of the weights must conform to the original shape, which may be transmitted, for example, through external topology information or known to the decoder. If tensor decomposition is applied in the encoder, the reconstruction of the tensor is performed in the decoder.

[0108] This variant proposes to simplify the reconstruction process and the required syntax for specifying the sizes of the decomposed tensors G, H, also referred to as decoders RecWeightG, RecWeightH, respectively.

[0109] The decoder needs to derive the sizes of the G tensor and the H tensor, for example, when the layer is of type convolution (CONV) or depthwise convolution (DWCONV). Therefore, some embodiments of the present disclosure propose to transmit information about the size of the G tensor and / or the H tensor. The size of the G tensor and / or the H tensor refers to the size of the dimensions of the G tensor and / or the H tensor, such as the number of rows or columns of the tensor.

[0110] According to an exemplary syntax, such information can be added to a high-level syntax (HLS), for example, in a compressed data unit header. For example, in an exemplary syntax (such as an exemplary syntax compatible with some drafts of MPEG NNR), information about the tensor size of G and / or H and the rank decomposition_rank can be transmitted to a decoder in an NNR compressed data unit header as follows:

[0111] [Table 12]

[0112] Here, g_number_of_rows specifies the number of rows of matrix g when reconstruction is performed on a tensor decomposed into NNR unit type NNR_PT_BLOCK.

[0113] This information allows the decoder to perform the following reconstruction for NNR_PT_BLOCK (by specifying that sub-decomposed weights exist (block_parameter_types&0x01)!=0):

[0114] If (block_parameter_types&0x01)!=0, then the following applies:

[0115] We can invoke the decoding process for integer weight tensors with the input variable TensorDims set to [g_number_of_rows,decomposition_rank]. We can set the variable RecWeightG to the output variable RecParam.

[0116] When reconstruction is performed on a tensor decomposed into NNR unit type NNR_PT_BLOCK, the number of columns h_number_of_columns of matrix h can be obtained by:

[0117]

number

[0118] The integer weight tensor decoding process can then be invoked with the input variable TensorDims set to [decomposition_rank, h_number_of_columns]. The variable RecWeightH can be set to the output variable RecPatam.

[0119] The variable RecWeightH can be derived as follows: RecWeight=TensorReshape(RecWeightG * RecWeightH,tensor_dimensions)

[0120] In the above example syntax, the variable g_number_of_rows is now available from the unit header, so the variable h_number_of_columns can be derived since the dimensions of the output tensor dimensions (tensor_dimensions) are known. Thus, with such an embodiment, for example, a decoder can recreate the product of tensor G and tensor H by decoding them separately to obtain the recovered tensor RecWeight.

[0121] In another variant, the variable h_number_of_columns can be transmitted (e.g., as available from the unit header, similar to g_number_of_rows in the third variant detailed above), with g_number_of_rows being derived on the decoder side.

[0122] In yet another variant, both the variables g_number_of_rows and h_number_of_columns can be sent (e.g., similar to g_number_of_rows in the third variant detailed above, so that both are available from the unit header), thereby avoiding corresponding calculations, for example, on the decoder side.

[0123] FIG. 6 illustrates an example of a method 600 for decoding a tensor resulting from tensor decomposition according to the above-described embodiment. A bitstream containing coded data representing a neural network is input to a decoder. At 601, it is determined whether the original tensor has been decomposed into a first tensor and a second tensor, e.g., the first and second tensors are G tensor and H tensor, respectively, resulting from the sub-decomposition. At 602, if the current unit to be decoded includes a weighted tensor resulting from the tensor decomposition, the size of the first tensor is decoded from the bitstream. For example, for a G tensor, the size of the first tensor is the number of rows of the G tensor.

[0124] At 603, the first tensor is decoded based on the decoded size. At 604, the size of the second tensor is derived from the decoded size. For example, if the second tensor is an H tensor, the size of the second tensor is the number of columns of the H tensor. At 605, the second tensor is decoded based on the derived size. At 606, a decoder can reconstruct the decomposed tensor from the decoded first and second tensors.

[0125] In some embodiments, the bitstream includes the size of the second tensor instead of the size of the first tensor, or the size of both.

[0126] 7 shows an example of a method 700 for encoding tensors of a DNN in a bitstream according to one embodiment described above. At 701, a first tensor is decomposed into a second tensor and a third tensor. At 702, the size of the second tensor is encoded in the bitstream. At 703, parameters of the second tensor are encoded in the bitstream. At 704, parameters of the third tensor are encoded in the bitstream. In a variant, the size of the third tensor can also be encoded in the bitstream.

[0127] 8 illustrates an example of a portion of a bitstream 800 including data representing a first tensor of at least one layer of a deep neural network created according to any one of the above methods. In some embodiments, the data includes information 801 indicating that the first tensor is decomposed into a second tensor and a third tensor, a size 802 of at least one of the second tensor and the third tensor, and parameters 803 of at least one of the second tensor and the third tensor. In a variation, the data also includes parameters 804 of at least one other tensor of the second tensor and the third tensor.

[0128] Further embodiments and information This application describes various aspects, including tools, features, embodiments, models, techniques, and the like. Many of these aspects are described in specific, and often definitive, terms to at least indicate their individual characteristics. However, this is for clarity of description and does not limit the applicability or scope of the aspects. In fact, all of the different aspects can be combined and substituted to provide additional aspects. Furthermore, these aspects can also be similarly combined and substituted with aspects described in previous applications.

[0129] The aspects described in this application and embraced as possible can be implemented in many different forms. While Figures 1, 2, and 3 below provide some embodiments, other embodiments are contemplated, and the discussion of Figures 1, 2, and 3 is not intended to limit the breadth of implementations. At least one of the aspects generally relates to encoding and decoding (e.g., video encoding and video decoding, and / or encoding and decoding at least some weights of at least some layers of a DNN), and at least one other aspect generally relates to transmitting a generated or encoded bitstream. Aspects such as these can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0130] In this application, the terms "reconstructed" and "decoded" can be used interchangeably, and the terms "pixel" and "sample" can be used interchangeably, and the terms "image," "picture," and "frame" can be used interchangeably. Typically, although not necessarily, the term "reconstructed" is used on the encoder side, while the term "decoded" is used on the decoder side.

[0131] Various methods and other aspects described herein can be used to modify modules, such as intra-prediction modules, entropy coding modules, and / or decoding modules (160, 260, 145, 230), of the encoder 100 and decoder 200 as shown in Figures 1 and 2. Furthermore, aspects of the present disclosure are not limited to VVC or HEVC, but may be applied to, for example, other standards and recommendations, whether existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC).

[0132] This aspect is also not limited to VVC or HEVC, or even video data, but may be applied to an encoder or decoder adapted to encode or decode, respectively, at least one tensor of at least one layer of a neural network that may be used in many technical fields other than video (of course, in such an embodiment, some modules, such as intra prediction module 160, may be optional).

[0133] Unless otherwise specified or technically precluded, the aspects described in this application can be used individually or in combination.

[0134] Various numerical values ​​are used in this application (e.g., tensor recovery mode). The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0135] 1 shows an encoder 100. While variations of this encoder 100 are open to possibility, for the sake of clarity, the following description of the encoder 100 will be given without necessarily describing all possible variations.

[0136] Before being encoded, the sequence may undergo pre-encoding processing (101), such as, in the case of a video sequence, applying a color transformation to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or remapping the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Pre-encoding processing may also include binarization, such as the exemplary binarization detailed above in connection with CABAC.

[0137] Metadata can be associated with the preprocessing and added to the bitstream.

[0138] In the encoder 100, for a video sequence, pictures are coded by the encoder elements as described below. The picture to be coded is divided into units, e.g., CUs (102), and processed. Each unit is coded, e.g., using either intra mode or inter mode. If a unit is coded in intra mode, it undergoes intra prediction (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder determines (105) whether to use one of the intra mode or inter mode to code the unit, and indicates the intra / inter decision, e.g., by a prediction mode flag. A prediction residual is calculated, e.g., by subtracting (110) the prediction block from the original image block.

[0139] The prediction residual is then transformed (125) and quantized (130).

[0140] The quantized transform coefficients as well as syntax elements such as motion vectors are entropy coded to output a bitstream (145).

[0141] The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can ignore both the transform and the quantization, i.e., the residual is coded directly without applying either the transform or the quantization process.

[0142] The encoder decodes the coded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (140), inverse transformed (150), and the prediction residual is decoded. For example, for a video sequence, an image block is reconstructed by combining the decoded prediction residual with the prediction block (155). An in-loop filter (165) is applied to the reconstructed picture to reduce coding artifacts, for example, by performing deblocking / sample adaptive offset (SAO) filtering. The filtered image is stored in a reference picture buffer (180).

[0143] Figure 2 shows a block diagram of a decoder 200. In the decoder 200, the bitstream is decoded by decoder elements as described below. The decoder 200 typically performs a decoding pass that is nearly the reverse of the encoding pass, as shown in Figure 1. The encoder 100 also typically performs decoding as part of encoding data.

[0144] Specifically, the input to decoder 200 includes a bitstream, which may be generated by encoder 100. The bitstream is first entropy decoded (230) to obtain coded information such as transform coefficients, motion vectors, etc.

[0145] For a video bitstream, picture partition information indicates how a picture is partitioned, allowing a decoder to divide the picture according to the decoded picture partition information (235). To decode the prediction residual, the transform coefficients are inverse quantized (240) and inverse transformed (250). The decoded prediction residual is combined with the prediction block (255) to reconstruct an image block. The prediction block can be obtained (270) through intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0146] The decoded elements (such as picture weights or layer weights) may undergo further post-decoding processing (285), such as, in the case of a decoded image, an inverse color conversion (e.g., YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that reverses the remapping process performed in pre-encoding processing (101). The post-decoding processing may use metadata derived in pre-encoding processing and signaled in the bitstream.

[0147] FIG. 3 illustrates a block diagram of an example system in which various aspects and embodiments may be implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 1000, singly or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to perform one or more of the aspects described herein.

[0148] System 1000 includes at least one processor 1010 configured to execute instructions loaded thereon, e.g., to implement various aspects described herein. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes storage devices 1040, which may include non-volatile and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random-access memory (DRAM), static random access memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. Storage devices 1040 may include, by way of non-limiting example, internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0149] The system 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, an encoded or decoded data stream (e.g., a video stream and / or a stream representing at least one weight of at least one layer of at least one DNN), which may include its own processor and memory. The encoder / decoder module 1030 corresponds to a module that may be included in a device that performs encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0150] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the performance of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, data representing at least one weight of at least one tensor in at least one layer of at least one DNN, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and operational logic.

[0151] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and to provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, high-speed external dynamic volatile memory such as RAM is used as working memory during coding and decoding operations such as MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0152] Inputs to the elements of system 1000 can be provided through various input devices, shown in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals transmitted throughout a broadcast by, for example, a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Another example, not shown in FIG. 3, is composite video.

[0153] In various embodiments, the input devices of block 1130 have associated respective input processing elements known in the art. For example, the RF section can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band (e.g., a channel), which in certain embodiments may also be referred to as a channel, (iv) demodulating the downconverted, band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error correction device, and a demultiplexer. The RF section can include, for example, a tuner that performs a variety of these functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and re-filtering RF signals transmitted over a wired (e.g., cable) medium to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0154] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices over USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, within a separate interface IC or within processor 1010. For example, processor 1010 working in combination with memory and storage elements, and encoder / decoder 1030 provide the demodulated, error corrected, and demultiplexed stream to various processing elements, including encoder / decoder 1030, to process this data stream as desired for presentation to an output device.

[0155] The various elements of system 1000 may be provided within an integrated housing in which the various elements may be interconnected and data may be transmitted therebetween using suitable connection devices 1140, such as internal buses known in the art, including Inter-IC (I2C) buses, wiring, and printed circuit boards.

[0156] The system 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit data and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented within a wired and / or wireless medium, for example.

[0157] In various embodiments, data is streamed or otherwise provided to system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received over communication channel 1060 and communication interface 1050, which are adapted for Wi-Fi communication. Communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable over-the-top communication, such as streaming applications. In other embodiments, streaming data is provided to system 1000 using a set-top box that delivers data over an HDMI connection in input block 1130. In still other embodiments, streaming data is provided to system 1000 using an RF connection in input block 1130. As noted above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0158] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be for a television, a tablet, a laptop, a mobile phone, or another device. Display 1100 can also be integrated with other components (e.g., as found in a smartphone) or can be separate (e.g., an external monitor for a laptop). Other peripheral devices 1120, in various example embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (DVR, as an abbreviation for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of system 1000. For example, a disc player serves to play the output of the system 1000 .

[0159] In various embodiments, control signals are communicated between system 1000 and display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable inter-device control with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 using communication channel 1060 via communication interface 1050. Display 1100 and speaker 1110 can be integrated into a single unit with other components of system 1000 in an electronic device such as, for example, a television. In various embodiments, display interface 1070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0160] For example, if the RF portion of input 1130 is part of a separate set-top box, display 1100 and speakers 1110 may alternatively be separate from one or more of the other components. In various embodiments in which display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0161] The embodiments may be performed by computer software implemented by the processor 1010, by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type appropriate to the technology environment, and may be implemented using any suitable data storage technology, such as, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and rib-bubble memory. The processor 1010 may be of any type appropriate to the technology environment, and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0162] Various implementations involve decoding. As used herein, "decoding" can encompass all or some of the processes performed on a received encoded sequence to produce a final output suitable for, for example, a display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as, for example, entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by decoders in various implementations described herein.

[0163] As a further example, in one embodiment, "decoding" refers to entropy decoding only, in another embodiment, "decoding" refers to differential decoding only, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "encoding process" is intended to refer specifically to a work subset or to the broader encoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0164] Various implementations involve encoding. Similar to the above discussion of "decoding," "encoding," as used herein, can encompass, for example, all or some of the processes performed on an input sequence to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as, for example, partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by the encoders of the various implementations described herein.

[0165] As a further example, in one embodiment, "encoding" refers to entropy encoding only, in another embodiment, "encoding" refers to differential encoding only, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0166] It should be noted that syntax element, as used herein, is a descriptive term and therefore does not preclude the use of other syntax element names.

[0167] Where a figure is presented as a flow diagram, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow diagram of the corresponding method / process.

[0168] Various embodiments refer to parametric models or rate-distortion optimization. Specifically, during the encoding process, the balance or trade-off between rate and distortion is typically considered, often providing a computational complexity constraint. This can be measured through a Rate Distortion Optimization (RDO) metric, or through Least Mean Square (LMS), Mean of Absolute Errors (MAE), or other such measures. Rate-distortion optimization is typically formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all coding options, including all possible modes or coding parameter values, with a thorough evaluation of their coding costs and associated distortions of the reconstructed signal after encoding and decoding. Furthermore, more rapid approaches may also be used to reduce coding complexity, particularly with computational calculation of approximate distortions based on prediction or prediction residual signals rather than the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some of the possible encoding choices and full distortion for others. Other approaches evaluate only a subset of the possible encoding choices. More generally, many approaches employ any of a variety of techniques to perform optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and the associated distortion.

[0169] Implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of one form of implementation (e.g., discussed only as a method), the implementation of the discussed feature may also be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus may be implemented in appropriate hardware, software, and firmware. A method may be implemented in, for example, a processor, which refers to a general processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0170] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, means that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "an embodiment" or "in one implementation" or "in an implementation," as well as other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0171] Also, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0172] Also, in this application, reference may be made to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0173] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from memory). Furthermore, "receiving" typically involves in some way, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0174] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it will be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded to include as many items as are listed, as would be apparent to one skilled in this and related arts.

[0175] Also, as used herein, the term "signal" refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals at least one of multiple transforms, coding modes, or coding flags. Thus, in some embodiments, the same parameters are used at both the encoder and decoder sides. Thus, for example, an encoder may transmit a specific parameter to a decoder (explicit signaling) so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling without transmission (implicit signaling) may be used to simply allow the decoder to know and select the specific parameter. By avoiding transmitting any actual functionality, bit savings are realized in various embodiments. It will be understood that signaling can be achieved in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. The above relates to the verb form of the word "signal," which may also be used as a noun herein.

[0176] As will be apparent to those skilled in the art, implementations can provide a variety of signals formatted to carry information that can be stored or transmitted, for example. Information can include, for example, instructions to perform a method or data provided by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. For example, such a signal can be formatted as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

[0177] Several embodiments are described. Features of these embodiments may be provided alone or in any combination across various claim categories and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types. A process or device for encoding and decoding using deep neural network compression of a pre-trained deep neural network. A process or device that performs encoding and decoding of at least one layer of a pre-trained deep neural network to perform deep neural network compression. A process or device that performs deep neural network compression of a pre-trained deep neural network with one or more layers by encoding and decoding using information inserted into a bitstream representing parameters. A process or device that performs deep neural network compression of deep neural networks by encoding and decoding using information inserted into a bitstream representing parameters. A bitstream or signal containing one or more of the described syntax elements, or variations thereof. A bitstream or signal containing syntax conveying information generated by any of the described embodiments. · Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments. A method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments. Inserting syntax elements into the signaling that allow the decoder to determine the coding mode in a manner that corresponds to the method used by the encoder. · Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof. A television, set-top box, mobile phone, tablet, or other electronic device that performs a conversion method according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that performs the conversion method determination according to any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display). A television, set-top box, mobile phone, tablet, or other electronic device that selects, bandlimits, or tunes (e.g., using a tuner) a channel to receive a signal containing encoded images and performs a conversion method according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that receives signals containing coded images over the air and performs a conversion process.

Claims

1. 1. A method comprising: In response to determining that a first tensor of a layer of a deep neural network is decomposed in the bitstream into a second tensor and a third tensor whose parameters are encoded, decoding from the bitstream one or more sizes corresponding to at least one of the second tensor and the third tensor; decoding the second tensor and the third tensor based on the one or more decoded sizes to obtain the decoded second tensor and the decoded third tensor; A method for providing the above.

2. decoding, from the bitstream, a decomposition rank of a tensor decomposition of the first tensor decomposed into the second tensor and the third tensor; The method of claim 1 further comprising:

3. deriving one or more sizes of one or more tensors of the second tensor or the third tensor based on the one or more decoded sizes; decoding one or more of the second tensor and the third tensor based on the one or more derived sizes; The method of claim 1 further comprising:

4. Reconstructing the first tensor based on the decoded second tensor and the decoded third tensor. The method of claim 1 further comprising:

5. storing one or more of the decoded second tensor and the decoded third tensor in a decoded tensor buffer; determining whether one or more of the decoded second tensor and the decoded third tensor are in the decoded tensor buffer by looking up a tensor associated with an identifier, the identifier including the same layer as the one or more of the decoded second tensor and the decoded third tensor; The method of claim 1 further comprising:

6. 1. An apparatus comprising one or more processors, the one or more processors comprising: In response to determining that a first tensor of a layer of a deep neural network is decomposed in the bitstream into a second tensor and a third tensor whose parameters are encoded, decoding one or more sizes corresponding to at least one of the second tensor and the third tensor from the bitstream; Decoding the second tensor and the third tensor based on the one or more decoded sizes to obtain the decoded second tensor and the decoded third tensor. An apparatus configured to:

7. The one or more processors: deriving one or more sizes of one or more tensors of the second tensor or the third tensor based on the one or more decoded sizes; Decoding one or more of the second tensor and the third tensor based on one or more derived sizes. The apparatus of claim 6 further configured to:

8. 7. The apparatus of claim 6, wherein the one or more processors are further configured to recover the first tensor based on the decoded second tensor and the decoded third tensor.

9. The one or more processors: storing one or more of the decoded second tensor and the decoded third tensor in a decoded tensor buffer; Determine whether one or more of the decoded second tensor and the decoded third tensor are in the decoded tensor buffer by looking up a tensor associated with an identifier, the identifier including the same layer as the one or more of the decoded second tensor and the decoded third tensor. The apparatus of claim 6 further configured to:

10. 1. A method comprising: Decomposing a first tensor of a layer of a deep neural network into a second tensor and a third tensor; deriving one or more sizes corresponding to at least one of the second tensor and the third tensor; encoding the second tensor and the third tensor in a bitstream based on the determined one or more sizes, wherein the one or more sizes corresponding to at least one of the second tensor and the third tensor are encoded in the bitstream; A method comprising:

11. 11. The method of claim 10, further comprising encoding into the bitstream a decomposition rank of a tensor decomposition of the first tensor decomposed into the second tensor and the third tensor.

12. The method of claim 10 , further comprising storing one or more of the second tensor and the decomposed third tensor in a tensor buffer.

13. 1. An apparatus comprising one or more processors, the one or more processors comprising: Decomposing a first tensor of a layer of a deep neural network into a second tensor and a third tensor; deriving one or more sizes corresponding to at least one of the second tensor and the third tensor; encoding the second tensor and the third tensor in a bitstream based on the determined one or more sizes, and the one or more sizes corresponding to at least one of the second tensor and the third tensor are encoded in the bitstream. An apparatus configured to:

14. 14. The apparatus of claim 13, wherein the one or more processors are further configured to encode into the bitstream a decomposition rank of a tensor decomposition of the first tensor decomposed into the second tensor and the third tensor.

15. 14. The apparatus of claim 13, wherein the one or more processors are further configured to store one or more of the second tensor and the decomposed third tensor in a tensor buffer.

Images