Neural network decoding method, neural network coding method, visual media data processing method and device, and storage medium

Abstract

The embodiment of the invention provides a neural network decoding method, a neural network coding method, a visual media data processing method, a device and a storage medium, and the method comprises the steps: receiving a first syntax element in a model parameter set from a bit stream of a compressed neural network representation NNR, the first syntax element indicates whether coding tree unit (CTU) block partitioning is enabled for a tensor in an NNR aggregation unit; and reconstructing a tensor in the NNR aggregation unit based on the first syntax element.

Description

[0001] This application is a divisional application of Chinese patent application number 202180005471.X, filed on April 19, 2021, and entitled “Method, device, computer equipment and computer-readable medium for neural network decoding”. Technical Field

[0002] This disclosure describes embodiments generally related to neural network model compression / decompression. Background Art

[0003] The background description provided herein is intended to generally present the context of the present disclosure. To the extent described in this background section, the works of the currently named inventors and aspects of the description that do not qualify as prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present disclosure.

[0004] Various applications in computer vision, image recognition, and speech recognition rely on neural networks to achieve improved performance. Neural networks are based on a collection of connected nodes (also called neurons), loosely modeled after neurons in a biological brain. Neurons can be organized into multiple layers. Neurons in one layer can connect to neurons in both the immediately preceding and following layers.

[0005] A connection between two neurons, such as a synapse in a biological brain, can transmit a signal from one neuron to another. The receiving neuron then processes the signal and can send signals to other connected neurons. In some examples, to find the output of a neuron, the inputs to the neuron are weighted by the weights of the connections from the inputs to the neuron, and the weighted inputs are summed to generate a weighted sum. A bias can be added to the weighted sum. Furthermore, the weighted sum is then passed through an activation function to produce the output. Summary of the Invention

[0006] Aspects of the present invention provide methods and apparatus for neural network model compression / decompression. In some examples, an apparatus for neural network model decompression includes a processing circuit. The processing circuit may be configured to receive a first syntax element in an NNR aggregation unit header of a compressed neural network representation (NNR) of a neural network from a bitstream of a compressed NNR aggregation unit. The first syntax element may indicate a coding tree unit (CTU) scanning order for processing tensors in the NNR aggregation unit. The tensors in the NNR aggregation unit may be reconstructed based on the CTU scanning order indicated by the first syntax element.

[0007] In one embodiment, a first value of the first syntax element may indicate that the CTU scan order is a first raster scan order along the horizontal direction, and a second value of the first syntax element may indicate that the CTU scan order is a second raster scan order along the vertical direction. In one embodiment, a second syntax element may be received in an NNR aggregation unit header of the NNR aggregation unit from a bitstream. The second syntax element may indicate a maximum bit depth of quantized coefficients of tensors in the NNR aggregation unit.

[0008] In one embodiment, a third syntax element may be received, the third syntax element indicating whether CTU block partitioning is enabled for tensors in the NNR aggregation unit. The third syntax element may be a model-related syntax element or a tensor-related syntax element, the model-related syntax element is used to specify whether CTU block partitioning is enabled for layers of a neural network, and the tensor-related syntax element is used to specify whether CTU block partitioning is enabled for tensors in the NNR aggregation unit.

[0009] In one embodiment, a fourth syntax element related to a model or a tensor may be received, the fourth syntax element indicating the CTU dimension of the tensor in the NNR aggregation unit. In one embodiment, the NNR unit may be received before any NNR aggregation unit is received. The NNR unit may include a fifth syntax element indicating whether CTU splitting is enabled.

[0010] In some examples, another apparatus for decompressing a neural network model includes processing circuitry. The processing circuitry may be configured to receive one or more first syntax elements from a bitstream representing a compressed neural network representation, the first syntax elements associated with a three-dimensional coding unit (CU3D) partitioned from a first three-dimensional coding tree unit (CTU3D). The first CTU3D may be derived from a tensor partition in the neural network. The one or more first syntax elements may indicate that the CU3D is partitioned based on a 3D pyramid tree structure comprising multiple depths. Each depth corresponds to one or more nodes. Each node has a node value. Second syntax elements corresponding to node values of nodes in the 3D pyramid tree structure may be received from the bitstream in a breadth-first scan order for scanning the nodes in the 3D pyramid tree structure. Model parameters of the tensor may be reconstructed based on the received second syntax elements corresponding to the node values of each node in the 3D pyramid tree structure. In various embodiments, the 3D pyramid tree structure is one of an octree structure, a single-tree structure, a tag tree structure, and a single-tag tree structure.

[0011] In one embodiment, a second syntax element is received starting from a starting depth within the depth of the 3D pyramid tree structure. The starting depth may be indicated in the bitstream or inferred at the decoder. In one embodiment, a third syntax element may be received, indicating a starting depth for receiving a second syntax element, where the second syntax element indicates a node value of a node in the 3D pyramid tree structure. When the starting depth is the last depth of the 3D pyramid tree structure, a non-3D pyramid tree-based decoding method may be used to decode the tensor model parameters from the bitstream.

[0012] In one embodiment, a third syntax element may be received, the third syntax element indicating a starting depth for receiving a second syntax element, the second syntax element indicating a node value of each node in the 3D pyramid tree structure. When the starting depth is the last depth of the 3D pyramid tree structure, the second syntax element is received starting from the second-to-last depth in the 3D pyramid tree structure.

[0013] In another embodiment, a third syntax element may be received, the third syntax element indicating a starting depth for receiving a second syntax element, the second syntax element indicating a node value of a node in a 3D pyramid tree structure. When the starting depth is the last depth of the 3D pyramid tree structure and the 3D pyramid tree structure is a single-branch tag tree structure associated with single-branch tree partial coding and tag tree partial coding, for the single-branch tree partial coding, the second syntax element is received starting from the second-to-last depth in the depth of the 3D pyramid tree structure; and for the tag tree partial coding, the second syntax element is received starting from the last depth in the depth of the 3D pyramid tree structure.

[0014] In one embodiment, when one or more first syntax elements indicate that the CU3D is partitioned based on a 3D pyramid tree structure, dependent quantization is disabled. In one embodiment, when one or more first syntax elements indicate that the CU3D is partitioned based on a 3D pyramid tree structure, a dependent quantization construction process may be performed. Model parameters of tensors skipped during the encoding process based on the 3D pyramid tree structure are excluded from the dependent quantization construction process.

[0015] In one embodiment, a fourth syntax element associated with a CU3D may be received, the fourth syntax element indicating whether all model parameters of the CU3D are unified. In one embodiment, a zero value may be used as a forward neighbor value for a first coefficient in a kernel of a tensor to determine a context model for entropy decoding the first coefficient in the kernel. In one embodiment, one or more fifth syntax elements may be received in the bitstream, the fifth syntax element indicating a width or height of a second CTU3D in the tensor. When the width, height, or both width and height are model parameters, a determination may be made to decode the model parameters of the second CTU3D based on the baseline coding method.

[0016] In some examples, another apparatus for decompressing a neural network model includes processing circuitry. The processing circuitry may be configured to receive a first syntax element in a bitstream of a compressed neural network representation of a neural network, the first syntax element associated with a CTU3D resulting from tensor partitioning in a layer of the neural network. The first syntax element may indicate whether all child nodes at a bottom depth of a pyramid tree structure associated with the CTU3D are uniform. When the first syntax element indicates that all child nodes at a bottom depth of the pyramid tree structure associated with the CTU3D are uniform, the CTU3D may be decoded based on a three-dimensional unary tree (3D unary tree) encoding method.

[0017] In one embodiment, a second syntax element associated with a layer of a neural network may be received in a bitstream. The second syntax element may indicate whether the layer is encoded using a pyramid tree structure-based encoding method. In one embodiment, child nodes at a bottom depth that do not share the same parent node have different uniform values.

[0018] In one embodiment, the starting depth of the 3D unary tree encoding method can be inferred to be the bottom depth of the pyramid tree structure. In one embodiment, a unified flag for nodes at the bottom depth of the pyramid tree structure is not encoded in the bitstream. In one embodiment, a unified value encoded for all child nodes sharing the same parent node at the bottom depth can be received in the bitstream. A sign bit for all child nodes sharing the same parent node at the bottom depth can be received. The sign bit follows the unified value in the bitstream.

[0019] In one embodiment, a uniform value for each group of child nodes sharing the same parent node at the bottom depth may be received from the bitstream.Sign bits for child nodes in each group of child nodes sharing the same parent node at the bottom depth may be received.

[0020] In one embodiment, in response to a first syntax element indicating that all child nodes at a bottom depth of a pyramid tree structure associated with the CTU 3D are not uniform, the CTU 3D may be decoded based on a three-dimensional tag tree (3D tag tree) encoding method. In one embodiment, the starting depth of the 3D tag tree encoding method may be inferred to be the bottom depth of the pyramid tree structure.

[0021] In one embodiment, the value of the node at the bottom depth of the pyramid tree structure may be decoded according to one of the following: receiving the value of the node at the bottom depth of the pyramid tree structure, the value of each node being encoded in a bitstream based on a predetermined scanning order; receiving the absolute value of each node at the bottom depth of the pyramid tree structure in the bitstream based on the predetermined scanning order, and then receiving the sign of each node if the absolute value is not zero; or receiving the absolute value of each node at the bottom depth of the pyramid tree structure in the bitstream based on the predetermined scanning order, and then receiving the sign of each node at the bottom depth of the pyramid tree structure in the bitstream based on the predetermined scanning order if the node has a non-zero value.

[0022] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for neural network model decompression, cause the computer to perform a method for neural network model decompression. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Other features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0024] Figure 1 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0025] Figure 2 An exemplary compressed neural network representation (NNR) unit is shown;

[0026] Figure 3 An exemplary polymeric NNR unit is shown;

[0027] Figure 4 An exemplary NNR bitstream is shown;

[0028] Figure 5 An exemplary NNR unit grammar is shown;

[0029] Figure 6 An exemplary NNR unit header syntax is shown;

[0030] Figure 7 An exemplary NNR aggregation unit header syntax is shown;

[0031] Figure 8 An exemplary NNR unit payload syntax is shown;

[0032] Figure 9 An exemplary NNR model parameter set payload syntax is shown;

[0033] Figure 10 An exemplary syntax for a model parameter set is shown;

[0034] Figure 11 An exemplary syntax of an aggregation unit header is shown, which includes signaling of a coding tree unit (CTU) scan order for one or more weight tensors;

[0035] Figure 12 shows an example of the syntax for scanning weight coefficients in a weight tensor;

[0036] Figure 13 An example for decoding the absolute value of a quantized weight coefficient according to some embodiments of the present disclosure is shown;

[0037] Figure 14 Two examples of adaptive 3D CTU (CTU3D) / 3D Coding Unit (CU3D) partitioning using raster scanning along the vertical direction are shown;

[0038] Figure 15 An exemplary partitioning process based on a 3D pyramid tree structure is shown;

[0039] Figure 16 Two scalar quantizers according to an embodiment of the present disclosure are shown;

[0040] Figure 17 The decoding process of CTU partitioning according to one embodiment of the present disclosure is shown;

[0041] Figure 18 The decoding process of 3D pyramid coding according to one embodiment of the present disclosure is shown;

[0042] Figure 19 The present invention shows a decoding process based on unified coding according to an embodiment of the present disclosure;

[0043] Figure 20 is a schematic diagram of a computer system according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] Various aspects of the present disclosure provide various techniques for neural network model compression / decompression. These techniques involve coding tree unit (CTU) block partitioning, 3D pyramid tree structure-based encoding, and unified-based encoding.

[0045] Artificial neural networks can be used for a wide range of tasks in multimedia analysis and processing, media coding, data analysis and many other fields. The success of using artificial neural networks is based on the feasibility of processing much larger and more complex neural networks (deep neural networks, DNNs) and the availability of large-scale training data sets compared to the past. Therefore, a trained neural network can contain a large number of model parameters, resulting in a considerable capacity (e.g., hundreds of MB). Model parameters may include coefficients of the trained neural network, such as weights, biases, scaling factors, batch normalization (batch norm, batchnorm) parameters, etc. These model parameters can be organized into model parameter tensors. Model parameter tensors are used to refer to a multidimensional structure (e.g., an array or matrix) that combines the relevant model parameters of a neural network. For example, when available, the coefficients of a layer in a neural network can be divided into weight tensors, bias tensors, scaling factor tensors, batch norm tensors, etc.

[0046] Many applications require the potential deployment of a particular trained network instance to a larger number of devices that may have limitations in processing power and storage (e.g., mobile devices or smart cameras) as well as limitations in communication bandwidth. These applications can benefit from the neural network compression / decompression techniques disclosed herein.

[0047] I. Neural Network-Based Devices and Applications

[0048] Figure 1 A block diagram of an electronic device (130) according to one embodiment of the present disclosure is shown. The electronic device (130) may be configured to run a neural network-based application. In some embodiments, the electronic device (130) receives and stores a compressed (encoded) neural network model (e.g., a compressed representation of a neural network in the form of a bitstream). The electronic device (130) may decompress (or decode) the compressed neural network model to restore the neural network model, and may run an application based on the neural network model. In some embodiments, the compressed neural network model is provided from a server such as an application server (110).

[0049] exist Figure 1In an example, the application server (110) includes a processing circuit (120), a memory (115), and an interface circuit (111) coupled together. In some examples, a neural network is appropriately generated, trained, or updated. The neural network can be stored in the memory (115) as a source neural network model. The processing circuit (120) includes a neural network model codec (121). The neural network model codec (121) includes an encoder that can compress the source neural network model and generate a compressed neural network model (a compressed representation of the neural network). In some examples, the compressed neural network model is in the form of a bitstream. The compressed neural network model can be stored in the memory (115). The application server (110) can provide the compressed neural network model to other devices, such as the electronic device (130), in the form of a bitstream through the interface circuit (111).

[0050] It should be noted that the electronic device (130) may be any suitable device, such as a smartphone, a camera, a tablet computer, a laptop computer, a desktop computer, a gaming headset, etc.

[0051] exist Figure 1 In some examples, the electronic device (130) includes a processing circuit (140), a cache (150), a main memory (160), and an interface circuit (131) coupled together. In some examples, the compressed neural network model is received by the electronic device (130) through the interface circuit (131), for example, in the form of a bit stream. The compressed neural network model is stored in the main memory (160).

[0052] The processing circuit (140) includes any suitable processing hardware, such as a central processing unit (CPU), a graphics processing unit (GPU), etc. The processing circuit (140) includes components suitable for executing neural network-based applications and includes components suitable for being configured as a neural network model codec (141). The neural network model codec (141) includes a decoder that can decode a compressed neural network model received, for example, from the application server (110). In one example, the processing circuit (140) includes a single chip (e.g., an integrated circuit) on which one or more processors are disposed. In another example, the processing circuit (140) includes multiple chips, each of which can include one or more processors.

[0053] In some embodiments, the main memory (160) has a relatively large storage space and can store various information, such as software code, media data (e.g., video, audio, images, etc.), compressed neural network models, etc. The cache (150) has a relatively small storage space, but has a much faster access speed than the main memory (160). In some examples, the main memory (160) may include a hard disk drive, a solid-state drive, etc., and the cache (150) may include a static random access memory (SRAM), etc. In one example, the cache (150) may be an on-chip memory provided on, for example, a processor chip. In another example, the cache (150) may be an off-chip memory provided on one or more memory chips separated from the processor chip. Generally, on-chip memory has a faster access speed than off-chip memory.

[0054] In some embodiments, when the processing circuit (140) executes an application using a neural network model, the neural network model codec (141) may decompress the compressed neural network model to restore the neural network model. In some examples, the cache (150) is large enough so that the restored neural network model can be buffered in the cache (150). The processing circuit (140) can then access the cache (150) to use the restored neural network model in the application. In another example, the cache (150) has limited memory space (e.g., on-chip memory), the compressed neural network model can be decompressed layer by layer or block by block, and the cache (150) can buffer the restored neural network model layer by layer or block by block.

[0055] It should be noted that the neural network model codec (121) and the neural network model codec (141) can be implemented using any suitable technology. In some embodiments, the encoder and / or decoder can be implemented by an integrated circuit. In some embodiments, the encoder and decoder can be implemented as one or more processors that execute a program stored in a non-transitory computer-readable medium. The neural network model codec (121) and the neural network model codec (141) can be implemented according to the encoding and decoding features described below.

[0056] This disclosure provides techniques for compressing neural network representations (NNRs), which can be used to encode and decode neural network models, such as deep neural network (DNN) models, to save storage and computation. Deep neural networks (DNNs) can be used in a wide range of video applications, such as semantic classification, object detection / recognition, and video quality enhancement.

[0057] A neural network (or artificial neural network) typically includes multiple layers between an input layer and an output layer. In some examples, a layer in a neural network corresponds to a mathematical transformation that converts the input of that layer into the output of that layer. The mathematical transformation can be a linear relationship or a nonlinear relationship. The neural network can move through each layer to calculate the probability of each output. In this way, each mathematical transformation is considered a layer, and a complex DNN can have multiple layers. In some examples, the mathematical transformation of a layer can be represented by one or more tensors (e.g., a weight tensor, a bias tensor, a scaling factor tensor, a batch norm tensor, etc.).

[0058] II. Block Definition and Usage of Neural Network Model Compression

[0059] 1. Advanced Syntax

[0060] In some embodiments, a high-level syntax for a bitstream carrying a neural network (model) in a compressed or encoded representation can be defined based on the concept of an NNR unit. An NNR unit is a data structure for carrying neural network data and related metadata. An NNR unit carries compressed or uncompressed information related to neural network metadata, topology information, all or part of layer data, filters, kernels, biases, quantized weights, tensors, etc.

[0061] Figure 2 An exemplary NNR unit (200) is shown. As shown, the NNR unit (200) may include the following data elements:

[0062] - NNR Unit Size: This data element signals the total byte size of the NNR unit including the NNR Unit Size itself.

[0063] - NNR Unit Header: This data element contains information related to the NNR unit type and related metadata.

[0064] -NNR Unit Payload: This data element contains compressed or uncompressed data related to the neural network.

[0065] Figure 3 An exemplary aggregated NNR unit (300) is shown. An aggregated NNR unit (300) is an NNR unit that can carry multiple NNR units in its payload. The aggregated NNR unit provides a grouping mechanism for several NNR units that are related to each other and benefit from being aggregated under a single NNR unit. Figure 4 An exemplary NNR bitstream (400) is shown. The NNR bitstream (400) may include a sequence of NNR units. The first NNR unit in the NNR bitstream may be an NNR start unit (ie, an NNR unit of type NNR_STR).

[0066] Figure 5An exemplary NNR unit syntax is shown. Figure 6 An exemplary NNR unit header syntax is shown. Figure 7 An exemplary NNR aggregation unit header syntax is shown. Figure 8 An exemplary NNR unit payload syntax is shown. Figure 9 An exemplary NNR model parameter set payload syntax is shown.

[0067] 2. Reshape and scan sequence

[0068] In some examples, the dimension of the weight tensor is greater than 2 (e.g., in a convolutional layer, the dimension is 4), and the weight tensor can be reshaped into a two-dimensional (2D) tensor. In one example, if the dimension of the weight tensor does not exceed 2 (e.g., a fully connected layer or a bias layer), no reshaping is performed. To encode the weight tensor, the weight coefficients in the weight tensor are scanned in a certain order. In some examples, the weight coefficients in the weight tensor can be scanned from left to right for each row and from the top row to the bottom row, for example, in a row-first manner.

[0069] 3. Block division mark

[0070] In some embodiments, the weight tensor may be reshaped into a 2D tensor, which is then partitioned into blocks called coding tree units (CTUs). The coefficients of the 2D tensor may then be encoded based on the resulting CTU blocks. For example, a scan order may be defined based on these CTU blocks, and encoding may be performed according to the scan order. In some other embodiments, the weight tensor may be reshaped into a 3D tensor, for example, with the number of input channels and output channels as the first and second dimensions, respectively, and the number of elements in the kernel (filter) as the third dimension. Block partitioning may then be performed along the planes of the first and second dimensions, resulting in 3D blocks called CTU3Ds. Thus, encoding of the tensor may be performed based on the scan order of these CTU3Ds. In some examples, the CTU blocks or CTU3D blocks may be blocks of equal size.

[0071] In some embodiments, a model-related syntax element ctu_partition_flag is used to specify whether block partitioning (CTU partitioning) is enabled for the weight tensors of each layer of the neural network. For example, a first value of the syntax element (e.g., a value of 0) indicates that block partitioning is disabled, and a second value of the syntax element (e.g., a value of 1) indicates that block partitioning is enabled. In one embodiment, the syntax element is a 1-bit flag. In another embodiment, the syntax element may be represented by multiple bits. One value of the syntax element indicates whether block partitioning is performed. For example, a value of zero may indicate that partitioning is not performed. Other values of the syntax element may be used to indicate the size of a CTU or CTU3D block.

[0072] Figure 10 An exemplary syntax of a model parameter set is shown. The model parameter set includes ctu_partition_flag. The ctu_parition_flag can be used to enable or disable block partitioning of the neural network model controlled by the model parameter set.

[0073] In some embodiments, a syntax element ctu_partition_flag associated with a tensor may be used to specify whether block partitioning (CTU partitioning) is enabled for one or more individual weight tensors of a neural network. For example, a first value of the syntax element (e.g., a value of 0) indicates that block partitioning is disabled for the corresponding one or more tensors, and a second value of the syntax element (e.g., a value of 1) indicates that block partitioning is enabled for the corresponding one or more tensors. In one embodiment, the syntax element is a 1-bit flag. In another embodiment, the syntax element may be represented by multiple bits. One value of the syntax element indicates whether block partitioning is performed on the corresponding one or more tensors. For example, a value of zero may indicate that partitioning is not performed on the corresponding one or more tensors. Other values of the syntax element may be used to indicate the size of the CTU or CTU3D block partitioned from the corresponding one or more tensors.

[0074] In one example, the syntax element ctu_partition_flag related to the tensor is included in the compressed data unit header. In one example, the syntax element ctu_partition_flag related to the tensor is included in the aggregation unit header.

[0075] 4. Signal representation in CTU dimension

[0076] In one embodiment, when the model-related flag ctu_partition_flag has a value indicating that block partitioning is enabled, the model-related 2-bit max_ctu_dim_flag can be used to specify the model-related maximum CTU dimension (denoted as gctu_dim) of the weight tensor of the neural network. For example, gctu can be determined according to the following formula:

[0077] gctu_dim=(64>>max_ctu_dim_flag).

[0078] For example, corresponding to a max_ctu_dim_flag value of 0, 1, 2, or 3, gctu_dim may have values of 64, 32, 16, and 8.

[0079] In one embodiment, for a 2D reshaped tensor, the maximum CTU width associated with the tensor may be scaled proportionally with the kernel size of each convolution tensor as follows:

[0080] max_ctu_height=gctu_dim,

[0081] max_ctu_width=gctu_dim*kernel_size.

[0082] The height / width of the right / bottom CTU can be less than max_ctu_height / max_ctu_width. Note that the number of bits in max_ctu_dim_flag can be changed to other values (e.g., greater than 2 bits). Other mapping functions involving max_ctu_dim_flag can be used to calculate gctu_dim. max_ctu_width can scale non-proportionally with kernel_size (e.g., for dividing CTU 3D blocks). Alternatively, max_ctu_width can scale proportionally with any arbitrary value to form appropriate 2D or 3D blocks of various sizes.

[0083] In another embodiment, when the model-related or tensor-related flag ctu_partition_flag has a value indicating that block partitioning is enabled, the 2-bit max_ctu_dim_flag associated with the tensor may be used to specify the maximum CTU dimension associated with the tensor of the corresponding weight tensor of the neural network according to the following formula:

[0084] gctu_dim=(64>>max_ctu_dim_flag).

[0085] In one embodiment, the maximum CTU width associated with a tensor may scale proportionally with the kernel size of each convolution tensor as follows:

[0086] max_ctu_height=gctu_dim,

[0087] max_ctu_width=gctu_dim*kernel_size.

[0088] Similarly, the height / width of the right / bottom CTU can be less than max_ctu_height / max_ctu_width. The number of bits of the tensor-related max_ctu_dim_flag can be changed to other values (e.g., greater than 2 bits). Other mapping functions involving the tensor-related max_ctu_dim_flag can be used to calculate gctu_dim. max_ctu_width can not scale proportionally with kernel_size (e.g., for dividing CTU 3D blocks). max_ctu_width can scale proportionally with any value to form appropriate 2D or 3D blocks of various sizes.

[0089] 5.CTU scanning order

[0090] In some embodiments, the tensor-related syntax element ctu_scan_order is used to specify a CTU-related (or CTU3D) scan order for the corresponding one or more tensors. For example, a first value (e.g., a value of 0) of the tensor-related syntax element ctu_scan_order indicates that the scan order associated with the CTU is a raster scan order along the horizontal direction. A second value (e.g., a value of 1) of the tensor-related syntax element ctu_scan_order indicates that the scan order associated with the CTU is a raster scan order along the vertical direction. In one example, the tensor-related syntax element ctu_scan_order is included in a compressed data unit header. In one example, the tensor-related syntax element ctu_scan_order is included in an aggregation unit header.

[0091] In one embodiment, when the model-related syntax element flag ctu_partition_flag has a value indicating that block (CTU) partitioning is enabled, the tensor-related syntax element ctu_scan_order is included in the syntax table.

[0092] Figure 11 An exemplary syntax of an aggregation unit header is shown, which includes signaling of a CTU scan order for one or more weight tensors. At line (1101), a FOR loop is performed on a plurality of NNR units. Each of the plurality of NNR units may include, for example, a weight tensor. For each of the plurality of NNR units, when ctu_partition_flag has a value indicating that CTU partitioning is enabled (line (1102)), ctu_scan_order[i] may be received, where i may be an index of each of the plurality of NNR units. ctu_partition_flag may be a model-dependent syntax element.

[0093] Additionally, at line (1102), a syntax element quant_bitdepth[i] may be received for each of one or more NNR units. Quant_bitdepth[i] may specify a maximum bit depth for quantized coefficients of each tensor in the NNR aggregation unit.

[0094] In another embodiment, when the tensor-related syntax element flag ctu_partition_flag has a value indicating that block (CTU) partitioning is enabled for the corresponding one or more tensors, the tensor-related syntax element ctu_scan_order is included in the syntax table.

[0095] 6. Flag Dependency

[0096] In one example, the ctu_partition_flag is defined as a model-dependent flag. The ctu_scan_order flag is placed in the nnr_aggregate_unit_header section (e.g., in Figure 11 ctu_partition_flag is placed in the nnr_model_parameter_set_payload section inside nnr_aggregate_unit. nnr_aggregate_unit_header may be serialized before nnr_model_parameter_set_payload, which makes it impossible to decode ctu_scan_order.

[0097] To solve this problem, in one example, the model-related nnr_unit can be serially arranged before any nnr_aggregate_unit. The ctu_partition_flag and max_ctu_dim_flag can be included in the nnr_unit. Therefore, the NNR aggregation unit following the nnr_unit can use any information defined in the nnr_unit.

[0098] III. 3D Pyramid Coding for Neural Network Model Compression

[0099] 1. Scanning order

[0100] Figure 12 An example of a syntax for scanning weight coefficients in a weight tensor is shown. For example, the dimension of the weight tensor is greater than 2 (e.g., in a convolutional layer, the dimension is 4), and the weight tensor can be reshaped into a two-dimensional tensor. In one example, if the dimension of the weight tensor does not exceed 2 (e.g., a fully connected layer or a bias layer), no reshaping is performed. In order to encode the weight tensor, the weight coefficients in the weight tensor are scanned in a certain order. In some examples, the weight coefficients in the weight tensor can be scanned from left to right for each row and from the top row to the bottom row, for example, in a row-first manner.

[0101] exist Figure 12 In the example, the 2D integer array StateTransTab[][] specifies the state transition table for scalar-dependent quantization and can be configured as follows:

[0102] StateTransTab[][]={{0, 2}, {7, 5}, {1, 3}, {6, 4}, {2, 0}, {5, 7}, {3, 1}, {4, 6}}.

[0103] 2. Quantification

[0104] In various embodiments, three types of quantization methods may be used: a baseline quantization method, a codebook-based quantization method, and a dependent scalar quantization method.

[0105] In the baseline quantization method, uniform quantization can be applied to the model parameter tensor (or parameter tensor) using a fixed step size. In one example, the fixed step size can be represented by the parameters qpDensity and qp. A flag denoted as dq_flag can be used to enable uniform quantization (e.g., dq_flag is equal to 0). The reconstructed values in the decoded tensor can be integer multiples of the step size.

[0106] In a codebook-based approach, a model parameter tensor can be represented as a tensor of indices and a codebook, where the indexed tensor has the same shape as the original tensor. The size of the codebook can be selected at the encoder and sent as a metadata parameter. The indices have integer values and can be further entropy encoded. In one example, the codebook consists of floating-point 32-bit values. The reconstructed values in the decoded tensor are the values of the codebook elements referenced by the index values of these decoded tensors.

[0107] In the dependent scalar quantization method, dependent scalar quantization can be applied to the parameter tensor using a fixed step size, such as represented by the parameters qpDensity and qp, and a state transition table of size 8. A flag denoted dq_flag equal to 1 can be used to enable dependent scalar quantization. The reconstructed values in the decoded tensor are integer multiples of the step size.

[0108] 3. Entropy Coding

[0109] To encode the quantized weight coefficients, entropy coding techniques may be used.In some embodiments, the absolute values of the quantized weight coefficients are encoded in a sequence comprising a unary sequence, which may be followed by a fixed length sequence.

[0110] In some examples, the distribution of weight coefficients in the layer generally follows a Gaussian distribution, and the proportion of weight coefficients with large values is very small, but the maximum value of the weight coefficients can be very large. In some embodiments, very small values can be encoded using unary coding, and larger values can be encoded based on Golomb coding. For example, when Golomb coding is not used, an integer parameter called maxNumNoRem is used to indicate the maximum number. When the quantized weight coefficient is not greater than (e.g., equal to or less than) maxNumNoRem, the quantized weight coefficient can be encoded by unary coding. When the quantized weight coefficient is greater than maxNumNoRem, the portion of the quantized weight coefficient equal to maxNumNoRem is encoded by unary coding, and the remainder of the quantized weight coefficient is encoded by Golomb coding. Therefore, the unary sequence includes a first part of the unary coding and a second part for encoding the exponential Golomb remainder bits.

[0111] In some embodiments, the quantized weight coefficients may be encoded by the following two steps.

[0112] In the first step, for the quantized weight coefficient, the binary syntax element sig_flag is encoded. The binary syntax element sig_flag specifies whether the quantized weight coefficient is equal to 0. If sig_flag is equal to 1 (indicating that the quantized weight coefficient is not equal to 0), the binary syntax element sign_flag is further encoded. The binary syntax element sign_flag indicates whether the quantized weight coefficient is positive or negative.

[0113] In a second step, the absolute value of the quantized weight coefficient may be encoded into a sequence comprising a unary sequence, which may be followed by a fixed-length sequence. When the absolute value of the quantized weight coefficient is equal to or less than maxNumNoRem, the sequence comprises a unary encoding of the absolute value of the quantized weight coefficient. When the absolute value of the quantized weight coefficient is greater than maxNumNoRem, the unary sequence may comprise a first portion for encoding maxNumNoRem using a unary encoding and a second portion for encoding exponential Golomb remainder bits, and the fixed-length sequence is used to encode the fixed-length remainder.

[0114] In some examples, unary coding is applied first. For example, a variable such as j is initialized to 0, and another variable X is set to j + 1. The syntax element abs_level_greater_X is encoded. In one example, when the absolute value of the quantized weight level is greater than the variable X, abs_level_greater_X is set to 1 and the unary coding continues; otherwise, abs_level_greater_X is set to 0 and the unary coding is completed. When abs_level_greater_X equals 1 and the variable j is less than maxNumNoRem, the variable j is incremented by 1, and the variable X is also incremented by 1. Then, another syntax element abs_level_greater_X is encoded. This process continues until abs_level_greater_X equals 0 or the variable j equals maxNumNoRem. When the variable j equals maxNumNoRem, the encoded bits are the first part of the unary sequence.

[0115] When abs_level_greater_X equals 1 and j equals maxNumNoRem, the encoding continues with Golomb coding. Specifically, the variable j is reset to 0, and X is set to 1 << j. The unary coding remainder can be calculated based on the absolute value of the quantized weight coefficient minus maxNumNoRem. The syntax element abs_level_greater_thanX is encoded. In one example, when the unary coding remainder is greater than the variable X, abs_level_greater_X is set to 1; otherwise, abs_level_greater_X is set to 0. If abs_level_greater_X equals 1, the variable j is incremented by 1, 1 << j is added to X, and another abs_level_greater_X is encoded. This process continues until abs_level_greater_X equals 0, thus encoding the second part of the unary sequence. When abs_level_greater_X equals 0, the unary coding remainder can be one of these values (X, X - 1,... X - (1 << j)+1). A code of length j can be used to encode an index pointing to one of the values in (X, X - 1,... X - (1 << j)+1), and this code can be referred to as a fixed-length remainder.

[0116] Figure 13 Examples for decoding the absolute value of a quantized weight coefficient according to some embodiments of the present disclosure are shown. In Figure 13In the example, QuantWeight[i] represents the quantized weight coefficient at the i-th position in the array; sig_flag specifies whether the quantized weight coefficient QuantWeight[i] is not 0 (for example, sig_flag is 0, indicating that QuantWeight[i] is 0); sign_flag specifies whether the quantized weight coefficient QuantWeight[i] is positive or negative (for example, sign_flag is 1, indicating that QuantWeight[i] is negative); abs_level_greater_x[j] indicates whether the absolute level of QuantWeight[i] is greater than j+1 (for example, the first part of the unary sequence); abs_level_greater_x2[j] includes the unary part of the exponential Golomb remainder (for example, the second part of the unary sequence); and abs_remainder indicates a fixed-length remainder.

[0117] According to one aspect of the present disclosure, a context modeling approach can be used to encode the three flags sig_flag, sign_flag, and abs_level_greater_X. Thus, flags with similar statistical behavior can be associated with the same context model, allowing the probability estimator (within the context model) to adapt to the underlying statistics.

[0118] In one example, the context modeling method uses three context models for sig_flag according to whether the left adjacent quantized weight coefficient is 0, less than 0, or greater than 0.

[0119] In another example, the context model method uses three other context models for sign_flag according to whether the left neighboring quantized weight coefficient is 0, less than 0, or greater than 0.

[0120] In another example, for each of the abs_level_greater_X flags, the context modeling method uses one or two separate context models. In one example, when X ≤ maxNumNoRem, two context models are used depending on sign_flag. In another example, when X > maxNumNoRem, only one context model is used.

[0121] 4. CTU3D and recursive CU3D block partitioning

[0122] In some embodiments, the model parameter tensor can be divided into CTU3D blocks, each of which is further divided into 3D coding unit (CU3D) blocks. CU3D can be further divided and encoded based on a pyramid tree structure. For example, the pyramid tree structure can be a 3D octree, a 3D unary tree, a 3D label tree, or a 3D unary label tree structure. After a specific training / retraining operation, the weight coefficients may have a local structure. Coding methods utilizing 3D octrees, 3D unary trees, 3D label trees, and / or 3D unary label tree structures can generate more efficient representations by utilizing the local distribution of CTU3D / CU3D blocks. These pyramid tree structure-based methods can be coordinated with baseline methods (i.e., coding methods based on non-pyramid tree structures).

[0123] Typically, for convolutional layers with a layout of [R][S][C][K], the dimension of the weight tensor (or model parameter tensor) can be 4; for fully connected layers with a layout of [C][K], the dimension can be 2; and for bias and batch norm layers, the dimension can be 1. R and S represent the convolution kernel size (width and height), C represents the input feature size, and K represents the output feature size.

[0124] In one embodiment, for convolutional layers, the 2D [R] [S] dimensions can be reshaped into 1D [RS] dimensions, so that the 4D tensor [R] [S] [C] [K] is reshaped into a 3D tensor [RS] [C] [K]. Fully connected layers are considered as a special case of a 3D tensor where R = S = 1.

[0125] In one embodiment, the 3D tensor [RS][C][K] can be divided into non-overlapping smaller blocks (CTU3D) along the [C][K] plane. Each CTU3D has a shape of [RS][ctu3d_height][ctu3d_width], where, in one example, ctu3d_height = max_ctu3d_height and ctu3d_width = max_ctu3d_width. For CTU3Ds located on the right and / or bottom of the tensor, their ctu3d_height is the remainder of C / max_ctu3d_height, and their ctu3d_width is the remainder of K / max_ctu3d_width.

[0126] In one embodiment, the values of max_ctu3d_height and max_ctu3d_width may be explicitly signaled in the bitstream or may be implicitly inferred. In one example, when max_ctu3d_height = C and max_ctu3d_width = K, block partitioning is disabled.

[0127] In one embodiment, a quadtree structure can be used to implement a simplified block structure, where a CTU3D / CU3D is recursively partitioned into smaller CU3Ds until a maximum recursion depth is reached. Starting from the CTU3D node, this quadtree of CU3D blocks can be scanned and processed using a depth-first quadtree scan order. Child nodes under the same parent node are scanned and processed using a raster scan order in either the horizontal or vertical direction.

[0128] Figure 14 Two examples of adaptive CTU3D / CU3D partitioning using raster scanning along the vertical direction are shown.

[0129] In one embodiment, for CU3Ds of a given quadtree depth, the following formula is used to calculate max_cu3d_height / max_cu3d_width for these CU3Ds.

[0130] max_cu3d_height=max_ctu3d_height>>depth

[0131] max_cu3d_width=max_ctu3d_width>>depth

[0132] The maximum recursion depth is reached when both max_cu3d_height and max_cu3d_width are less than or equal to a predetermined threshold. This threshold can be explicitly included in the bitstream or can be a predetermined number (e.g., 8) that can be implicitly inferred by the decoder. When the predetermined threshold is the size of the CTU3D, this recursive partitioning is disabled.

[0133] In one embodiment, a rate-distortion (RD)-based coding algorithm decides whether to split a parent CU3D into multiple smaller child CU3Ds. If the combined RD of these smaller child CU3Ds is less than the RD from the parent CU3D, the parent CU3D is split into multiple smaller child CU3Ds. Otherwise, the parent CU3D is not split further. A split flag is defined to record the corresponding split decision. This flag can be skipped at the last depth of the CU partition.

[0134] In one embodiment, a recursive CU3D block partitioning operation is performed based on a quadtree structure to partition a CTU3D into CU3D blocks, and a split flag is defined to record each split decision at a node in the quadtree structure.

[0135] In another embodiment, the recursive CU3D block partitioning operation is not performed in the CTU3D block, and the split flag for recording the split decision is not defined. In this case, the CU3D block is the same as the CTU3D block.

[0136] 5.3D Pyramid Tree Structure

[0137] In various embodiments, the pyramid tree structure (or 3D pyramid tree structure) may be a tree data structure in which each internal node may have eight child nodes. The 3D pyramid tree structure may be used to partition a 3D tensor (or a sub-block such as a CTU3D or CU3D) by recursively subdividing the 3D tensor into eight octants along the z, y, and x axes.

[0138] Figure 15 An exemplary partitioning process based on a 3D pyramid tree structure is shown. Figure 15 As shown, the 3D octree (1505) is a tree data structure in which each internal node (1510) has exactly 8 child nodes (1515). The 3D octree (1505) is used to partition the 3D tensor (1520) by recursively subdividing the 3D tensor (1520) into 8 octants (1525) along the z, y, and x axes. The nodes at the last depth of the 3D octree (1505) can be blocks with a size of 2×2×2 coefficients.

[0139] In various embodiments, different methods may be employed to construct a 3D pyramid tree structure to represent coefficients in a CU3D at the encoder side or the decoder side.

[0140] In one embodiment, a 3D octree for CU3D may be constructed as follows: A node value of 1 at a 3D octree position at the last depth indicates that the codebook index (in the case of a codebook encoding method) or the coefficient (in the case of a direct quantization encoding method) in the corresponding node is not 0. A node value of 0 at a 3D octree position at the bottom depth indicates that the codebook index or coefficient in the corresponding node is 0. The node value of a 3D octree position at other depths is defined as the maximum value of its 8 child nodes.

[0141] In one embodiment, a 3D singular tree for CU3D may be constructed as follows: a node value of 1 at a 3D singular tree position at a depth other than the last depth indicates that its child nodes (and the child nodes of the child nodes, including the node at the last depth) have non-uniform (different) values; a node value of 0 at a 3D singular tree position at a depth other than the last depth indicates that all its child nodes (and the child nodes of the child nodes, including the node at the last depth) have uniform (same) values.

[0142] In one embodiment, a 3D label tree for a CU3D may be constructed as follows. The node value of the 3D label tree position at the last depth indicates that the absolute value of the codebook index (in the case of a codebook encoding method) or the absolute coefficient (in the case of a direct quantization encoding method) in the corresponding CU3D is not zero. The node value of the 3D label tree position at other depths is defined as the minimum value of its eight child nodes. In another embodiment, the node value of the 3D label tree position at other depths may be defined as the maximum value of its eight child nodes.

[0143] In one embodiment, the 3D unique tag tree of CU3D is constructed by combining the 3D tag tree and the 3D unique tree.

[0144] It should be noted that for some CU3D blocks with different depths / heights / widths, there may not be enough coefficients to construct a complete 3D pyramid in which all 8 child nodes of all parent nodes are available. If a parent node does not have all 8 child nodes, scanning and encoding of these non-existent child nodes may be skipped.

[0145] 6.3D Pyramid Scanning Order

[0146] After the 3D pyramid is constructed, all nodes may be traversed using a predetermined scan order to encode node values at the encoder side or decode node values at the decoder side.

[0147] In one embodiment, starting from the top node, a depth-first search scan order can be used to scan all nodes. The scanning order of child nodes sharing the same parent node can be arbitrarily defined, for example, (0, 0, 0) -> (0, 0, 1) -> (0, 1, 0) -> (0, 1, 1) -> (1, 0, 0) -> (1, 0, 1) -> (1, 1, 0) -> (1, 1, 1).

[0148] In another embodiment, starting from the top node, a breadth-first search can be used to walk through all nodes. Because each pyramid depth is a 3D shape, the scan order within each depth can be arbitrarily defined. In one embodiment, the scan order is defined using the following pseudo code to align with the pyramid encoding method:

[0149]

[0150] In another embodiment, the encoding_start_depth syntax element may be used to indicate the first depth involved in the encoding or decoding process. When traversing all nodes using a predetermined scan order, if the depth of the node is greater than the encoding_start_depth, encoding of the current node value is skipped. Multiple CU3Ds, CTU3Ds, layers, or models can share a single encoding_start_depth. This syntax element may be explicitly signaled in the bitstream or predefined and implicitly inferred.

[0151] In one embodiment, encoding_start_depth is explicitly signaled in the bitstream. In another embodiment, encoding_start_depth is predefined and implicitly inferred. In another embodiment, encoding_start_depth is set to the last depth of the 3D pyramid tree structure and implicitly inferred.

[0152] 7.3D Pyramid Coding Method

[0153] On the decoder side, after constructing the 3D pyramid tree structure, the corresponding encoding method can be executed to walk through all nodes and encode the coefficients represented by different 3D trees. On the decoder side, corresponding to different encoding methods, the encoded coefficients can be decoded accordingly.

[0154] For a 3D octree, if the value of the parent node is 0, scanning and encoding of its child nodes (and child nodes of the child nodes) are skipped because the value of the child node should always be 0. If the value of the parent node is 1 and the values of all child nodes except the last child node are 0, the last child node can be scanned, but the encoding of the value of the last child node can be skipped because the value of the last child node should always be 1. If the current depth is the last depth of the pyramid and if the current node value is 1, the sign of the map value is encoded when the codebook method is not used, and then the map value (quantized value) itself is encoded.

[0155] For a 3D singly-branched tree, in one embodiment, the value of a given node can be encoded. If the node value is 0, the corresponding uniform value can be encoded, and the encoding of its child nodes (and children of child nodes) can be skipped because the absolute value of the child nodes should always be equal to the uniform value. The child nodes can be scanned until the bottom depth is reached, where the sign bit of each child node can be encoded if the node value is not 0.

[0156] For a 3D singular tree, in another embodiment, the value of a given node can be encoded. If the node value is 0, the corresponding uniform value can be encoded, and the encoding of its child nodes (and children of children) can be skipped because the absolute value of the child nodes should always be equal to the uniform value. After processing all nodes in the CU3D, the pyramid tree structure can be scanned again, and if the node value is not 0, the sign bit of each child node at the bottom depth can be encoded.

[0157] For a 3D label tree, if a node is a top node without a parent node, the value of the node can be encoded. For any child node, the difference between the parent node and the child node can be encoded. If the parent node's value is X and all child nodes except the last child have values greater than X, the last child node can be scanned, but encoding the value of the last child node can be skipped because the last child node should always have a value of X.

[0158] For a 3D single-branch label tree, the value of a given node from the single-branch tree can first be encoded. Then, if the node is a top node without a parent node, the label tree value can be encoded using the label tree encoding method, or the difference in label tree value between the parent node and the child node can be encoded. The node skipping method introduced in the label tree encoding section is also used. If the single-branch tree node value is 0, the scanning and encoding of its child nodes (and child nodes of child nodes) can be skipped because the value of the child node should always be equal to the uniform value.

[0159] In one embodiment, when encoding_start_depth is the last depth, the coefficient skipping methods described herein may be disabled to encode all coefficients. In one example, a syntax element may be received in the bitstream at the decoder side, indicating the starting depth in the 3D pyramid tree structure. When the starting depth is the last depth in the 3D pyramid tree structure, a non-3D pyramid tree-based decoding method may be used to decode the model parameters of the model parameter tensor from the bitstream.

[0160] In another embodiment, when encoding_start_depth is the last depth, in order to utilize the coefficient skipping methods described herein, the 3D pyramid tree can be encoded by adjusting the starting depth so that the starting depth is the second-to-last depth. In one example, a syntax element indicating the starting depth in the 3D pyramid tree structure can be received in the bitstream at the decoder. When the starting depth is the last depth in the 3D pyramid tree structure, the decoding process can start at the decoder from the second-to-last depth in the depth of the 3D pyramid tree structure.

[0161] In another embodiment, when encoding_start_depth is the last depth, for a 3D single-branch tag tree, the single-branch tree portion of the 3D pyramid tree can be encoded by adjusting encoding_start_depth so that encoding_start_depth is the second-to-last depth. The tag tree portion of the 3D pyramid tree can be encoded without adjusting encoding_start_depth. In one example, a syntax element indicating the starting depth in the 3D pyramid tree structure can be received in the decoder's bitstream. When the starting depth is the last depth in the 3D pyramid tree structure and the 3D pyramid tree structure is a single-branch tag tree structure associated with single-branch tree portion encoding and tag tree portion encoding, the decoding process can be performed at the decoder as follows. For single-branch tree portion encoding, the decoding process can start from the second-to-last depth in the depth of the 3D pyramid tree structure. For tag tree portion encoding, the decoding process can start from the last depth in the depth of the 3D pyramid tree structure.

[0162] 8. Dependence on Quantification

[0163] In some embodiments, a dependent scalar quantization method is used for neural network parameter approximation. A related entropy coding method can be used in conjunction with the quantization method. This method introduces dependencies between quantized parameter values, which reduces distortion during the parameter approximation process. Additionally, dependencies can be exploited during the entropy coding stage.

[0164] In dependent quantization, the allowed reconstruction values of neural network parameters (e.g., weight parameters) depend on the selected quantization index of the preceding neural network parameter in the reconstruction order. The main effect of this approach is that the allowed reconstruction vectors (given by all reconstructed neural network parameters of a layer) are more densely packed in the N-dimensional vector space (N represents the number of parameters in the layer) compared to traditional scalar quantization. This means that for a given average number of allowed reconstruction vectors per N-dimensional unit volume, the average distance (e.g., mean squared error (MSE) or mean absolute error (MAE) distortion) between the input vector and the closest reconstruction vector is reduced (for a typical distribution of input vectors).

[0165] In the dependent quantization process, the dependency between the reconstructed values allows the parameters to be reconstructed in a scanning order (the scanning order is the same as the order in which the parameters are entropy decoded). The dependent scalar quantization method can then be implemented by defining two scalar quantizers with different reconstruction levels and defining a procedure for switching between the two scalar quantizers. Thus, for each parameter, there may be two available scalar quantizers, such as Figure 16 shown.

[0166] Figure 16 Two scalar quantizers used according to an embodiment of the present disclosure are shown. The first quantizer Q0 maps the neural network parameter levels (numbers from -4 to 4 below the dot) to even multiples of the quantization step size Δ. The second quantizer Q1 maps the neural network parameter levels (numbers from -5 to 5) to odd multiples of the quantization step size Δ or to 0.

[0167] For quantizers Q0 and Q1, the positions of the available reconstruction levels are uniquely specified by the quantization step size Δ. The two scalar quantizers Q0 and Q1 are characterized as follows:

[0168] Q0: The reconstruction level of the first quantizer Q0 is given by an even multiple of the quantization step size Δ. When using this quantizer, the reconstructed neural network parameter t is calculated according to the following formula ′ :

[0169] t ′ =2·k·Δ,

[0170] where k represents the associated parameter level (the quantization index sent).

[0171] Q1: The reconstruction level of the second quantizer Q1 is given by an odd multiple of the quantization step size Δ and a reconstruction level equal to zero. Neural network parameter level k to reconstruction parameter t ′ The mapping is specified by:

[0172] t ′ =(2·k-sgn(k))·Δ,

[0173] where sgn(·) represents the sign function

[0174]

[0175] Instead of explicitly signaling the quantizer (Q0 or Q1) used by the current weight parameter in the bitstream, the quantizer is determined by the parity of the weight parameter level that precedes the current weight parameter in the encoding / reconstruction order. Switching between quantizers is achieved by a state machine represented by Table 1. The state has 8 possible values (0, 1, 2, 3, 4, 5, 6, 7) and is uniquely determined by the parity of the weight parameter level that precedes the current weight parameter in the encoding / reconstruction order. For each layer, the state variable is initially set to 0. When the weight parameters are reconstructed, the state is then updated according to Table 1, where k represents the value of the transform coefficient level. The next state depends on the current state and the parity of the current weight parameter level k (k & 1). Therefore, the state update can be obtained by the following formula:

[0176] state = sttab[state][k&1]

[0177] Where sttab represents Table 1.

[0178] Table 1 shows a state transition table for determining a scalar quantizer for a neural network parameter, where k represents the value of the neural network parameter:

[0179] Table 1

[0180]

[0181] The state uniquely specifies the scalar quantizer to use. If the state value of the current weight parameter is an even number (0, 2, 4, 6), the scalar quantizer Q0 is used. Otherwise, if the state value is an odd number (1, 3, 5, 7), the scalar quantizer Q1 is used.

[0182] In some embodiments, a baseline encoding method (in which an encoding / decoding method based on a 3D pyramid tree structure is not used) may be employed. In the baseline encoding method, all coefficients of the model parameter tensor may be scanned and entropy encoded according to a scan order. For a dependent quantization process used in combination with the baseline encoding method, the coefficients may be reconstructed in the scan order (the scan order is the same as the order in which the coefficients are entropy decoded).

[0183] Due to the nature of the 3D pyramid coding method described herein, certain coefficients in the model parameter tensor may be skipped from the entropy coding process. Therefore, in one embodiment, when using the 3D pyramid coding method, the dependent quantization process (which operates on all coefficients in the model parameter tensor) may be disabled.

[0184] In another embodiment, when using a 3D pyramid coding method, a dependent quantization process may be enabled. For example, the dependent quantization construction process may be modified so that if these coefficients are skipped from the entropy coding process, these coefficients may be excluded from the construction process of the dependent quantization coefficients. In one example, when one or more syntax elements indicate that a CU3D is partitioned based on a 3D pyramid tree structure, a dependent quantization construction process may be performed on the CU3D. Model parameters of the CU3D that were skipped during the encoding process based on the 3D pyramid tree structure are excluded from the dependent quantization construction process.

[0185] In another embodiment, the absolute values of the coefficients are used in dependent quantization.

[0186] 9. Entropy Coding Context

[0187] In some embodiments, when dependency quantization is not used, context modeling may be performed as follows.

[0188] For the octree node value represented as Oct_flag and the symbol represented as sign in the 3D octree-based coding method, the context model index represented as ctx can be determined according to the following formula:

[0189] Oct_flag:

[0190] int p0=(z>=1)? oct[d][z-1][y][x]: 0;

[0191] int p1=(z>=2)? oct[d][z-2][y][x]: 0;

[0192] int ctx=(p0==p1)? ! ! p0:2;

[0193] sign:

[0194] int p0=(z>=1)? map[z-1][y][x]: 0;

[0195] int ctx=(p0==0)? 0: (p0<0)? 1:2.

[0196] In the above calculation, oct[d][z][y][x] represents the node value at position [z][y][x] at depth d; map[z][y][x] represents the quantized value at position [z][y][x].

[0197] For a non-zero flag denoted as nz_flag and a symbol denoted as sign in a 3D singular tree-based coding method, a context model index denoted as ctx may be determined according to the following formula:

[0198] nz_flag:

[0199] int p0=(last_depth&&map_z>=1)? std::abs(map[map_z

[0200] -1][map_y][map_x]): 0;

[0201] int ctx=(p0==0)? 0:1;

[0202] sign:

[0203] int p0=(z>=1)? map[z-1][y][x]: 0;

[0204] int ctx=(p0==0)? 0: (p0<0)? 1:2.

[0205] For the 3D label tree based coding method, the context model index denoted as ctx with a non-zero flag denoted as nz_flag and a sign denoted as sign can be determined according to the following formula:

[0206] nz_flag:

[0207] int p0=(z>=1)? tgt[d][z-1][y][x]: 0;

[0208] int p1=(z>=2)? tgt[d][z-2][y][x]: 0;

[0209] int ctx=(p0==p1)? ! ! p0:2;

[0210] sign:

[0211] int p0=(z>=1)? tgt[d][z-1][y][x]: 0;

[0212] int ctx=(p0==0)? 0: (p0<0)? 1:2.

[0213] where tgt[d][z][y][x] represents the value of the node at position [z][y][x] at depth d.

[0214] For a 3D single-branch label tree, the context model index denoted as ctx with a non-zero flag denoted as nz_flag and a sign denoted as sign can be determined according to the following formula:

[0215] nz_flag:

[0216] int p0=(z>=1)? tgt[d][z-1][y][x]: 0;

[0217] int p1=(z>=2)? tgt[d][z-2][y][x]: 0;

[0218] int ctx=(p0==p1)? ! ! p0:2;

[0219] sign:

[0220] int p0=(z>=1)? map[z-1][y][x]: 0;

[0221] int ctx=(p0==0)? 0: (p0<0)? 1:2.

[0222] In one example, when dependent quantization is used, the context modeling of nz_flag may be adjusted such that ctx=ctx+3*state_id.

[0223] 10. Syntax Cleanup

[0224] All coefficients in a CU3D may be made uniform. In one embodiment, a uaflag may be defined in the CU3D header to indicate whether all coefficients in the CU3D are uniform. In one example, a value of uaflag = 1 indicates that all coefficients in the CU3D are uniform.

[0225] In one embodiment, ctu3d_map_mode_flag may be defined to indicate whether all CU3D blocks in a CTU3D share the same map_mode. If ctu3d_map_mode_flag = 1, map_mode is signaled. Note that this flag may also be inferred implicitly (to 0).

[0226] In one embodiment, enable_start_depth may be defined to indicate whether cu3d encoding may start from a depth other than the bottom depth. If enable_start_depth = 1, start_depth (or encoding_start_depth) is signaled. Note that this flag may also be inferred implicitly (to 1).

[0227] In one embodiment, an enable_zdep_reorder flag may be defined to indicate whether zdep_array reordering is allowed. Note that this flag may also be inferred implicitly (to 0).

[0228] 11. Coordination of Baseline Coding Method and Pyramid Coding Method

[0229] In the baseline method, the weight tensor is reshaped into a 2D matrix with a shape of [output_channel][input_channel * kernel size]. Coefficients in the same kernel are stored in consecutive memory locations. When calculating the context of sig_flag and sign_flag, the adjacent coefficient is defined as the last coefficient processed before the current coefficient. For example, the adjacent coefficient of the first coefficient in a kernel is the last coefficient in the previous kernel.

[0230] In one embodiment, to calculate the context model index, for the first coefficient in a kernel, the values of its neighboring coefficients are set to 0, rather than the value of the last coefficient in the previous kernel being set to 0.

[0231] In one embodiment, during the 3D pyramid coding process, if the quantization mode is not a codebook mode and if start_depth is the last pyramid depth, a baseline coding method is selected, and its RD is calculated and compared with other modes (e.g., coding methods based on a 3D pyramid tree structure).

[0232] In one embodiment, if ctu3d_width and / or ctu3d_height is 1, the baseline encoding method is automatically selected.

[0233] 12.3D Pyramid Coding Syntax Table

[0234] In Appendix B of the present disclosure, syntax tables 2 to 17 are listed as examples of the encoding method based on the 3D pyramid tree structure disclosed herein. Syntax elements introduced in the listed syntax tables are defined at the end of each corresponding syntax table.

[0235] 13. Unified Encoding Method for Neural Network Model Compression

[0236] In some embodiments, a uniform-based encoding method may be used. A layer_uniform_flag flag may be defined for convolutional and fully connected layers to indicate whether the layer is encoded using a 3D pyramid tree-based encoding method. In one example, if the layer_uniform_flag flag is equal to a first value (e.g., 0), the layer is encoded using a baseline method.

[0237] If layer_uniform_flag is equal to the second value (e.g., 1), a coding method based on a 3D pyramid tree structure may be used. For example, the layer may be reshaped into a CTU3D layout. For each CTU3D, a ctu3d_uniform_flag flag may be defined to indicate whether all child nodes sharing the same parent node at the bottom depth are uniform (nodes that do not share the same parent node may have different uniform values).

[0238] If the ctu3d_uniform_flag flag is equal to the first value (e.g., 1) for this CTU3D, all child nodes sharing the same parent node at the bottom depth are unified (nodes that do not share the same parent node may have different uniform values). In one embodiment, a 3D unary tree encoding method may be used to encode this CTU3D. encoding_start_depth is set to the last depth of the 3D pyramid tree structure (e.g., associated with a CU3D or CTU3D) and is implicitly inferred. Encoding of the uniform value of a node may be skipped because the uniform value of the node should always be 0.

[0239] In one embodiment, a uniform value may be encoded for all child nodes that share the same parent node at the bottom depth, and then the sign bits of these child nodes may be encoded if the node value is not 0. In another embodiment, a uniform value may be encoded for all child nodes that share the same parent node at the bottom depth, and after processing all nodes in the CU3D, if the node value is not 0, the pyramid (pyramid tree structure) may be scanned again to encode the sign bit of each child node at the bottom depth.

[0240] In one embodiment, if the ctu3d_uniform_flag flag is equal to the second value (e.g., 0), the CTU3D may be encoded using a 3D tag tree encoding method. encoding_start_depth is set to the last depth of the 3D pyramid tree structure (e.g., associated with a CU3D or CTU3D) and is implicitly inferred.

[0241] In one embodiment, the value of each child node may be encoded based on a predetermined scan order. In another embodiment, the absolute value of each child node may be encoded based on a predetermined scan order, and then the sign bit of each child node may be encoded. In another embodiment, the absolute values of all child nodes may be encoded based on a predetermined scan order. After processing all nodes in the CU3D, if the node value is not 0, the sign bits of all child nodes may be encoded.

[0242] 14. Syntax table based on unified encoding

[0243] In Appendix C of the present disclosure, syntax tables 18 to 21 are listed as examples of the unified coding method disclosed herein. Syntax elements introduced in the listed syntax tables are defined at the end of each corresponding syntax table.

[0244] IV. Example of the Coding Process

[0245] Figure 17 The decoding process (1700) of CTU partitioning according to one embodiment of the present disclosure is shown. The process (1700) may start from (S1701) and proceed to (S1710).

[0246] At (S1710), a first syntax element in an NNR aggregation unit header of an NNR aggregation unit may be received from a bitstream. The first syntax element may indicate a CTU scanning order for processing a model parameter tensor transmitted in the NNR aggregation unit. For example, a first value of the first syntax element may indicate that the CTU scanning order is a first raster scanning order along a horizontal direction. A second value of the first syntax element may indicate that the CTU scanning order is a second raster scanning order along a vertical direction.

[0247] In one example, another syntax element may be received in advance to control whether CTU block partitioning is enabled for tensors in the NNR aggregation unit. For example, the other syntax element may be a model-related syntax element or a tensor-related syntax element, the model-related syntax element is used to specify whether CTU block partitioning is enabled for a layer of a neural network, and the tensor-related syntax element is used to specify whether CTU block partitioning is enabled for a tensor in the NNR aggregation unit.

[0248] At (S1720), the tensors in the NNR aggregation unit may be reconstructed based on the CTU scanning order. When CTU block partitioning is enabled for the tensors in the NNR aggregation unit, the tensors may be divided into CTUs on the encoder side. The CTUs may be scanned and encoded according to the scanning order indicated by the first syntax element. On the decoder side, based on the indicated scanning order, the decoder may understand the order in which the CTUs were encoded and, accordingly, organize the decoded CTUs into tensors. The process (1700) may proceed to (S1799) and terminate at (S1799).

[0249] Figure 18 The decoding process (1800) of 3D pyramid coding according to one embodiment of the present disclosure is shown. The process (1800) may start from (S1801) and proceed to (S1810).

[0250] At (S1810), one or more first syntax elements may be received from a bitstream represented by a compressed neural network. The one or more first syntax elements may be associated with a CU3D partitioned from a CTU3D. The CTU3D may be obtained by partitioning a tensor in a neural network. The one or more first syntax elements may indicate that the CU3D is partitioned based on a coding mode corresponding to a 3D pyramid tree structure. The 3D pyramid tree structure may include multiple depths. Each depth corresponds to one or more nodes. Each node has a node value. For example, the 3D pyramid tree structure may be one of an octree structure, a single-branch tree structure, a tag tree structure, a single-branch tag tree structure, and the like.

[0251] At (S1820), a second sequence of syntax elements corresponding to node values of each node in the 3D pyramid tree structure may be received from the bitstream in a breadth-first scan order for scanning the nodes in the 3D pyramid tree structure. Thus, at the decoder side, the node values (represented by syntax elements) may be received based on a depth-first scan order. In other embodiments (not implementing process (1800)), the 3D pyramid tree structure may be scanned at the encoder side according to a depth-first scan order.

[0252] At (S1830), the model parameters of the tensor may be reconstructed based on the received second syntax elements corresponding to the node values of the nodes in the 3D pyramid tree structure. As described above, at the decoder, corresponding to an octree structure, a single-tree structure, a tag tree structure, or a single-tag tree structure, a 3D pyramid coding method may be used to encode the node values of the 3D pyramid tree structure and the coefficient values of the CU3D partitioned using the 3D pyramid tree structure. At the decoder, the node values and coefficient values may be reconstructed accordingly, corresponding to the 3D pyramid coding method used. The process may proceed to (S1899) and terminate at (S1899).

[0253] Figure 19 A decoding process (1900) based on unified encoding according to one embodiment of the present disclosure is shown. The process (1900) may start from (S1901) and proceed to (S1910).

[0254] At (S1910), a syntax element associated with a CTU3D may be received. The CTU3D may be derived from tensor partitioning in a layer of the neural network in a bitstream of a compressed neural network representation of the neural network. The syntax element may indicate whether all child nodes at the bottom depth of the pyramid tree structure associated with the CTU3D are unified. Child nodes that do not share the same parent node at the bottom depth may have different unified values.

[0255] At (S1920), in response to the syntax element indicating that all child nodes at the bottom depth of the pyramid tree structure associated with the CTU3D are unified, the CTU3D may be decoded based on a 3D unary tree encoding method. In one example, the starting depth of the 3D unary tree encoding method may be inferred to be the bottom depth of the pyramid tree structure. In one embodiment, a unification flag for nodes at the bottom depth of the pyramid tree structure is not encoded in the bitstream.

[0256] In one embodiment, a uniform value encoded in a bitstream for all child nodes that share the same parent node at the bottom depth may be received. A sign bit for all child nodes that share the same parent node at the bottom depth may be received. The sign bit follows the uniform value in the bitstream.

[0257] In one embodiment, a uniform value may be received for each group of child nodes that share the same parent node at the bottom depth. Sign bits of child nodes in each group of child nodes that share the same parent node at the bottom depth may then be received.

[0258] At (S1930), in response to the syntax element indicating that all child nodes at the bottom depth of the pyramid tree structure associated with the CTU3D are not uniform, the CTU3D may be decoded based on a 3D tag tree encoding method. In one example, the starting depth of the 3D tag tree encoding method may be inferred to be the bottom depth of the pyramid tree structure.

[0259] In various embodiments, the values of nodes at the bottom depth of the pyramid tree structure may be decoded according to one of the following methods: In a first method, the values of nodes at the bottom depth of the pyramid tree structure may be received, and the value of each node may be encoded in a bitstream based on a predetermined scanning order.

[0260] In a second method, the absolute value of each node at the bottom depth of the pyramid tree structure may be received in a bitstream based on a predetermined scan order, and then (if the absolute value is not zero) the sign of each node may be received. In a third method, the absolute value of each node at the bottom depth of the pyramid tree structure may be received in a bitstream based on a predetermined scan order, and then (if the node has a non-zero value) the sign of each node at the bottom depth of the pyramid tree structure may be received in the bitstream based on a predetermined scan order. Process (1900) may proceed to (S1999) and terminate at (S1999).

[0261] V.Computer Systems

[0262] The above technology can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media. Figure 20 A computer system (2000) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0263] Computer software may be encoded using any suitable machine code or computer language that may be subjected to assembly, compilation, linking, or similar mechanisms to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpretation, microcode execution, etc.

[0264] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[0265] Figure 20The components of the computer system (2000) shown are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Neither should the configuration of the components be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary embodiment of the computer system (2000).

[0266] The computer system (2000) may include certain human interface input devices. Such human interface input devices may be responsive to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., speech, hand clapping), visual input (e.g., gestures), and olfactory input (not depicted). The human interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images from a still camera), and video (e.g., 2D video, 3D video including stereoscopic video).

[0267] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard (2001), mouse (2002), touchpad (2003), touch screen (2010), data gloves (not shown), joystick (2005), microphone (2006), scanner (2007), camera (2008).

[0268] The computer system (2000) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback from a touch screen (2010), a data glove (not shown), or a joystick (2005), but may also be a tactile feedback device that is not an input device), audio output devices (e.g., speakers (2009), headphones (not depicted)), visual output devices (e.g., screens (2010) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which are capable of outputting 2D visual output or output beyond 3D through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)), and printers (not depicted).

[0269] The computer system (2000) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2020) including CD / DVD etc. media (2021), thumb drives (2022), removable hard drives or solid-state drives (2023), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), etc.

[0270] Those skilled in the art will also understand that the term "computer-readable medium" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0271] The computer system (2000) may also include an interface to one or more communication networks. The network may be, for example, a wireless network, a wired network, an optical network. The network may further be a local area network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter connected to some general data port or peripheral bus (2049) (e.g., a USB port of the computer system (2000)); as described below, other network interfaces are typically integrated into the core of the computer system (2000) by attaching to the system bus (e.g., an Ethernet interface in a PC computer system or a cellular network interface in a smartphone computer system). The computer system (2000) can use any of these networks to communicate with other entities. Such communications can be one-way receive only (e.g., broadcast television), one-way send only (e.g., a CANbus connected to certain CANbus devices), or bidirectional, for example, connecting to other computer systems using a local area network or wide area network digital network. As described above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

[0272] The above-mentioned human interface device, human-accessible storage device and network interface may be attached to the kernel (2040) of the computer system (2000).

[0273] The core (2040) may include one or more central processing units (CPUs) (2041), graphics processing units (GPUs) (2042), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (2043), hardware accelerators (2044) for certain tasks, and the like. These devices, as well as read-only memory (ROM) (2045), random access memory (2046), and internal mass storage (2047) such as internal non-user accessible hard drives, SSDs, and the like, may be connected via a system bus (2048). In some computer systems, the system bus (2048) may be accessed in the form of one or more physical plugs to enable expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus (2048) or to the core's system bus (2048) via a peripheral bus (2049). Peripheral bus architectures include PCI, USB, and the like.

[0274] The CPU (2041), GPU (2042), FPGA (2043), and accelerator (2044) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (2045) or RAM (2046). Transient data can also be stored in RAM (2046), while permanent data can be stored, for example, in internal mass storage (2047). Fast storage and retrieval to any storage device can be achieved by using a cache, which can be closely associated with the following: one or more CPUs (2041), GPUs (2042), mass storage (2047), ROM (2045), RAM (2046), etc.

[0275] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0276] As a non-limiting example, a computer system having the architecture (2000), particularly the kernel (2040), may provide functionality due to one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage as described above, as well as certain non-transitory memories of the kernel (2040), such as kernel internal mass storage (2047) or ROM (2045). Software implementing the various embodiments of the present disclosure may be stored in such devices and executed by the kernel (2040). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software may cause the kernel (2040), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (2046) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality due to logic hardwired or otherwise embodied in circuitry (e.g., accelerator (2044)), which may replace software or operate in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0277] Although the present disclosure has described a number of exemplary embodiments, there are modifications, permutations, and various alternative equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to devise many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and therefore fall within the spirit and scope of the present disclosure.

[0278] Appendix A: Acronyms

[0279] DNN: Deep Neural Network

[0280] NNR: Compressed Neural Network Representation; Encoded Representation of Neural Networks

[0281] CTU: Coding Tree Unit

[0282] CTU3D: 3D Coding Tree Unit

[0283] CU: Coding Unit

[0284] CU3D: 3D Coding Unit

[0285] RD: rate distortion

[0286] Appendix B: Syntax table of pyramid encoding method

[0287] Table 2

[0288]

[0289]

[0290] Table 3

[0291]

[0292] total_layer Total number of layers (can be removed if the model structure is available)

[0293] enable_escape_reorder 0 indicates that escape reordering is not allowed, 1 indicates that escape reordering is allowed

[0294] enable_max_ctu3d_size 0 indicates that ctu3d_height / ctu3d_width remain unchanged regardless of kernel size, 1 indicates that ctu3d_height / ctu3d_width are scaled based on kernel size

[0295] max_ctu3d_idx max_ctu3d_size(max_ctu3d_idx==0)? 64: (max_ctu3d_idx==1)? 32: (max_ctu3d_idx==2)? 16:8

[0296] Table 4

[0297]

[0298]

[0299]

[0300] total_layer_minus_one total_layer=1+total_layer_minus_one

[0301] stepsize sub-layer step size

[0302] bitdepth 0max bit depth of the quantized coefficients of the sub-layer

[0303] dim_minus_one1 The first sublayer dim=1+dim_minus_one1

[0304] ly_shape1 The shape of the first sublayer

[0305] include_bias 0 indicates that there is no bias tensor in this layer, 1 indicates that there is a bias tensor in this layer

[0306] include_scaling 0 indicates that there is no scaling tensor in this layer, 1 indicates that there is a scaling tensor in this layer

[0307] dim_minus_one2 sub-layer dim=1+dim_minus_one2

[0308] ly_shape2 The shape of the sublayer

[0309] scan_order 0 indicates the raster scan order along the horizontal direction, 1 indicates the raster scan order along the vertical direction

[0310] Table 5

[0311]

[0312] nzflag non-zero flag of quantized coefficients

[0313] sign The sign bit of the quantized coefficient

[0314] Table 6

[0315]

[0316] Table 7

[0317]

[0318]

[0319] split_flag: A flag indicating whether the parent CU3D is split into four smaller child CU3Ds

[0320] start_depth_delta start_depth=total_depth-1-start_depth_delta

[0321] cbook_esc_mode 0 indicates that escapes are not reordered, 1 indicates that escapes are reordered

[0322] map_mode 0 indicates the selection of octree 3d / single tree 3d method, 1 indicates the selection of label tree 3d / single label tree 3d method

[0323] uni_mode 0 indicates not to select the unitree 3d method, 1 indicates to select the unitree 3d method

[0324] tgt_mode 0 indicates not to select the tag tree 3d method, 1 indicates to select the tag tree 3d method

[0325] uaflag 0 indicates that all coefficients are not uniform, 1 indicates that all coefficients are uniform

[0326] Table 8

[0327]

[0328] sign predicted_size-prev_predicted_size sign bit predicted_flag 0 indicates that position n is not a predicted entry, 1 indicates that position n is a predicted entry

[0329] Table 9

[0330]

[0331] signalled_size Codebook size represented by the signal

[0332] nzflag_delta Non-zero flag of Δ

[0333] sign_delta Δ sign bit

[0334] sign The sign bit of the codebook entry

[0335] Table 10

[0336]

[0337]

[0338] nzflag non-zero flag of quantized coefficients

[0339] sign The sign bit of the quantized coefficient

[0340] Table 11

[0341]

[0342]

[0343]

[0344] uni_flag unidirectional tree node value

[0345] nzflag non-zero flag of quantized coefficients

[0346] sign The sign bit of the quantized coefficient

[0347] Table 12

[0348]

[0349]

[0350] oct_flag octree node value

[0351] sign The sign bit of the quantized coefficient

[0352] Table 13

[0353]

[0354]

[0355] nz_flag non-zero flag of the node value

[0356] sign The sign bit of the quantized coefficient

[0357] Table 14

[0358]

[0359]

[0360] uni_flag unidirectional tree node value

[0361] nz_flag non-zero flag of the node value

[0362] sign The sign bit of the quantized coefficient

[0363] Table 15

[0364]

[0365]

[0366] nz_flag non-zero flag of the node value

[0367] sign The sign bit of the quantized coefficient

[0368] Table 16

[0369]

[0370]

[0371] nz_flag non-zero flag

[0372] nzflag non-zero flag

[0373] uiBit_pre Unary part of the exponential Golomb remainder

[0374] uiBit EGK's bit

[0375] Table 17

[0376]

[0377]

[0378] nzflag non-zero flag

[0379] uiBit Unary part of the exponential Golomb remainder

[0380] uiBits fixed length remainder

[0381] Appendix C: Syntax table based on unified encoding

[0382] Table 18

[0383]

[0384] Ndim(arrayName[]) returns the number of dimensions of arrayName[].

[0385] scan_order specifies the block scan order for parameters with more than one dimension according to the following table:

[0386] 0: Do not scan blocks

[0387] 1: 8×8 blocks

[0388] 2: 16×16 blocks

[0389] 3: 32×32 blocks

[0390] 4: 64×64 blocks

[0391] layer_uniform_flag specifies whether to use a uniform method to encode the quantization weights QuantParam[]. layer_uniform_flag equal to 1 indicates that the uniform method is used to encode QuantParam[].

[0392] Table 19

[0393]

[0394] The 2D integer array StateTransTab[][] specifies the state transition table used for scalar-dependent quantization, as follows:

[0395] StateTransTab[][]={{0, 2}, {7, 5}, {1, 3}, {6, 4}, {2, 0}, {5, 7}, {3, 1}, {4, 6}} Table 20

[0396]

[0397]

[0398] ctu3d_uniform_flag specifies whether to use the uniform method to encode the quantized CTU3D weight QuantParam[]. ctu3d_uniform_flag equal to 1 indicates that the uniform method is used to encode QuantParam[].

[0399] sign_flag specifies whether the quantization weight QuantParam[i] is positive or negative. sign_flag equal to 1 indicates that QuantParam[i] is negative.

[0400] Table 21

[0401]

[0402] sig_flag specifies whether the quantization weight QuantParam[i] is non-zero. sig_flag equal to 0 indicates that QuantParam[i] is zero. sign_flag specifies whether the quantization weight QuantParam[i] is positive or negative. sign_flag equal to 1 indicates that QuantParam[i] is negative. abs_level_greater_x[j] indicates whether the absolute level of QuantParam[i] is greater than j+1. abs_level_greater_x2[j] contains the unary part of the exponential golomb remainder.

[0403] abs_remainder indicates the fixed-length remainder.

Claims

1. A method for neural network decoding, characterized in that: The method comprises: receiving, from a bitstream of a compressed neural network representation (NNR), a first syntax element in a model parameter set, the first syntax element indicating whether coding tree unit (CTU) block partitioning is enabled for a tensor in an NNR aggregation unit; and Reconstruct a tensor in the NNR aggregation unit based on the first syntax element.

2. The method according to claim 1, characterized in that The first syntax element is a model-related syntax element or a tensor-related syntax element, the model-related syntax element is used to specify whether the CTU block partitioning is enabled for the layer of the neural network, and the tensor-related syntax element is used to specify whether the CTU block partitioning is enabled for the tensor in the NNR aggregation unit.

3. The method according to claim 1, characterized in that The method further comprises: A second syntax element is received from the bitstream, the second syntax element indicating a coding tree unit (CTU) scanning order for processing tensors in the NNR aggregation unit.

4. The method according to claim 3, characterized in that A first value of the second syntax element indicates that the CTU scanning order is a first raster scanning order along a horizontal direction, and a second value of the second syntax element indicates that the CTU scanning order is a second raster scanning order along a vertical direction.

5. The method according to claim 1, wherein The method further comprises: A third syntax element in an NNR aggregation unit header of the NNR aggregation unit is received from the bitstream, the third syntax element indicating a maximum bit depth of quantized coefficients of tensors in the NNR aggregation unit.

6. The method according to claim 1, characterized in that The method further comprises: A fourth syntax element related to the model or the tensor is received, the fourth syntax element indicating a CTU dimension of the tensor in the NNR aggregation unit.

7. The method according to claim 1, characterized in that The method further comprises: An NNR unit is received before any NNR aggregation unit is received, the NNR unit including a fifth syntax element indicating whether CTU partitioning is enabled.

8. A method for neural network encoding, characterized in that The method comprises: encoding, in an NNR aggregation unit header of the compressed neural network representation NNR, a first syntax element in a model parameter set, the first syntax element indicating whether coding tree unit (CTU) block partitioning is enabled for tensors in the NNR aggregation unit; and A bitstream is generated that includes a compressed NNR of the encoded first syntax element.

9. The method according to claim 8, characterized in that The first syntax element is a model-related syntax element or a tensor-related syntax element, the model-related syntax element is used to specify whether the CTU block partitioning is enabled for the layer of the neural network, and the tensor-related syntax element is used to specify whether the CTU block partitioning is enabled for the tensor in the NNR aggregation unit.

10. The method according to claim 8, characterized in that The method further comprises: A second syntax element in the NNR aggregation unit header is encoded in the bitstream, the second syntax element indicating a coding tree unit (CTU) scanning order for processing tensors in the NNR aggregation unit.

11. The method according to claim 10, characterized in that A first value of the second syntax element indicates that the CTU scanning order is a first raster scanning order along a horizontal direction, and a second value of the second syntax element indicates that the CTU scanning order is a second raster scanning order along a vertical direction.

12. The method according to claim 8, characterized in that The method further comprises: A third syntax element is encoded in an NNR aggregation unit header of a compressed neural network representation (NNR), the third syntax element indicating a maximum bit depth of quantized coefficients of tensors in the NNR aggregation unit.

13. The method according to claim 8, characterized in that The method further comprises: A fourth syntax element related to the model or the tensor is encoded, wherein the fourth syntax element indicates a CTU dimension of the tensor in the NNR aggregation unit.

14. The method according to claim 8, characterized in that The method further comprises: An NNR unit is encoded before encoding any NNR aggregation unit, the NNR unit including a fifth syntax element indicating whether CTU partitioning is enabled.

15. A method for processing visual media data, characterized in that The method comprises: A bitstream of visual media data is processed according to a format rule, wherein the bitstream is a bitstream of a compressed neural network representation (NNR), the bitstream includes a first syntax element in a model parameter set, the first syntax element indicating whether coding tree unit (CTU) block partitioning is enabled for a tensor in an NNR aggregation unit; and the format rule indicates that the tensor in the NNR aggregation unit is reconstructed based on the first syntax element.

16. The method according to claim 15, characterized in that The first syntax element is a model-related syntax element or a tensor-related syntax element, the model-related syntax element is used to specify whether the CTU block partitioning is enabled for the layer of the neural network, and the tensor-related syntax element is used to specify whether the CTU block partitioning is enabled for the tensor in the NNR aggregation unit.

17. The method according to claim 15, characterized in that The method further comprises: A second syntax element in the NNR aggregation unit header is received from the bitstream, the second syntax element indicating a coding tree unit (CTU) scanning order for processing tensors in the NNR aggregation unit.

18. The method according to claim 17, wherein: A first value of the second syntax element indicates that the CTU scanning order is a first raster scanning order along a horizontal direction, and a second value of the second syntax element indicates that the CTU scanning order is a second raster scanning order along a vertical direction.

19. The method according to claim 15, characterized in that The method further comprises: A third syntax element in an NNR aggregation unit header of the NNR aggregation unit is received from the bitstream, the third syntax element indicating a maximum bit depth of quantized coefficients of tensors in the NNR aggregation unit.

20. The method according to claim 15, wherein The method further comprises: A fourth syntax element related to the model or the tensor is received, the fourth syntax element indicating a CTU dimension of the tensor in the NNR aggregation unit.

21. The method according to claim 15, wherein The method further comprises: An NNR unit is received before any NNR aggregation unit is received, the NNR unit including a fifth syntax element indicating whether CTU partitioning is enabled.

22. A neural network decoding device, characterized in that: include: A first processing circuit is configured to: Receiving, from a bitstream of a compressed neural network representation (NNR), a first syntax element in a model parameter set, the first syntax element indicating whether coding tree unit (CTU) block partitioning is enabled for a tensor in an NNR aggregation unit; as well as Reconstruct a tensor in the NNR aggregation unit based on the first syntax element.

23. A neural network encoding device, characterized in that: include: A second processing circuit, wherein the second processing circuit is configured to: encoding, in an NNR aggregation unit header of the compressed neural network representation NNR, a first syntax element in a model parameter set, the first syntax element indicating whether coding tree unit (CTU) block partitioning is enabled for tensors in the NNR aggregation unit; as well as A bitstream is generated that includes a compressed NNR of the encoded first syntax element.

24. A non-transitory computer-readable storage medium, characterized in that Computer-readable instructions are stored, and the computer-readable instructions are executed by a processor, so that the processor performs the method of neural network decoding according to any one of claims 1 to 7, or performs the method of neural network encoding according to any one of claims 8 to 14, or performs the method of processing visual media data according to any one of claims 15 to 21.

25. A method for storing or transmitting a bit stream of visual media data, characterized in that: Execute the neural network encoding method described in any one of claims 8 to 14 to generate the bit stream, or execute the neural network decoding method described in any one of claims 1 to 7 to decode the bit stream.

Images