Multidistribution Entropy Modeling of Latency Features in Image and Video Coding Using Neural Networks

Abstract

A method, system, and bitstream syntax are described for entropy modeling of latent features in image and video coding using a combination of probability density functions. Using high-level syntax elements, an encoder can signal a multi-distribution entropy model to a compliant decoder using syntax elements indicating the number of one or more PDFs used, an identifier for each PDF used in a list of available PDFs, the number of PDF parameters for each PDF, and which PDF parameters are shared between two or more PDFs used.

Description

[Technical field]

[0001] [Related Applications] This application claims the priority benefit of Indian Provisional Patent Application No. 202141013582, filed March 26, 2021, U.S. Provisional Patent Application No. 63 / 211,793, filed June 17, 2021, and European Patent Application No. 21180014.9, filed June 17, 2021, which are hereby incorporated by reference in their entireties.

[0002] [Technical field] FIELD OF THE DISCLOSURE This disclosure relates generally to images. More specifically, embodiments of the present invention relate to multi-distribution entropy modeling of latent features in image and video coding using neural networks. [Background technology]

[0003] In 2020, the MPEG group of the International Organization for Standardization (ISO), in collaboration with the International Telecommunications Union (ITU), released the first version of the Versatile Video coding Standard (VVC), also known as H.266. More recently, the same group has been working on developing a next-generation coding standard that will improve coding performance over existing video coding techniques. As part of this research, coding techniques based on artificial intelligence and deep learning are also being considered. As used herein, the term "deep learning" refers to neural networks with at least three layers, and preferably more than three layers.

[0004] As recognized by the inventors, improved techniques for neural network based image and video coding are described herein.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise noted, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, problems identified with one or more approaches should not be assumed to have been recognized in any prior art under this section unless specifically indicated. [Brief description of the drawings]

[0006] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the accompanying drawings, in which like reference symbols represent similar elements and in which:

[0007] [Figure 1] 1 shows an exemplary neural network model for image and video coding based on probabilistic modeling of latent features according to the prior art.

[0008] [Diagram 2] 1 illustrates an exemplary neural network model for image and video coding based on probabilistic modeling of latent features in accordance with an embodiment of the present invention.

[0009] [Diagram 3] 1 illustrates an exemplary neural network network for generating parameters of a multi-distribution probability density function in accordance with an embodiment of the present invention.

[0010] [Figure 4] Examples of Laplace probability density functions and Gaussian probability density functions are shown below.

[0011] [Figure 5A] Histograms of low variance y-latency groups are shown compared to Laplace and Gaussian probability density functions of the same variance. [Figure 5B] Histograms of high variance y-latency groups are shown compared to Laplace and Gaussian probability density functions of the same variance.

[0012] [Figure 6A] 1 illustrates an exemplary process for encoding a multi-distribution entropy model for latent features according to an embodiment of the present invention.

[0013] [Figure 6B] 4 illustrates an exemplary process for decoding a multi-distribution entropy model for latent features according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Exemplary embodiments relating to multi-distribution entropy modeling of latent features in image and video coding using neural networks are described herein. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. It will be apparent, however, that various embodiments of the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices have not been described in exhaustive detail to avoid unnecessarily occluding, obscuring, or obscuring embodiments of the present invention.

[0015] <Summary> Exemplary embodiments described herein relate to image and video coding using neural networks. In an embodiment, a processor receives an encoded image or an encoded video sequence of an encoded picture and syntax parameters for an entropy model of latent features of the encoded image or the encoded video sequence, the entropy model including one or more probability density functions (PDFs) selected from a list of available PDFs including different distribution types. The processor analyzes the syntax parameters for the entropy model of the latent features to generate model parameters for the entropy model, the syntax parameters including a number of one or more PDFs used, an identifier of each PDF used in the list of available PDFs, and optionally a number of PDF parameters for each PDF, and further optionally a syntax element indicating which PDF parameters are shared between two or more PDFs used. The processor decodes the encoded image or the encoded video sequence using the generated model parameters for the entropy model using a neural network. In an example, the entropy model includes a weighted average of two or more PDFs.

[0016] In another embodiment, a processor receives a video sequence including an image or a picture, and analyzes the image or video sequence using a neural network to determine an entropy model for the latent features, the entropy model including one or more probability density functions (PDFs) selected from a list of available PDFs including different distribution types. The processor generates syntax parameters for the entropy model, the syntax parameters including the number of PDFs used, an identifier for each PDF used in the list of available PDFs, and optionally a number of PDF parameters for each PDF, and further optionally which PDF parameters are shared between two or more PDFs used. The processor encodes the image or video sequence into an encoded bitstream using the determined entropy model for the latent features, and includes the syntax parameters for the entropy model in the encoded bitstream. In an example, the entropy model includes a weighted average of two PDFs.

[0017] <Example of an end-to-end video coding model> FIG. 1 shows an exemplary process (100) for modeling image and video coding based on neural learning and coding of latent features (references [1-4]). As used herein, the term "latent feature" or "latent variable" refers to a feature or variable that is not directly observable, but rather is inferred from other observable features or variables, e.g., by processing directly observable variables. In image and video coding, the term "latent space" can refer to a representation of compressed data in which similar data points are closer together. In video coding, examples of latent features include representations of transform coefficients, residuals, motion representations, syntax elements, model information, etc. In the context of neural networks, the latency space is useful for learning features of the data and for finding simpler representations of image data for analysis. As shown in Fig. 1, for a given input image 102, there are two main sub-networks: an autoencoder (encoder 105 and decoder 135 blocks) that learns a quantized latency representation of the image, and a sub-network that learns a probabilistic model of the quantized latency (ŷ) (107) used for entropy coding. The sub-network combines a context model (125), an autoregressive model for the latency, with a hypernetwork (hyperencoder and hyperdecoder blocks 110 and 115) that learns to represent information useful for correcting the context-based prediction. The data from these two sources are combined by an entropy modeling network (130) that generates the parameters (e.g., mean and variance) of a conditional Gaussian entropy model.

[0018] In Figure 1, the arithmetic encoding (AE) block produces a compressed representation of the latency symbols (ŷ) (107) coming from the quantizer (Q) that is stored in a file. Thus, at the time of decoding, any information that depends on the quantized latency can be used by the decoder after it has been decoded. In order for the context model 125 to function, it can only have access to previously decoded latencies.

[0019] The model jointly optimizes an autoregressive component that predicts the latency (y) from the causal context (context model125) together with a hyperprior and an underlying autoencoder.

[0020] The real-valued latency representations are quantized (Q) to produce quantized integer-valued latencies (ŷ) (107) and quantized hyperlatencies (ẑ) (119), which are compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The shaded blocks correspond to the components performed by the receiver to reconstruct the image (137) from the compressed bitstream.

[0021] As described in references [1-2], hierarchical priors (or hyperpriors) are used to improve the entropy model of latencies by capturing their spatial dependencies. Such models allow for end-to-end training, including joint optimization of the hyperprior, the conditional entropy model, and the quantized representation of the base autoencoder. Under this model, the compressed hyperprior can be added to the generated bitstream as side information that allows the decoder to use the conditional entropy model. In this way, a separate entropy model (120) for the hyperprior allows for a richer and more accurate model.

[0022] The training goal is to minimize the expected length of the bitstream and the expected distortion of the reconstructed image with respect to the original image, posing the following rate-distortion (R / D) optimization problem:

number

number

[0023] As in [1], since both the compressed latency and the compressed hyper-latency are part of the generated bitstream, the rate-distortion loss from equation (1) can be extended to include the cost of transmitting ẑ. Combined with the distortion metric D, the full loss function becomes:

number

[0024] This kind of end-to-end deep learning compression-based framework, which generally uses neural networks, consists of two parts: a core autoencoder and an entropy sub-network. The core autoencoder is used to learn the quantized latency vector of the input image or video signal. For this aspect, how to define an efficient neural net (NN) architecture is key. The entropy sub-network is responsible for learning a probability model on the quantized latency representation utilized for entropy coding. From this point of view, finding a suitable entropy model is very important to reduce the bitrate overhead. The embodiments of the present invention propose a novel entropy modeling of the latency features.

[0025] In reference [1], the latency feature y^ is distributed as a Gaussian distribution N(μ, σ 2 ) We modeled the mean and standard deviation (also called the scale parameter) as the previously reconstructed latency y^ <i We estimate the ŷ using the autoregressive context model parameters Φ(127) from the ŷ vector, together with the learned hyperprior feature parameters Ψ(117) derived from the hyperprior latencies ẑ encoded in the bitstream as side information. i The distribution of latencies was considered to be independently conditioned by the hyperprior and the context model.

[0026] In the literature (references [1-4]), typically either a single Gaussian model or a Gaussian Mixture Model (GMM) is used in neural networks for image / video based codecs. In both cases, the entropy model is N(μ g ,σ g 2 ) However, experimental data shows that latency variables do not always follow a Gaussian distribution. Such an example is shown in Figure 5, which shows histograms of low variance y latency groups (505) (Figure 5A) and high variance y latency groups (505) (Figure 5B) compared to Laplace (515) and Gaussian (510) probability density functions of the same variance (scale).

[0027] Therefore, it may be more appropriate to model the distribution of the latency features using an alternative probability density function (PDF), such as Laplace, exponential, exponential-powered, chi-squared, gamma, or other types of distributions known in the art.

[0028] As another example, screen-captured content has different statistical characteristics than camera-captured and natural images, therefore, using a more accurate entropy distribution model based on the source of the image data can improve coding efficiency.

[0029] In an embodiment, a mixture of multiple entropy distribution models is proposed to improve coding efficiency for image and video coding. For example, in an embodiment, it is proposed to replace block 130 of (100) shown in FIG. 1, which uses a Gaussian model, with block 230 of FIG. 2, which uses a model representing entropy modeling of latent features based on a mixture of Laplace and Gaussian probability density functions (LGMs), where the Gaussian model is N(μ g ,σ g 2 ), and the Laplace model can be written as L(μ l ,σl 2 ) The same idea can be applied to other mixture models that use combinations of two or more different probability density functions (e.g., Gaussian and uniform, Laplace and uniform, etc.).

[0030] As seen in Figure 4, the Laplace distribution (405) has a more pronounced peak and broader tails than the Gaussian distribution (410). From the analysis of the y-latency histograms, as shown in Figure 5, it was observed that low variance latencies (see 505 in Figure 5A) follow a much more peaked distribution than the Gaussian distribution (510) and are more appropriately represented by the Laplace distribution (515), while high variance latencies (see 505 in Figure 5B) are closer to the Gaussian distribution (510). Since the variance of the latencies is not known in advance, a mixture of Laplace and Gaussian models is more appropriate. From the experiments, the proposed combined model produced a coding rate gain of 1.2% for natural images and a gain of about 5% for computer-generated images (screen content).

[0031] From an implementation point of view, under the proposed LGM model, the entropy parameter block 130 of the system 100 is replaced by a block 230 with parameters of the Laplace-Gaussian mixture model 200, as shown in FIG. 2. The proposed model can be adapted to a wide range of autoencoder-based frameworks, including hyperprior and joint autoregressive hyperprior models. The training of the entropy parameter network, gep (230), requires adjusting the input features accordingly. For example, the joint autoregressive hyperprior framework in FIG. 1 can be transformed into the proposed model by replacing the Gaussian entropy parameter block (130) with the multi-distribution entropy block (230) shown in FIG. 2. The output Φ (127) of the context model (125) and the output Ψ (117) of the hyperdecoder (115) can be concatenated and fed into a multi-layer convolutional neural network to estimate the entropy parameters.

[0032] FIG. 3 shows an example of a neural network for estimating parameters of a multi-distribution probability model (e.g., the parameters μ g , σ g 2 , w g , μ l , σ l 2 , and w l where w g represents the weight associated with the Gaussian distribution, and w l Since represents the weights associated with the Laplace distribution, we can express the connectivity model for the i-th y latency as:

number

[0033] For example, consider the problem of modeling an end-to-end image or video encoder (200) with N (e.g., N=320) y latency channels using a weighted average of unimodal distributions. In an embodiment, this is accomplished by a mixture of one Gaussian and one Laplace distribution that share a mean. (μ=μ g =μ l ). Conv1, Conv2, and Conv3 are all 1x1 convolutional layers with channel specifications of (1280, 1066), (1066, 853), and (853, 1600), respectively, where the (a, b) pairs represent the number of input and output channels. In an embodiment, the LGM parameter network (230) is g 2 , σ l 2 , w g , and wl It outputs (232) five parameters per channel for . The ReLU unit stands for rectified linear activation unit. ReLU is a piecewise linear function that outputs directly if the input is positive and 0 otherwise. Models that use it are easy to train and often perform better, making it the default activation function for many types of neural networks.

[0034] Given the coupled LGM model in equation (5), the parameter μ gi , σ gi 2 , μ li , σ li 2 , and the mixture model weights w gi And w li i is the previously reconstructed latency feature y^ <i Context parameters φ generated by the autoregressive context model from i (127) and the hyperdecoder parameter ψ derived from the hyperlatency ẑ(119) coded into the bitstream. i Using (117), the entropy parameter network g ep (230). Weights of mixture components (w g and w l ) need to be normalized using an addition layer (e.g., using a Softmax activation function) so that they sum to 1. According to the studies in references [1-3], each latency y^ i We model θ as an LGM convolved with a unit uniform distribution, which ensures a good match between the distributions of the quantized latencies in both the encoder and decoder and the continuous-valued latencies with added uniform noise used during training.

[0035] The distribution of y^ is modeled as:

number

[0036] The proposed scheme offers flexibility to model different kinds of unimodal and multimodal mixtures. In embodiments, a common mean parameter or different mean parameters can be used. Similarly, the standard deviation (or scale) can be chosen to be the same or different, depending on the underlying characteristics of the dataset of interest. In another embodiment, the number of mixture components can be varied by changing the number of layers of the output. The Laplace-Gaussian mixture entropy model thereby aids learning-based image or video codecs to improve the compression efficiency of various contents.

[0037] Some embodiments may use a simpler model than those shown in Figure 1 or 2 (e.g., the coding-decoding loop between the hyper-encoder 110 and the hyper-decoder 115) without the hyper-prior modeling. In such a scenario, the entropy modeling (230) is derived independently of the hyper-prior modeling (117).

[0038] Although the embodiments described herein provide examples of multi-distribution entropy modeling in image and video coding, similar multi-distribution modeling can also be applied to coding of audio and speech signals.

[0039] <Entropy model syntax example> The proposed framework can support a single probability distribution model or a mixture of two or more probability distribution models. In one embodiment, the learning-based codec is trained to generate model parameters for multiple cases of one distribution model (Laplace model or Gaussian model) and / or a mixture of multiple distributions (e.g., Laplace and Gaussian mixture model). When encoding a picture, the encoder decides which distribution model to use (e.g., based on a rate-distortion decision, such as minimizing R+λD, or some statistics of the picture), and then codes the model-related information in a high level syntax (HLS), such as a sequence header, a picture header, etc. The HLS is transmitted together with the encoded bitstream. At the decoder, the decoder reads and analyzes the model-related information from the HLS. The decoder can then select the appropriate model parameters, decode the bitstream, and obtain a reconstructed picture. For example, if the entropy model uses two PDFs (Gaussian and Laplace), then for each model there is a corresponding super-latency neural network that assists in generating the model parameters in block 230. For example, without limitation, the encoder may have three entropy sub-networks: N0 Gaussian only, N1 Laplace only, and N2 LGM (mixture of Laplace and Gaussian). During decoding, depending on the HLS syntax, the decoder may determine which of these three entropy sub-networks to apply.

[0040] Table 1 shows an example of an HLS where the codec supports LGM with either the Gaussian or Laplace model, or a weighted combination of the Laplace and Gaussian models. Table 1: GM syntax example [Table 1] num_of_gaussian_model specifies the number of Gaussian models used to describe one latent variable. The value of num_of_gaussian_model must be 0 or greater. num_of_laplace_model specifies the number of Laplace models used to describe one latency variable. The value of num_of_laplace_model must be 0 or greater. The value of num_of_gaussian_model plus num_of_laplace_model must be greater than 0. is_mean_value_shared equal to 1 specifies that the same mean value is used for both the Gaussian and Laplace models. Is_mean_value_shared equal to 0 specifies that different mean values ​​are used for the Gaussian and Laplace models. If is_scale_value_shared is equal to 1, it specifies that the same scale value is used for both the Gaussian and Laplace models. If is_scale_value_shared is equal to 0, it specifies that different scale values ​​are used for the Gaussian and Laplace models. An example of the syntax and semantics for supporting multiple distribution models is given below. Table 2: Example syntax for multi-distribution entropy models [Table 2]

[0041] In another embodiment, the information in Table 2 can be written as shown in Table 3, where model_id[i] starts at i=0 instead of i=1. Additionally, in Table 3, the value of num_model_params_minus1[i] is inferred from Table 4 and does not need to be sent explicitly. Table 3: Examples of Multi-Distribution Entropy Model Syntax [Table 3] num_models specifies the maximum number of distribution models that will be used to describe one latent variable. The value of num_models must be greater than or equal to 0. model_id[i] identifies the type of distribution model as specified in Table 4. The value of model_id[i] shall be in the range 0 to 4. model_id values ​​5, 6, and 7 are reserved for future use. When not present, the value of valueofmodel_id[i] is inferred to be equal to 0. num_model_params_minus1[i] plus 1 specifies the maximum number of model parameters for the i-th distribution model. When not present, the value of num_model_params_minus1 is inferred to be equal to the value specified in Table 4. Table 4: model_id values [Table 4] shared_model_params[i][j] equal to 1 specifies that the jth model parameter of the ith distribution model is equal to the jth model parameter of the distribution model indicated by shared_model_idx[i][j]. shared_model_params[i][j] equal to 0 indicates that the jth model parameter of the ith distribution model may not be equal to the jth model parameter of any other distribution model. When not present, the value of shared_model_params[i][j] is inferred to be equal to 0. shared_model_idx[i][j] specifies the index k of model_id[k] indicating that the value of the jth model parameter of the i-th distribution model is equal to the value of the jth model parameter of the k-th distribution model. The value of shared_model_idx[i][j] is assumed to be less than i.

[0042] As an example, consider the syntax in Table 3 used with a combination of two models, Gaussian and Laplace. Thus, num_models=2, and from Table 4, model_id[0]=0 and model_id[1]=1. Since both have two parameters (mean and variance), num_model_params_minus1[i]=1 for i=0 and 1. Assuming the two models share a mean (first parameter) but no variance (scale or standard deviation), we get:

number

[0043] FIG. 6A illustrates an example of a process for generating a high-level syntax for an entropy model of latent features according to an embodiment. As shown in FIG. 6A, given an input image or sequence of images, in step 605, the input content is analyzed and the encoder determines the best entropy model for the latent features. For example, but not by way of limitation, this determination may be based on a statistical analysis of the image or latent features, coding requirements, or the source of image content (e.g., natural images, animation, screen content, etc.). For example, as shown in Table 3, but not by way of limitation, the encoder may determine to apply a combination of probability distributions functions (PDFs), such as Gaussian, Laplace, uniform, etc. Then, in step 610, the encoder needs to determine the extent to which the selected probability distributions share model parameters such as minimum, maximum, mean, and scale (standard deviation). This information is also used by the encoder to train a neural network (e.g., 230) to generate all the necessary entropy model parameters, including the weights used to combine the PDFs. In step 615, the selected PDFs and information regarding their model parameter sharing are used to generate a high-level syntax (e.g., as shown in Tables 1-4). Such syntax can be updated at the sequence level, picture level, or sub-picture level as needed to balance coding efficiency and coding complexity. For example, and without limitation, the encoder can choose to use a Laplacian PDF for the first scene, a Gaussian PDF for the second scene, and a weighted average of the Laplacian and Gaussian PDFs for the third scene. In step 620, the generated high-level syntax for the latency entropy model is embedded into the bitstream.

[0044] FIG. 6B illustrates an example of a process for decoding a high-level syntax for an entropy model of latent features according to an embodiment. As shown in FIG. 6B, the encoded bitstream includes both encoded data and high-level syntax related to the entropy model for latent features. In step 650, the bitstream is analyzed to identify information related to the entropy model for latent features. Given this information, in step 655, the decoder identifies the PDF being used and the degree to which the selected probability distributions share model parameters such as minimum, maximum, mean, and scale (standard deviation). This information is also used by the decoder to obtain a deep neural network model (e.g., 230) with all necessary parameters, including the weights used to combine the PDFs. Finally, in step 660, the entropy model generated for the latent features is applied to decode the encoded bitstream. <References> Each of the references cited herein is hereby incorporated by reference in its entirety. [1] D. Minnen, J. Balle', and G. Toderici. “Joint autoregressive and hierarchical priors for learned image compression.” 32nd Conf. on Neural Information Processing Systems (NeurIPS 2018), Montreal, Canada, 2018. [2] J. Balle', D. Minnen, S. Singh, SJ Hwang, and N. Johnston. “Variational image compression with a scale hyperprior.” In International Conference on Learning Representations (ICLR), 2018, also arXiv:1802.01436v2 (2018). [3] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto. “Learned image compression with discretized Gaussian mixture likelihoods and attention modules.” In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR)”, pages 7939-7948, 2020, also arXiv e-prints (2020): arXiv-2001.01568v3, 30 March 2020. [4] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “DVC: An end-to-end deep video compression framework,” in 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 10998-11007). IEEE Computer Society, 2019, also arXiv:1812.00101v3, 7 April 2019. [5] J. Duda, “Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding,” arXiv preprint arXiv:1311.2540v2, 6 Jan. 2014.

[0045] <Implementation of an exemplary computer system> Embodiments of the invention may be implemented by a computer system, a system configured in electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or other configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to entropy modeling of latency features in images and videos as described herein. The computer and / or IC may calculate any of various parameters or values ​​related to entropy modeling of latency features in images and videos as described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0046] A particular implementation of the present invention includes a computer processor executing software instructions that cause the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. may perform the method related to entropy modeling of latent features in images and videos described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention b may be provided in the form of a program product. The program product may include any non-transitory tangible medium that carries a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the method of the present invention. A program product according to the present invention may be in any of a variety of non-transitory tangible forms. The program product may include a physical medium, such as, for example, a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD-ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signals on the program product may be optically compressed or encrypted. Although components (e.g., software modules, processors, components, devices, circuits, etc.) have been referred to above, unless otherwise indicated, references to those components (including references to "means") should be interpreted to include equivalents of those components, any components that perform the functions of the described components (e.g., are functionally equivalent), and components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated exemplary embodiments of the present invention.

[0047] <Equivalents, Extensions, Alternatives and Miscellaneous> Exemplary embodiments have been described that relate to entropy modeling of latency features in image and video coding. In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what the invention is, and what applicant intends to be the invention, is set forth in the claims issued in a particular form hereby, including any subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall control the meaning of such terms as used in the claims. Thus, any limitations, elements, features, advantages, or attributes not expressly set forth in the claims should not limit the scope of the claims in any manner. The specification and drawings should therefore be considered illustrative, and not restrictive.

[0048] Various aspects of the invention may be apparent from the enumerated example embodiments (EEE) described below. (EEE1) A method of decoding an image or an encoded video sequence by a processor using a neural network, comprising: receiving an encoded image or an encoded video sequence comprising an encoded picture and syntax parameters for an entropy model of latent features of the image or encoded video sequence, the entropy model comprising one or more probability density functions (PDF); analyzing the syntax parameters for the entropy model of the latent features to generate model parameters for the entropy model, the syntax parameters including a number of one or more PDFs used, and an identifier for each PDF used in a list of available PDFs; - decoding the encoded image or the encoded video sequence using the neural network, using the model parameters generated for the entropy model; The method includes: (EEE2) The method of EEE1, wherein the syntax parameters further include the number of PDF parameters in each PDF. (EEE3) The method according to EEE1 or 2, wherein the syntax parameters further include syntax elements indicating which PDF parameters are shared between two or more PDFs being used. (EEE4) The method according to any one of EEE1 to 3, wherein the number of PDFs used is greater than one. (EEE5) The method of EEE4, wherein the one or more PDFs include at least one Gaussian PDF having a first mean and a first variance, and at least one Laplace PDF having a second mean and a second variance. (EEE6) The method of EEE3 and 5, wherein a syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their mean values ​​are the same. (EEE7) The method of EEE3 and 5, wherein a syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their variance values ​​are the same. (EEE8) The method described in EEE3 and 5, wherein a syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their mean values ​​are the same and that their variance values ​​are the same. (EEE9) The method according to any one of EEE1 to 8, wherein the list of available PDFs includes Gaussian PDF, Laplace PDF, and Uniform PDF. (EEE10) A method according to EEE1 to EEE9 which is dependent on EEE3, wherein a syntax element indicating which PDF parameters are shared between two or more PDFs being used includes a set of shared model parameter flags denoted as shared_model_params[i][j] and a set of shared model indexes denoted as shared_model_idx[i][j], where shared_model_params[i][j]=1 indicates that the jth PDF parameter of the ith PDF is equal to the jth PDF parameter of the PDF indicated by shared_model_idx[i][j], and shared_model_idx[i][j]=k indicates that the jth PDF parameter of the ith PDF is equal to the jth PDF parameter of the kth PDF. (EEE11) The method according to any of EEE1 to 10, wherein the syntax parameters of the entropy model include high level syntax at sequence, picture or slice level. (EEE12) The method described in any of EEE1 to 11, wherein for an entropy model of latent features comprising two or more PDFs, the entropy model further comprises weights for calculating a weighted average between one or more PDFs. (EEE13) In a coded video sequence, the entropy model comprises: PDF No. 1, The second PDF, and a weighted average of said first PDF and said second PDF; The method according to any of EEE1 to 12, wherein the method can alternate or switch between: (EEE14) A method of encoding, by a processor, an image or an encoded video sequence using a neural network, comprising the steps of: receiving a video sequence including images or pictures; using the neural network to analyze the images or video sequences to determine an entropy model of latent features in the images or video sequences, the entropy model comprising one or more probability density functions (PDFs); generating syntax parameters for the determined entropy model, the syntax parameters including a number of the one or more PDFs used, and an identifier for each PDF used in a list of available PDFs; encoding the image or the video sequence into an encoded bitstream using the determined entropy model of the latent features; including the syntax parameters of the entropy model in the encoded bitstream; The method includes: (EEE15) The method of EEE14, wherein the syntax parameters further include a number of PDF parameters in each PDF. (EEE16) The method according to EEE14 or 15, wherein the syntax parameters further comprise syntax elements indicating which PDF parameters are shared between two or more PDFs being used. (EEE17) A method according to any one of EEE14 to 16, wherein the number of PDFs used is greater than one. (EEE18) The method of EEE17, wherein the one or more PDFs include at least one Gaussian PDF having a first mean and a first variance, and at least one Laplace PDF having a second mean and a second variance. (EEE19) The method according to EEE16 and 18, wherein the syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their mean values ​​are the same. (EEE20) The method according to EEE16 and 18, wherein a syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their variance values ​​are the same. (EEE21) The method described in EEE16 and 18, wherein the syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their mean values ​​are the same and that their variance values ​​are the same. (EEE22) The method according to any one of EEE14 to 21, wherein the list of available PDFs includes Gaussian PDF, Laplace PDF, and Uniform PDF. (EEE23) A method according to EEE14 to 22, dependent on EEE16, in which a syntax element indicating which PDF parameters are shared between two or more PDFs being used comprises a set of shared model parameter flags, denoted as shared_model_params[i][j], and a set of shared model indexes, denoted as shared_model_idx[i][j], where shared_model_params[i][j]=1 indicates that the jth PDF parameter of the ith PDF is equal to the jth PDF parameter of the PDF indicated by shared_model_idx[i][j], and shared_model_idx[i][j]=k indicates that the jth PDF parameter of the ith PDF is equal to the jth PDF parameter of the kth PDF. (EEE24) The method according to any one of EEE14 to 23, wherein the syntax parameters of the entropy model include high level syntax at sequence, picture or slice level. (EEE25) The method described in any of EEE14 to 24, wherein for an entropy model of latent features comprising two or more PDFs, the entropy model further comprises weights for calculating a weighted average between one or more PDFs. (EEE26) In an encoded video sequence, the entropy model comprises: PDF No. 1, The second PDF, and a weighted average of said first PDF and said second PDF; The method according to any one of EEE14 to 25, wherein the method can alternate or switch between: (EEE27) The method described in EEE14, further comprising identifying shared and non-shared PDF parameters between two or more PDFs in the entropy model when the entropy model comprises a mixture of two or more PDFs, and further generating syntax parameters identifying which PDF parameters are shared between the two or more PDFs used. (EEE28) A non-transitory computer-readable storage medium storing computer-executable instructions for executing, by one or more processors, the method of any one of EEE1 to EEE27. (EEE29) An apparatus including a processor and configured to perform any one of the methods described in EEE1-27.

Claims

1. 1. A method for decoding an encoded image or an encoded video sequence using a neural network by a processor, the method comprising: receiving an encoded image or an encoded video sequence comprising an encoded picture and syntax parameters for an entropy model of latent features of the encoded image or the encoded video sequence, the entropy model comprising one or more probability density functions (PDFs) selected from a list of available PDFs comprising different distribution types; analyzing the syntax parameters for the entropy model of the latent features to generate model parameters for the entropy model, the syntax parameters including a number of one or more PDFs used and an identifier for each PDF used in the list of available PDFs; - decoding the encoded image or the encoded video sequence using the neural network, using the model parameters generated for the entropy model; The method includes:

2. The method of claim 1 , wherein the number of PDFs used is greater than one.

3. The method of claim 2 , wherein the syntax parameters further include a number of PDF parameters in each PDF and syntax elements indicating which PDF parameters are shared between two or more PDFs being used.

4. 4. The method of claim 2 or 3, wherein the one or more PDFs include at least one Gaussian PDF having a first mean and a first variance and at least one Laplace PDF having a second mean and a second variance.

5. 5. The method of claim 4 when dependent on claim 3, wherein the syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their mean values ​​are the same.

6. 5. The method of claim 4 when dependent on claim 3, wherein the syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their variance values ​​are the same.

7. The method according to any of claims 1 to 6, wherein the list of available PDFs includes Gaussian PDF, Laplace PDF and Uniform PDF.

8. The shared_model_params syntax element indicates which PDF parameters are shared between two or more PDFs that are used. A set of shared model parameter flags denoted as [i][j] and a set of shared model indexes denoted as shared_model_idx[i][j], A method according to any one of claims 4 to 7 dependent on claim 3, wherein [i][j]=1 indicates that the jth PDF parameter of the i-th PDF is equal to the jth PDF parameter of the PDF indicated by shared_model_idx[i][j], and shared_model_idx[i][j]=k indicates that the jth PDF parameter of the i-th PDF is equal to the jth PDF parameter of the k-th PDF.

9. The method according to any of claims 1 to 8, wherein the entropy model syntax parameters include high level syntax at sequence, picture or slice level.

10. The method of any of claims 1 to 9, wherein for an entropy model of latent features comprising two or more PDFs, the entropy model further comprises weights for computing a weighted average between one or more PDFs.

11. For an encoded video sequence, the entropy model is 1st PDF, The second PDF, and a weighted average of the first PDF and the second PDF; The method according to any one of claims 1 to 10, wherein the step of:

12. A non-transitory computer readable storage medium storing computer executable instructions for carrying out, by one or more processors, the method of any one of claims 1 to 11.

13. Apparatus including a processor and configured to carry out the method according to any of claims 1 to 11.

14. 1. A method of encoding an image or video sequence using a neural network in a processor, comprising the steps of: receiving a video sequence including images or pictures; using the neural network to analyze the image or video sequence and determine an entropy model of latent features in the image or video sequence, the entropy model comprising one or more probability density functions (PDFs) selected from a list of available PDFs comprising different distribution types; generating syntax parameters for the determined entropy model, the syntax parameters including a number of one or more PDFs used, and an identifier for each PDF used in the list of available PDFs; encoding the image or the video sequence into an encoded bitstream using the determined entropy model of the latent features; including the syntax parameters of the entropy model in the encoded bitstream; The method includes:

15. The method of claim 14, wherein the number of PDFs used is greater than one.

16. The method described in claim 15, wherein the syntax parameters further include a number of PDF parameters in each PDF and syntax elements indicating which PDF parameters are shared between two or more PDFs being used.

17. A method as described in claim 15 or 16, wherein the one or more PDFs include at least one Gaussian PDF having a first mean value and a first variance value, and at least one Laplace PDF having a second mean value and a second variance value.

18. A method according to claim 17 dependent on claim 16, wherein a syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their average values ​​are the same.

19. A method according to claim 17 dependent on claim 16, wherein a syntax element indicating which PDF parameters are shared between the Gaussian PDF and the Laplace PDF indicates that their variance values ​​are the same.

20. The method of claim 15, wherein the list of available PDFs includes a Gaussian PDF, a Laplace PDF, and a uniform PDF.

21. A syntax element indicating which PDF parameters are shared between two or more PDFs being used is shared_model_params A set of shared model parameter flags denoted as [i][j] and a set of shared model indexes denoted as shared_model_idx[i][j], A method according to any one of claims 17 to 20 dependent on claim 4, wherein [i][j]=1 indicates that the jth PDF parameter of the i-th PDF is equal to the jth PDF parameter of the PDF indicated by shared_model_idx[i][j], and shared_model_idx[i][j]=k indicates that the jth PDF parameter of the i-th PDF is equal to the jth PDF parameter of the k-th PDF.

22. A method according to any one of claims 15 to 21, wherein the syntax parameters of the entropy model include high level syntax at the sequence, picture or slice level.

23. A method according to any one of claims 15 to 22, wherein for an entropy model of latent features comprising two or more PDFs, the entropy model further comprises weights for calculating a weighted average between one or more PDFs.

24. In an encoded video sequence, the entropy model comprises: 1st PDF, The second PDF, and a weighted average of the first PDF and the second PDF; The method according to any of claims 15 to 23, wherein the step of

25. The method of claim 14, further comprising identifying shared and non-shared PDF parameters between two or more PDFs in the entropy model when the entropy model includes a mixture of two or more PDFs, and further generating syntax parameters that identify which PDF parameters are shared between the two or more PDFs being used.

26. A non-transitory computer-readable storage medium storing computer-executable instructions for executing the method of any one of claims 14 to 25 by one or more processors.

27. An apparatus including a processor and configured to perform a method according to any one of claims 14 to 25.

Images