Threshold model for skipping entropy coding in end-to-end image compression using neural networks

Abstract

A method for determining thresholds for skipping entropy coding of latent features in image and video coding using neural networks is described. A threshold based on the mean of the estimated standard deviations of all latents is proposed. For autoregressive neural networks, it is proposed to have two separate entropy parameter estimation networks to avoid drift between context model parameter estimates computed during training and inference: an initial entropy estimation network used in entropy skipping of quantized latent coding and decoding, and a refined entropy estimation network used in arithmetic coding and decoding of quantized latents. Methods and systems for multi-stage entropy skipping are also proposed.

Description

[Technical Field]

[0001] Cross-references to related applications This patent application claims priority to Indian Provisional Patent Application No. 202311038186, filed on 2 June 2023, and Indian Provisional Patent Application No. 202311061566, filed on 13 September 2023, both of which are incorporated herein by reference in their entirety.

[0002] technology This paper broadly relates to images. More specifically, embodiments of the present invention relate to thresholding models for skipping entropy coding in image and video compression using neural networks. [Background technology]

[0003] In 2020, the MPEG group of the International Organization for Standardization (ISO), in collaboration with the International Telecommunication Union (ITU), released the first version of the Versatile Video Coding (VVC) standard, also known as H.266. More recently, the group has been working on developing next-generation coding standards that offer improved coding performance over existing video coding techniques. As part of this research, coding techniques based on artificial intelligence and deep learning are also being considered. As used herein, the term “deep learning” refers to a neural network having at least three layers, preferably more than three layers.

[0004] As used herein, the term “end-to-end image compression neural network” refers to a neural network that congruently optimizes all components of a video compression system, from the input point of the uncompressed input to the encoder to the output point of the reconfigured output image to the decoder, using a single loss function.

[0005] As acknowledged by the inventors of this specification, improved techniques for neural network-based image and video encoding are described herein. The approaches described in this section are approaches that can be pursued, but are not necessarily approaches that have been conceived or pursued previously. Therefore, unless otherwise specified, none of the approaches described in this section should be assumed to be eligible as prior art simply because they are included in this section. Similarly, any problem identified with respect to one or more approaches should not be assumed, unless otherwise specified, to have been recognized in any prior art based on this section. [Prior art documents] [Non-patent literature]

[0006] Each of the references listed herein is incorporated in its entirety by reference. [Non-Patent Document 1] D. Minnen, J. Balle', and G. Toderici, "Joint autoregressive and hierarchical priors for learned image compression." 32nd Conf. on Neural Information Processing Systems (NeurIPS 2018), Montreal, Canada, 2018 [Non-Patent Document 2] D. Liu, Y. Li, J. Lin, H. Li, F. Wu, "Deep learning-based video coding: A review and a case study." arXiv:1904.12462v1, April 29, 2019 [Non-Patent Document 3] Y. Shi, Y. Ge, J. Wang, J. Mao, "AlphaVC: High-Performance and Efficient Learned Video Compression," arXiv:2207.14678, July 29, 2022 [Non-Patent Document 4] Li, Jiahao, Bin Li, and Yan Lu, "Hybrid spatial-temporal entropy modeling for neural video compression." Proceedings of the 30th ACM International Conference on Multimedia. 2022 [Non-Patent Document 5] Li, Jiahao, Bin Li, and Yan Lu, "Neural video compression with diverse contexts." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023 [Brief explanation of the drawing]

[0007] Embodiments of the present invention are shown in the accompanying drawings as examples, and are not limiting; similar reference numerals refer to similar elements.

[0008] [Figure 1] This document presents an example of a neural network model for image and video coding based on probabilistic modeling of latent features using conventional techniques.

[0009] [Figure 2] A first exemplary neural network model for image and video coding to skip entropy coding is shown according to one embodiment of the present invention.

[0010] [Figure 3]A second exemplary neural network model for image and video coding that skips entropy coding is shown according to one embodiment of the present invention.

[0011] [Figure 4A] An exemplary process for two-stage entropy coding according to one embodiment of the present invention is shown.

[0012] [Figure 4B] An exemplary process for two-step entropy coding with entropy skipping, according to one embodiment of the present invention, is shown.

[0013] [Figure 5A] An example of partitioning a latent space using a quadtree partition according to one embodiment of the present invention is shown.

[0014] [Figure 5B] An example of a selection mask used with entropy skipping according to one embodiment of the present invention is shown.

[0015] [Figure 6] An exemplary process for multi-stage entropy skipping according to one embodiment of the present invention is shown. [Modes for carrying out the invention]

[0016] Exemplary embodiments relating to skipping entropy coding (also known as “entropy skipping”) in image and video coding using neural networks are described herein. The following description includes many specific details to provide a full understanding of the various embodiments of the invention for illustrative purposes. However, it will be apparent that various embodiments of the invention can be carried out without these specific details. In other examples, well-known structures and devices are not described in exhaustive detail to avoid unnecessarily obscuring, ambiguizing, or making the embodiments of the invention unclear.

[0017] overview The exemplary embodiments described herein relate to image and video coding using neural networks. Methods for determining thresholds for skipping the entropy coding of latent features in image and video coding using neural networks are described. In a first embodiment, a threshold based on the mean of the estimated standard deviations of all latents is proposed. In another embodiment, quantized latents with absolute values ​​below the threshold are replaced with very small positive values, and their entropy coding is skipped. For autoregressive neural networks, it is proposed to have two separate entropy parameter estimation networks to avoid drift between the estimated values ​​of context model parameters computed during training and inference: an initial entropy estimation network used in skipping the entropy coding and decoding of quantized latents, and a more sophisticated entropy estimation network used in the arithmetic coding and decoding of quantized latents. Embodiments for multi-stage entropy skipping are also proposed.

[0018] Exemplary end-to-end video coding model Figure 1 shows an exemplary process (100) for modeling image and video coding based on neural learning and latent feature coding (Non-Patent Literature 1, 2). As used herein, the terms “latent feature” or “latent variable” refer to features or variables that are not directly observable, but rather are inferred from other observable features or variables, for example, by processing directly observable variables. In image and video coding, the term “latent space” may refer to a representation of compressed data where similar data points are close to each other. In video coding, examples of latent features include representations such as transformation coefficients, residuals, motion representations, syntactic elements, and model information. In the context of neural networks, latent space is useful for learning data features and finding simpler representations of image data for analysis.

[0019] As shown in Figure 1, for a given input image 102, there are two main subnetworks: an autoencoder (encoder 105 and decoder 135 blocks) that learns the quantized latent representation of the image, and a subnetwork responsible for learning a probabilistic model for the quantized latent (^y) (107) used for entropy coding. The subnetwork combines a context model (125), which is an autoregressive model for the latent, with a hypernetwork (hyperencoder and hyperdecoder blocks 110 and 115) that learns to represent information useful for correcting context-based predictions. Data from these two sources are combined by an entropy modeling network (130) to generate parameters (e.g., mean and variance) for a conditional Gaussian entropy model.

[0020] In Figure 1, the arithmetic coding (AE) block generates a compressed representation of the latent symbol (^y)(107) coming from the quantizer (Q), which is stored in a file. Therefore, during decoding, any information that depends on the quantized latent can be used by the decoder after decoding. For context model 125 to function, it can only access latents that have been previously decoded.

[0021] This model constitutively optimizes an autoregressive component that predicts the latent (y) from its causal context (context model 125) together with a hyperplier and an underlying autoencoder. The real-valued latent representation is quantized (Q), creating quantized integer-valued latents (^y) (107) and quantized hyperlatent (^z) (119). These are compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). Blocks with a crosshatch background correspond to components performed by the receiver to reconstruct an image (137) from the compressed bitstream.

[0022] As discussed in Non-Patent Document 1, a hierarchical plier (or hyperplier) z(112) is used to improve the latent entropy model by capturing their spatial dependencies. Such a model allows end-to-end training including a quantized representation of the hyperplier, a conditional entropy model, and congruent optimization of a basic autoencoder. Under this model, the compressed hyperplier may also be added as side information to the generated bitstream, thereby allowing the decoder to use the conditional entropy model. In this way, the individual entropy models (120) of the hyperplier allow for a richer and more accurate model.

[0023] The training goal is to minimize the expected length of the bitstream and the expected distortion of the reconstructed image relative to the original image. This solves the rate-distortion (R / D) optimization problem: R+λD (1) This occurs. Here, λ is the Lagrange multiplier that determines the desired rate-distortion (RD) tradeoff, and R and D may be expressed as follows:

number

[0024] Similar to Non-Patent Document 1, since both the compressed latency and the compressed hyperlatency are part of the generated bitstream, the rate-distortion loss from equation (1) can be extended to include the cost of transmitting ^z. Combined with the distortion metric D, the complete loss function is as follows:

number

number

[0025] This type of end-to-end deep learning compression-based framework using neural networks generally consists of two parts: a core autoencoder and an entropy subnetwork. The core autoencoder is used to learn the quantized latent vectors of the input image or video signal. For this aspect, the key is how to define an efficient neural network (NN) architecture. The entropy subnetwork is responsible for learning a probabilistic model for the quantized latent representation used for entropy coding. For this aspect, finding a suitable entropy model is crucial for reducing bitrate overhead. Embodiments of the present invention propose a novel entropy modeling of latent features.

[0026] In Non-Patent Document 1, the latent feature ^y is a Gaussian distribution N(μ,σ) with mean μ and standard deviation (or scale) σ. 2 It is modeled by ). The mean and standard deviation (also called the scale parameter) are the previously reconstructed latent ^y <iJointly estimated using the autoregressive context model parameters (127) from and the learned hyper-prior feature parameters Ψ(117) derived from the hyper-prior latent ^z encoded in the bitstream as side information. ^y i The latent distribution is considered to be adjusted independently for the hyper-prior and the context model.

[0027] Entropy skip As discussed in Non-Patent Document 3, in video coding, due to the use of reference frames, the quantized latent entropy is relatively small, and thus has a distribution function with a relatively small variance. It has been proposed to skip the entropy coding and entropy decoding of certain elements and simply replace them with the peaks of their probability distributions. This can save both the bitrate and the runtime in entropy coding with little expected error.

[0028] In Non-Patent Document 3, a sufficiently high probability q i such that the latent y i is within ±0.5 from the center of the distribution is proposed to skip the coding of. The erf() function

Number

Number

Number

number

[0029] mean(σ i ) threshold based on In one embodiment, it is proposed to improve the threshold proposed in Non-Patent Document 3 by taking all latent distributions into account. i Instead of relying on an absolute threshold determined by probability, the proposed threshold relies on local thresholding, encouraging the network to discard approximately 50% of the latents and skip their coding. This is done by calculating the mean standard deviation term within the current set of latents and determining the skip threshold based on its value. Experiments have shown that this results in approximately 60-70% of the latents being skipped. This entropy skip model can be expressed as follows:

number

number

[0030] In one alternative embodiment, each y iThe value to be skipped is determined by the magnitude of the absolute value of y. i If the absolute value of falls below some threshold τ, the value is not encoded into the bitstream, and instead a non-zero ε (for example, ε=1E) is used. -6 ) can be replaced by ). Since each latent is in the feature space, this method is similar to the DCT quantization commonly used in the JPEG standard. Alternatively, in the decoder, y i Since the value is not available, the threshold may be defined using the estimated mean. Therefore,

number

[0031] Entropy skipping in autoregressive models In an autoregressive network as shown in Figure 1, the latent entropy (or context) model (125) is adjusted using the previously decoded latent, in addition to the hyperplier parameters (117). For example, in Figure 1, this is indicated by a link from the AD block before the decoder (135) to the context model (125). Because the adjustment in the entropy model uses more context, the autoregressive model is significantly more efficient in latent compression.

[0032] To directly apply the entropy skipping scheme of equation (9) to such a neural network model, in the autoregressive scheme, the entropy skipping threshold is, for example, mean(σ i ) needs to be calculated. Here, this mean is calculated against the previously decoded standard deviation. However, this approach does not work in practice. During training, for reasons of computational efficiency, the mean (μ) i ) and scale (i.e., standard deviation σ) iThe autoregressive computation of ) is performed in parallel for all latents using masked convolutions. This mask restricts the latents used in the convolution to only those previously decoded. Some of these latents are skipped in the latent coding process, and since the skipping decision depends on the scale, this creates a chicken-or-the-egg situation where the calculation of the mean and scale depends on knowledge of the scale. If this is not taken into account and all latents are used in the calculation of the standard deviation (scale) and mean without considering any possible latent skips, the model may show a significant performance improvement during training, but inference may show poor coding efficiency. This is due to some information leakage that occurs on the network's training path. The information leakage is due to not replacing skipped latents with means in the calculation of the output context parameters (127) in the context model (125). Training is forced to use information that is not available during inference. During inference, the decoder cannot access the skipped latents and therefore cannot reproduce the scale and mean used during encoding. Furthermore, it cannot obtain the skipping decisions calculated from the scale. This results in a discrepancy between the encoder and the decoder.

[0033] In one embodiment, this problem is addressed by modifying the original neural network to first generate an initial set of mean and scale from only hyperplier information. Figure 2 shows a block diagram of this system. Compared to Figure 1, the new network includes an entropy-skip encoder (205) and an entropy-skip decoder (210). Also, the original Gaussian entropy parameter estimation block (130) is replaced with two similar blocks: an initial entropy parameter estimation block (215) and a refined entropy parameter estimation block (220). The initial entropy parameter estimation block (215) contains the initial mean and scale (μ) used in entropy skipping. i 0 ,σi 0 ) is estimated. The output of the hyperdecoder (115) is used in the sophisticated mean and scale (μ) used in the quantized latent arithmetic coding (AE) and arithmetic decoding (AD). i r ,σ i r To estimate ), a new, sophisticated entropy parameter estimation block (220) is also applied. In some embodiments, though not limited thereto, the skip decision is now mean(σ i This is done using the following method. Here, the scale average is the initial scale σ i 0 (217) obtained, the entropy-skipped latent is the initial mean estimate μ i 0 It is replaced by the latent map obtained from the entropy-skip encoder (205), which is used in the context model block 125 for the calculation of context parameters, which are then used in block 220 to calculate a refined mean and scale. The refined mean and scale are then used to obtain a probability distribution for encoding the non-skipped latent. In the decoder, the entropy-skip decoder (210) provides the latent map (212) to the context modeler (125) to generate a context model during inference.

[0034] Experimental results confirm the benefits of this approach. The training and inference paths yield consistent results, and the resulting codec is significantly more efficient than a codec without entropy skipping. As previously mentioned, the codec now has two entropy parameter networks: one that generates the initial mean and scale, and the other that generates the refined mean and scale. The networks are trained by minimizing the rate-distortion loss from equation (3), where the rate R is the entropy of the autoencoder latent under a stochastic model using the refined mean and scale parameters. Since the initial mean and scale do not directly affect the rate-distortion loss, the rate R is changed to a weighted mean of the entropy of the autoencoder latent under a stochastic model using the refined mean and scale parameters and the initial mean and scale parameters, to ensure that the initial entropy parameter network is also properly trained. For example, this weighting can be expressed as follows:

number

[0035] During inference, the decoder first decodes the hyperlatency and generates the initial mean and scale using the initial entropy parameter network (215). The latent skipping decision is derived from the initial scale. For example, using the model of equation (9), given the initial mean and scale (μ i 0 , σ i 0 Given ), all missing quantized latents in the encoded latent stream are ^yi =μ i 0 The value will be assigned.

[0036] The bitstream contains only the encoded latents that are not skipped by the encoder. The entropy skip decoder (210) takes the arithmetic-decoded latents from the bitstream and fills the skipped latents with the corresponding initial mean estimates to produce the output latents that have been decoded so far, which are used as the context input by the context model (125). The refined entropy parameter network (220) uses the output of the context model (125) along with hyperplier parameters to generate a refined mean and scale. The refined mean and scale are then used to generate an entropy model that is used by the arithmetic decoder (AD) to decode subsequent latents from the bitstream in an autoregressive manner.

[0037] Entropy skipping in multi-stage entropy models Current state-of-the-art neural network-based compression models (Non-Patent Documents 4, 5) have chosen to relax the autoregressive entropy coding approach due to runtime constraints. While autoregressive models offer the best performance, they cannot be parallelized. A common alternative approach to entropy models is the multi-stage entropy model. These multi-stage models are similar to autoregressive entropy models, but with slight differences. Instead of sequentially encoding each latent element based on the previous latent element, multi-stage models divide the latent into multiple blocks and encode them in parallel.

[0038] This partitioning strategy is generally known as a two-stage checkerboard partition or a four-stage quadtree partition, but alternative partitioning trees with eight or more partitions can also be used. A checkerboard partition simply divides the latent tensor into odd and even spatial elements, and then divides the channels in half. This results in a partition containing the odd elements, which includes half of the channels, and another partition containing the remaining half of the channels, which includes the even elements. The selected latent is then the predicted μ i,0 , σ i,0 Encoded by: The first stage entropy parameter μ i,0 , σ i,0 The latent ^y processed as i Using this, the second-stage entropy parameter μ for the remaining latent that has not yet been encoded. i,1 , σ i,1 This is calculated. The quadtree partitioning is similar to the checkerboard partitioning strategy, but it divides the channel into four partitions and spatially divides the latent into a quadtree structure. Details can be found in Non-Patent Documents 4 and 5, and will be discussed later in this section.

[0039] Figure 3 shows an exemplary block diagram of such a system (300). Compared to Figure 1, the output 107 of the quantizer (Q) is now fed here to an arbitrary temporal feature model block (305), which in turn is fed to a checkerboard or quadtree entropy parameter block (310), replacing the context model (125) and Gaussian entropy parameter (130) blocks, but also eliminating the need for feedback from decoder AD to context model (125).

[0040] The role of the Temporal Feature Model Block is to add temporal information (or modeling parameters), such as motion vector information, when encoding a video sequence. Therefore, this block is not used for still images or the first frame of a video sequence, but its use in subsequent frames of a video sequence adds additional modeling information that can improve the estimation of the entropy parameter.

[0041] In conventional "single-frame" entropy coding, given a sequence of quantized latents (hereinafter simply referred to as latents for simplicity) (405), entropy parameters μ and σ for the entire "frame" are calculated to generate an encoded bitstream. Figure 4A shows an example of a checkerboard partition that allows parallelization of the entropy coding process across the "white" (407) and "black" (409) regions of the latents in frame (405). The depiction of only four latents in frame 405 is merely an example and not limiting. As shown in Figure 4A, the first-stage bitstream (410) can be generated based only on the white latents (407) and their corresponding first-stage entropy parameters μ and σ (412). The second-stage bitstream (430) can be generated based solely on the black latent (409), but the second-stage entropy parameter (422) is adjusted based on the first-stage entropy parameter (412), hyperlatency, temporal context (if available), and previous latent (black or white).

[0042] Figure 4B shows an example of a two-stage entropy coding model using entropy skipping. As illustrated, for the first stage, given the entropy parameters (μ, σ)(412) and the corresponding threshold (see, for example, equation (9)), a Boolean selection mask (435) is computed to determine which of the white latents can be entropy skipped among the black and white latents.

[0043] Generally, each latent has associated mean and scale parameters. In one embodiment, for each set of stage latents (for example, in Figure 4B, Stage 1: latents A and D, Stage 2: latents B and C), the mean scale (variance) of all latents is taken, either in that stage or across all latents. Then, if the scale of a latent in a stage exceeds the mean scale, that latent is encoded; otherwise, it is entropy skipped. Typically, not all latents in a stage are entropy skipped, but often a large portion of the elements in each stage are skipped. As an example, in Figure 4B, based on selection mask 435, for Stage 1, latent D is entropy skipped and latent A is entropy encoded. For Stage 2, based on selection mask 440, only latent C is encoded and latent B is entropy skipped.

[0044] The selection mask identifies to the encoder and decoder which values ​​should be retained, skipped, and recalculated. Generally, at each stage, the selection mask value for a particular latent is set to 0 if its scale value is less than the calculated mean of all variances of the elements at that stage, and to 1 otherwise, indicating that no entropy skipping occurred. Note that the selection mask does not need to be sent from the encoder to the decoder. The decoder has all the information necessary to calculate the mean and scale entropy parameters, as well as the corresponding mask, based on the hyperlatent, any arbitrary temporal context, and previously decoded latents. The decoder simply replaces the skipped latents with the estimated mean for each stage.

[0045] Extending this method to quadtree partitioning, Figure 5A shows partitioning the latent space (505) into quadtree representations with quantized latent regions (or stages) 1, 2, 3, and 4. In such a scenario, we begin by entropically encoding latent "1", then entropically encoding latent "2" using updated context information, encoding latent "3" using updated context information, and finally encoding latent "4". Again, the depiction of only 16 latents (505) is an example and not limiting. Once all four latent regions are encoded, four bitstreams can be transmitted. When using entropy skipping, a Boolean selection mask (e.g., 435,440) must be adapted to support the new quadtree structure. An example of such a selection mask is shown in Figure 5B. Here, though not limited, we entropy encode two latents in step "1" and skip the other two, or in (row, column) representation: encode latents (1,1) and (3,3) and skip encoding latents (1,3) and (3,1). For example, after encoding step "1", we proceed to the latents of step "2", for example encoding latents (1,2), (3,2) and (3,4) (skipping latent (1,4)), proceed to the latents of step "3", for example encoding latent (2,1) (skipping latents (2,3), (4,1) and (4,3)), and encode all the step "4" latents, for example latents (2,2), (2,4), (4,2) and (4,4) (no entropy skipping).

[0046] In summary, in this type of neural network model, the application of the entropy skipping scheme of equation (9) is that for each stage of the partitioning strategy, the entropy skipping threshold, i.e., mean(σ) iThis requires calculating the entropy parameter. This is due to the stepwise refinement of the entropy parameter, in which the entropy parameter is refined at each stage and adjusted based on the previous stage. Using these new scale parameters, a new entropy skip threshold must be calculated. Otherwise, a discrepancy will arise between the actual entropy parameter and the initial entropy parameter for which the selection mask was calculated. This process is summarized in Figure 6. However, the latent processing is the same as in the models described earlier. That is, for the elements to be skipped, ^y i =μ i,m That is the case.

[0047] As shown in Figure 6, given a multi-stage entropy-skip model, the entropy-skip process begins in step 605 with the input of quantized latents and hyperlatencies, as well as an arbitrary temporal context (if applicable). For example, the first stage is denoted as stage j=0 (of the K total stages). For example, K=2 for a checkerboard partition and K=4 for a quadtree partition.

[0048] For step j=0, in step 610, the corresponding entropy parameter (μ i,0 , σ i,0 ) and the entropy skip threshold are calculated. Then, in step 615, based on the calculated threshold, select which latents should be entropy skipped and generate a selection mask for generating the encoded bitstream for this stage based on the latents that are not skipped.

[0049] After the initial stage, it is detected whether the entire latent space (all K stages) has been encoded. If not, stage-specific bitstreams are generated as follows: ● Corresponding entropy parameter (μ i,j , σ i,j ) hyperpotential and (μ i,j-1 , σ i,j-1) and optionally based on past latent and temporal context (if available) (step 620); then in step (625) ○Calculate the updated entropy skip threshold for stage j, ○Calculate a choice mask to define an entropy-skippable latent, ○ Generate an entropy-encoded bitstream of stage j using the latent that is not skipped. Steps 620 and 625 are repeated until all K steps have been encoded.

[0050] In the decoder, the process is the same, except that a computed selection mask is used to regenerate the skipped quantized latent elements by simply assigning an average latent value for each stage. The received decoded bitstream and the reconstructed missing elements are then combined to form the final decoded bitstream.

[0051] Experimental results show that training with multi-stage entropy skipping improves the efficiency of the codec. The network is trained by minimizing the rate-distortion loss from equation (3), but with some differences. Before calculating the rate term defined in equation (2), the elements to be skipped must be masked out to avoid overestimating the rate. This mask M is formed by merging the selection masks at each stage into a single mask by simply adding them together. The rate term is calculated as follows:

number

[0052] Exemplary Receiving System Implementation Embodiments of the present invention can be implemented in computer systems, systems composed of electronic circuits and components, integrated circuit (IC) devices such as microcontrollers, field-programmable gate arrays (FPGAs), or other configurable or programmable logic devices (PLDs), discrete-time or digital signal processors (DSPs), application-specific ICs (ASICs), and / or in apparatus comprising one or more such systems, devices, or components. Computers and / or ICs can execute, control, or run instructions related to skipping the entropy coding of latent features in image and video coding, as described herein. Computers and / or ICs can calculate any of the various parameters or values ​​related to skipping the entropy coding of latent features in image and video coding, as described herein. Embodiments of image and video can be implemented in hardware, software, firmware, and various combinations thereof.

[0053] Certain embodiments of the present invention include a computer processor that executes software instructions causing the processor to perform the method of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc., can implement the method relating to skipping the entropy coding of latent features in image and video coding as described above by executing software instructions in program memory accessible to the processor. Embodiments of the present invention may be provided in the form of a program product. The program product may include any non-temporary and tangible medium that carries a set of computer-readable signals including instructions causing the data processor to perform the method of the present invention when executed by the data processor. The program product according to the present invention may be any of a variety of non-temporary and tangible forms. The program product may include physical media such as magnetic data storage media including floppy disks, optical data storage media including hard disk drives, CD-ROMs, DVDs, and electronic data storage media including ROMs and flash RAM. The computer-readable signals on the program product may optionally be compressed or encrypted. Where components (e.g., software modules, processors, assemblies, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to such components (including references to “means”) should be interpreted as including any components that perform the function of the described component (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structure that performs the function of the described component, including components that are not structurally equivalent to the disclosed structure that performs the function of the described exemplary embodiment of the invention.

[0054] Equivalents, extensions, substitutes and others An exemplary embodiment relating to skipping the entropy coding of latent features in image and video coding is described herein. In the foregoing description, embodiments of the invention have been described with reference to numerous individual details, which may vary from implementation to implementation. Therefore, the sole and exclusive indicator of what constitutes an invention and what is intended to be an invention by the applicant is the set of claims, including any subsequent amendments, in the specific form issued pursuant to this application. The definitions expressly provided herein for any terms contained in such claims shall govern the meaning of such terms as used in such claims. Therefore, no limitations, elements, characteristics, features, advantages, or attributes not expressly stated in the claims shall in any way limit the scope of such claims. Accordingly, the specification and drawings should be considered illustrative and not restrictive.

[0055] Various aspects of this disclosure can be understood from the following Enumerated Example Embodiments (EEEs). [EEE1] A method for skipping quantized latent entropy coding in a neural network for video coding, the method being: Input video content into a neural network encoder to detect latent (y i The steps to generate ) and; The aforementioned latent is quantized to obtain the quantized latent (^y i The steps to generate ) and; The mean value of the aforementioned latent (μ i ) and standard deviation (σ i The steps to estimate ) and; A step of applying entropy coding to the quantized latent, wherein the quantized latent (^y i Regarding ) σ i <mean(σ i ) If so, ^y i =μ i And the entropy coding is skipped. Otherwise, the quantized latent (^y i ) is entropy-encoded, and an entropy-encoded latent is generated, and mean(σ i ) is for all σ for i=1,2,...N i The value represents the average, N represents the total number of latents generated, and the step and Methods that include... [EEE2] A method for skipping quantized latent entropy coding in a neural network for video coding, the method being: Input video content into a neural network encoder to detect latent (y i The steps to generate ) and; The aforementioned latent is quantized to obtain the quantized latent (^y i The steps to generate ) and; The mean value of the aforementioned latent (μ i The steps to estimate ) and; A step of applying entropy coding to the quantized latent, wherein the latent (y i Regarding ) |μ i If |<τ then ^y i =ε*sign(μ i ) and its entropy coding is skipped, Otherwise, the quantized latent (^y i ) is entropy-encoded to produce an entropy-encoded latent, where τ represents a threshold and ε represents a small positive value greater than 0 and less than 0.001, and the step and Methods that include... [EEE3] A method for skipping quantized latent entropy coding in an autoregressive neural network for video coding, the method being: Input video content into a neural network encoder to detect latent (y i The steps to generate ) and; The aforementioned latent is quantized to obtain the quantized latent (^y i The steps to generate ) and; estimating the potential initial mean value (μ i 0 ) and the initial standard deviation (σ i 0 ); applying the potential initial mean value (μ i 0 ) and the initial standard deviation (σ i 0 ) to an entropy skip encoder to determine an updated quantized potential value and to determine whether the quantized potential value is entropy encoded or the entropy encoding for the quantized potential is skipped; estimating the refined mean value (μ i r ) and the standard deviation (σ i r ) of the potential based on at least the entropy skipped potential and the entropy not skipped potential; applying the refined mean value (μ i r ) and the standard deviation (σ i r ) of the potential for arithmetic coding and arithmetic decoding of the entropy encoded potential; A method comprising: [EEE4] Determining the updated quantized potential value and determining whether the quantized potential value is entropy encoded or the entropy encoding is skipped may be: If σ i 0 < mean(σ i 0 ), then ^y i = μ i 0 and the entropy encoding thereof is skipped; Otherwise, ^y i is entropy encoded to generate an entropy encoded potential, mean(σ i 0) is for all σ for i=1,2,...N i 0 This represents the average of the values, and N represents the total number of latents generated. Methods described in EEE3. [EEE5] The updated quantized latent value is determined, and it is decided whether the quantized latent value is entropy coded or whether entropy coding is skipped: potential y i Regarding |μ i 0 If |<τ then ^y i =ε*sign(μ i 0 ) and its entropy coding is skipped, Otherwise, the quantized latent (^y i ) is entropy coded, which includes generating an entropy coded latent, τ represents the threshold, and ε represents a small positive value greater than 0 and less than 0.001. Methods described in EEE3. [EEE6] A method for decoding a picture using a neural network, the method being: The steps include receiving a bitstream containing encoded latent and encoded hyperlatent; The steps include decoding the hyperlatency to generate hyperplier parameters and initial entropy parameters including initial mean and initial scale; A step of applying the initial scale to an entropy-skipped decoder to identify skipped output latents, wherein the skipped output latents include latents that were not entropically encoded by the encoder of the bitstream; A step of determining the value of the skipped output potential based on the initial average value; The steps include: applying the skipped output potential and the hyperplier parameters to a context model to generate refined entropy parameters, including refined mean and refined scale; The steps include: applying the refined entropy parameters to decode the encoded latent and generate an unskipped output latent; The steps include generating a decoded picture of the output based on at least the values ​​of the skipped output potential and the unskipped output potential, and Methods that include... [EEE7] A method for skipping quantized latent entropy coding in a neural network for video coding, the method being: Input video content into a neural network encoder to detect latent (y i The steps to generate ) and; The aforementioned latent is quantized to obtain the quantized latent (^y i The steps to generate ) and; The steps include: dividing the quantized latent into K non-overlapping groups of the quantized latent; Based on the hyperlatency, the mean value (μ) of the quantized latent in group j=0 of the K groups. i 0 ) and standard deviation (σ i 0 The steps to estimate ) and; The aforementioned latent average value (μ i 0 ) and standard deviation (σ i 0 The steps are to apply ) to an entropy-skipping encoder to determine the output latent value for group j=0; For the group j=1 to K-1 of the aforementioned K groups, regarding the quantized latent within group j: Hyperlatency and (μ i j-1 ,σ i j-1Based on the value, the mean value (μ) of the quantized latent in group j. i j ) and standard deviation (σ i j ) estimate, The aforementioned latent average value (μ i j ) and standard deviation (σ i j The steps include: applying ) to the entropy-skip encoder to determine the coded latent value of the output for group j; A step of generating a bitstream of encoded quantized latents based on the coded latent values ​​of the output for K groups. Methods that include... [EEE8] The method described in EEE7, where K=2 or 4. [EEE9] The output latent value is determined, and then it is determined whether the quantized latent value is entropy coded or whether entropy coding is skipped: σ i j <mean(σ i j ) If so, ^y i =μ i k And the entropy coding is skipped. Otherwise, ^y i This involves entropy coding, which generates an entropy coded latent. mean( σ i j ) is all σ about the quantized latent within group j i j Represents the average of the values, The method described in EEE7 or 8. [EEE10] A method for decoding a picture using a neural network, the method being: A step of receiving a bitstream containing encoded latents and encoded hyperlatencies, wherein the encoded latents represent quantized latents divided into K non-overlapping groups; For the aforementioned encoded latent group j=0: The aforementioned hyperlatency is decoded to obtain the hyperprior parameter and the initial mean value (μ i 0 ) and initial scale (σ i 0 Generate initial entropy parameters including ); The initial scale is applied to the entropy-skipped decoder to identify the skipped output latent, which includes the latent that was not entropically encoded by the encoder of the bitstream; The step of determining the value of the skipped output potential for group j=0 based on the aforementioned initial mean value; For the aforementioned encoded latent group j (j=1 to K-1): Hyperlatency and (μ i j-1 ,σ i j-1 Based on the value, the mean value (μ) of the quantized latent in group j. i j ) and standard deviation (σ i j ) estimate, Estimated mean value (μ i j ) and standard deviation (σ i j The steps are: to decode the encoded latent by applying ) to generate skipped output latents and unskipped output latents; The steps include generating a decoded picture of the output based on the skipped output potential and the unskipped output potential values ​​of the output for at least K groups, and Methods that include... [EEE11] It is possible to generate decoded, skipped quantized latents and unskipped quantized latents: σ i j <mean(σ i j ) If so, ^y i The entropy coding of ^y is skipped, i =μ i k And, Otherwise, ^y i It is entropy-encoded, and its value is generated by an entropy decoder. mean( σ i j ) is all σ about the quantized latent within group j i j Represents the average of the values, Methods described in EEE10. [EEE12] A non-temporary computer-readable storage medium storing computer-executable instructions for performing the method described in any one of EEE1 to 11 on one or more processors. [EEE13] A device having a processor configured to perform the method described in any one of EEE1 to EEE11.

Claims

1. A method for skipping quantized latent entropy coding in a neural network for video coding, the method being: The video content is input into the neural network encoder to extract the latent (y i The steps to generate ) and; The aforementioned latent is quantized to obtain the quantized latent (^y i The steps to generate ) and; The mean value of the aforementioned latent (μ i ) and standard deviation (σ i The steps include: estimating ) and; A step of applying entropy coding to the quantized latent, wherein the quantized latent (^y i Regarding ) σ i <mean(σ i ) If so, ^y i = μ i is set, and its entropy encoding is skipped, Otherwise, the quantized latent (^y i ) is entropy-encoded, and an entropy-encoded latent is generated, and mean(σ i ) is for all σ for i = 1, 2, ..., N i The value represents the average, N represents the total number of latents generated, and the step and Methods that include...

2. A method for skipping quantized latent entropy coding in a neural network for video coding, the method being: The video content is input into the neural network encoder to extract the latent (y i The steps to generate ) and; The aforementioned latent is quantized to obtain the quantized latent (^y i The steps to generate ) and; The mean value of the aforementioned latent (μ i The steps include: estimating ) and; A step of applying entropy coding to the quantized latent, wherein the latent (y i Regarding ) |μ i |<τ, ^y i = ε*sign(μ i ) and its entropy coding is skipped, Otherwise, the quantized latent (^y i ) is entropy-encoded to produce an entropy-encoded latent, where τ represents a threshold and ε represents a small positive value greater than 0 and less than 0.001, and the step and Methods that include...

3. A method for skipping quantized latent entropy coding in an autoregressive neural network for video coding, the method being: The video content is input into the neural network encoder to extract the latent (y i The steps to generate ) and; The aforementioned latent is quantized to obtain the quantized latent (^y i The steps to generate ) and; The initial average value of the latent (μ) i 0 ) and initial standard deviation (σ i 0 The steps include: estimating ) and; The initial mean value (μ) of the latent i 0 ) and initial standard deviation (σ i 0 The steps include: applying ) to an entropy-skipping encoder to determine the updated quantized latent value and determining whether the quantized latent value is entropy-coded or whether entropy coding for that quantized latent is skipped; Based on at least the entropy-skipped latents and the entropy-free latents, the refined mean value (μ) of the latents is calculated. i r ) and standard deviation (σ i r The steps include: estimating ) and; For the latent arithmetic coding and arithmetic decoding that is entropy coded, the latent refined mean value (μ i r ) and standard deviation (σ i r The steps to apply ) Methods that include...

4. The updated quantized latent values ​​are determined, and it is decided whether the quantized latent values ​​are entropy-coded or whether entropy coding is skipped: σ i 0 <mean(σ i 0 ) If so, ^y i = μ i 0 And the entropy coding is skipped. Otherwise, ^y i This involves entropy coding, which generates an entropy coded latent. mean( σ i 0 ) is for all σ for i = 1, 2, ..., N i 0 This represents the average of the values, and N represents the total number of latents generated. The method according to claim 3.

5. The updated quantized latent values ​​are determined, and it is decided whether the quantized latent values ​​are entropy-coded or whether entropy coding is skipped: potential y i Regarding |μ i 0 |<τ, ^y i = ε*sign(μ i 0 ) and its entropy coding is skipped, Otherwise, the quantized latent (^y i ) is entropy coded, which includes generating an entropy coded latent, τ represents the threshold, and ε represents a small positive value greater than 0 and less than 0.

001. The method according to claim 3.

6. A method for decoding a picture using a neural network, the method being: The steps include receiving a bitstream containing encoded latent and encoded hyperlatent; The steps include: decoding the hyperlatency to generate hyperprior parameters and initial entropy parameters including initial mean and initial scale; Steps include: applying the initial scale to an entropy-skipping decoder to identify skipped output latents, wherein the skipped output latents include latents that were not entropy-encoded by the encoder of the bitstream; A step of determining the value of the skipped output potential based on the initial average value; The steps include: applying the skipped output potential and the hyperplier parameters to a context model to generate refined entropy parameters, including refined mean and refined scale; The steps include: applying the refined entropy parameters to decode the encoded latent and generate an unskipped output latent; The steps include generating a decoded picture of the output based on at least the values ​​of the skipped output potential and the unskipped output potential, and Methods that include...

7. A method for skipping quantized latent entropy coding in a neural network for video coding, the method being: The video content is input into the neural network encoder to extract the latent (y i The steps to generate ) and; The aforementioned latent is quantized to obtain the quantized latent (^y i The steps to generate ) and; The steps include: dividing the quantized latent into K non-overlapping groups of the quantized latent; Based on the hyperlatency, the mean value (μ) of the quantized latent in group j=0 of the K groups. i 0 ) and standard deviation (σ i 0 The steps include: estimating ) and; The aforementioned latent mean value (μ i 0 ) and standard deviation (σ i 0 The steps are: ) apply to the entropy-skipping encoder to determine the output latent value for group j=0; For the groups j=1 through K-1 of the aforementioned K groups, regarding the quantized latent within group j: Hyperlatency and (μ i j-1 ,σ i j-1 Based on the value, the mean value of the quantized latent in group j (μ i j ) and standard deviation (σ i j ) estimate, The aforementioned latent mean value (μ i j ) and standard deviation (σ i j The steps include: applying the above entropy-skipping encoder to determine the coded latent value of the output for group j; A step of generating a bitstream of encoded quantized latents based on the coded latent values ​​of the output for K groups. Methods that include...

8. The method according to claim 7, wherein K = 2 or 4.

9. The output latent value is determined, and then it is determined whether the quantized latent value is entropy coded or whether entropy coding is skipped: σ i j <mean(σ i j ) If so, ^y i = μ i k And the entropy coding is skipped. Otherwise, ^y i This involves entropy coding, which generates an entropy coded latent. mean(σ i j ) represents the mean of all σ i j values for the quantized latent within group j The method according to claim 7.

10. A method for decoding a picture using a neural network, the method being: A step of receiving a bitstream containing encoded latents and encoded hyperlatencies, wherein the encoded latents represent quantized latents divided into K non-overlapping groups; For the aforementioned encoded latent group j=0: The aforementioned hyperlatency is decoded to obtain the hyperprior parameter and the initial mean value (μ i 0 ) and initial scale (σ i 0 Generate initial entropy parameters including ); The initial scale is applied to the entropy-skipped decoder to identify the skipped output latent, which includes the latent that was not entropically encoded by the encoder of the bitstream; Steps include: determining the value of the skipped output potential for group j=0 based on the initial mean value; For the encoded latent group j (j = 1 to K-1): Hyper-potentials and (μ i j-1 , σ i j-1 ) values, estimate the mean value (μ i j ) and the standard deviation (σ i j ) for the quantized potential in group j, Estimated mean value (μ i j ) and standard deviation (σ i j Steps include: Decoding the encoded latent by applying ) to generate skipped output latents and unskipped output latents; The steps include generating a decoded picture of the output based on the skipped output potential and the unskipped output potential values ​​of the output for at least K groups, and Methods that include...

11. It is possible to generate decoded, skipped quantized latents and unskipped quantized latents: σ i j <mean(σ i j ) If so, ^y i The entropy coding of ^y is skipped, i = μ i k And, Otherwise, ^y i This includes the fact that it is entropy-encoded, and its value is generated by an entropy decoder. mean( σ i j ) is all σ about the quantized latent within group j i j Represents the average of the values, The method according to claim 10.

12. A non-temporary computer-readable storage medium storing computer-executable instructions for performing the method according to any one of claims 1 to 11 on one or more processors.

13. A device having a processor configured to perform the method described in any one of claims 1 to 11.

Images