Threshold model for skipping entropy coding in end-to-end image compression using neural networks

Abstract

A method for determining a threshold for entropy coding that skips potential features in image and video coding using a neural network is described. A threshold for estimating a mean of standard deviations based on all latent variables is proposed. For an autoregressive neural network, in order to avoid drifting between context model parameter estimations calculated during training and reasoning, two separate entropy parameter estimation networks, namely an initial entropy estimation network and a refined entropy estimation network, are adopted, the initial entropy estimation network is used for entropy skipping coding and decoding of quantized latent variables, and the refined entropy estimation network is used for entropy skipping coding and decoding of quantized latent variables. A refined entropy estimation network is used for arithmetic encoding and decoding of quantized latent variables. Methods and systems for multi-stage entropy skipping are also presented.

Description

[0001] Cross-reference to related applications This patent application claims priority to Indian Provisional Patent Application No. 202311038186, filed on June 2, 2023, and Indian Provisional Patent Application No. 202311061566, filed on September 13, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0002] This document generally relates to images. More specifically, embodiments of the invention relate to a thresholding model that uses neural networks to skip entropy coding in image and video compression. Background Technology

[0003] In 2020, the MPEG expert group within the International Organization for Standardization (ISO) and the International Telecommunication Union (ITU) jointly released the first version of the Universal Video Coding Standard (VVC), also known as H.266. More recently, the MPEG expert group has focused on developing a next-generation coding standard with improved coding performance compared to existing video coding technologies. As part of this research, coding techniques based on artificial intelligence and deep learning were also examined. As used herein, the term "deep learning" refers to a neural network with at least three layers, and preferably more than three layers.

[0004] As used in this paper, the term "end-to-end image compression neural network" refers to a neural network that uses a single loss function to jointly optimize all components of a video compression system, from the input point of the encoder's uncompressed input to the output point of the decoder's reconstructed output image.

[0005] As the inventors understand it herein, this document describes an improved technique for encoding images and videos based on neural networks.

[0006] The methods described in this section are permissible but not necessarily methods that have been previously conceived or employed. Therefore, unless otherwise instructed, no method described in this section should be considered prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise instructed, any issues concerning one or more methods should not be considered to be in any prior art based on this section. Attached Figure Description

[0007] Embodiments of the invention are illustrated in the accompanying drawings by way of example rather than limitation, and similar reference numerals refer to similar elements, and in the drawings: Figure 1 An example neural network model for image and video encoding based on probabilistic modeling of latent features, according to existing technologies, is described; and Figure 2A first example neural network model for skipping entropy coding in image and video coding according to an embodiment of the present invention is described; Figure 3 A second example neural network model for skipping entropy coding in image and video coding according to an embodiment of the present invention is described; Figure 4A An example process for two-stage entropy coding according to an embodiment of the present invention is described; Figure 4B An example process for two-stage entropy encoding using entropy skipping is described according to an embodiment of the present invention; Figure 5A An example of partitioning a potential space using a quadtree according to an embodiment of the present invention is described; Figure 5B An example of a selection mask to be used with entropy skipping according to an embodiment of the invention is depicted; and Figure 6 An example process for multi-stage entropy skipping according to an embodiment of the present invention is described. Detailed Implementation

[0008] This document describes example embodiments related to skipping entropy coding (also known as "entropy skipping") in image and video encoding using neural networks. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of various embodiments of the invention. However, it will be apparent that various embodiments of the invention can be practiced without these specific details. In other instances, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obscuring, or confusing embodiments of the invention. Summary of the Invention

[0009] The example embodiments described herein relate to image and video coding using neural networks. Methods for determining thresholds to skip entropy encoding of latent features in image and video coding using neural networks are described. In a first embodiment, a threshold based on the mean of the estimated standard deviations of all latent variables is proposed. In another embodiment, quantized latent variables with absolute values ​​less than the threshold are replaced with very small positive values, and their entropy encoding is skipped. For autoregressive neural networks, to avoid drift between estimates of context model parameters computed during training and inference, two separate entropy parameter estimation networks are proposed: an initial entropy estimation network and a refined entropy estimation network. The initial entropy estimation network is used to skip entropy encoding and decoding of quantized latent variables, and the refined entropy estimation network is used for arithmetic encoding and decoding of quantized latent variables. Embodiments for multi-stage entropy skipping are also proposed.

[0010] Example end-to-end video coding model Figure 1An example process for modeling image and video coding based on neural learning and encoding of latent features is described (100) (References [1-2]). As used herein, the term "latest feature" or "latent variable" refers to a feature or variable that is not directly observable but inferred from other observable features or variables (e.g., by processing directly observable variables). In image and video coding, the term "latent space" can refer to a representation of compressed data in which similar data points are grouped closer together. Examples of latent features in video coding include representations of transform coefficients, residuals, motion representations, syntax elements, model information, etc. In the context of neural networks, latent spaces are very useful for learning data features and finding simpler representations of image data for analysis.

[0011] like Figure 1 As depicted, given an input image 102, there are two main subnetworks: an autoencoder (encoder block 105 and decoder block 135) that learns the quantized latent representation of the image, and a network responsible for learning a probabilistic model on the quantized latent variables. (107) is a sub-network used for entropy encoding. The sub-network combines a context model (125) (an autoregressive model on latent variables) with a super-network (super encoder block 110 and super decoder block 115) that learns how to represent information useful for correcting context-based predictions. Data from these two sources are combined by an entropy modeling network (130), which generates parameters (e.g., mean and variance) of a conditional Gaussian entropy model.

[0012] exist Figure 1 In the process, the arithmetic coding (AE) block generates latent symbols from the quantizer (Q). The compressed representation of (107) is stored in a file. Therefore, during decoding, any information dependent on the quantized latent variables can be used by the decoder once decoded. For the context model 125 to function properly, it can only access previously decoded latent variables.

[0013] The model jointly optimizes the results based on latent variables ( y The causal context predicts the autoregressive components of these latent variables (context model 125) as well as the super-prior and low-level autoencoder. Real-valued latent representations are quantized (Q) to produce quantized integer-valued latent variables ( (107) and quantified latent variables ( (119) These are compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). Blocks with crosshair backgrounds correspond to parts that are reconstructed from the compressed bitstream by the receiver (137).

[0014] As discussed in reference [1], hierarchical priors (or super-priors) z (112) is used to improve the entropy model of latent variables by capturing the spatial dependencies of latent variables. Such a model allows for end-to-end training, which involves joint optimization of the quantized representation of the super-prior, the conditional entropy model, and the underlying autoencoder. Under this model, compressed super-prior can be added as auxiliary information to the generated bitstream, which allows the decoder to use the conditional entropy model. In this way, a separate entropy model of the super-prior (120) allows for a richer and more accurate model.

[0015] The training objective is to minimize the expected length of the bitstream and the expected distortion of the reconstructed image relative to the original image, which results in rate distortion (RDS). R / D Optimization problem: (1) in, It is the Lagrange multiplier that determines the required rate-distortion (RD) tradeoff, and R and D It can be represented as: (2) , in, Represents the unknown distribution of natural images. q The period (.) indicates rounding to the nearest integer. Indicates encoder output, Represents quantified latent variables. It is a discrete entropy model, and It is the decoder output, where, This represents the reconstructed image. The rate term corresponds to the cross-entropy between the marginal distribution of the latent variable and the learned entropy model, and it is minimized when these two distributions are equal. The distortion term might correspond to, for example, when... Measures of distortion can be closed-form probabilities, such as mean squared error (MSE), structural similarity of images (SSIM), multi-scale structural similarity (MS-SSIM), IW-SSIM (information content weighted structural similarity), FSIM (feature similarity index), PSNR-HVSM (peak signal-to-noise ratio of the human visual system, considering contrast masking between coefficients of the contrast sensitivity function (CSF) and DCT basis functions), VMAF (video multi-method evaluation fusion), VIF (visual information fidelity measure), VDP2 (visual difference predictor), NLPD (normalized Laplacian pyramid distortion), and learning-based distortion measures such as LPIPS (learned perceptual patch similarity) and DISTIS (deep image structure and texture similarity).

[0016] As shown in reference [1], since both the compressed latent variable and the compressed super-latent variable are part of the generated bitstream, the rate distortion loss in equation (1) can be extended to include the transmission loss. The cost. And distortion metrics. D Combining these, the complete loss function becomes: (3) in, (4) This indicates the rate caused by the over-latent variable.

[0017] This framework for end-to-end deep learning-based compression using neural networks generally consists of two parts: a core autoencoder and an entropy subnetwork. The core autoencoder is used to learn the quantized latent vector of the input image or video signal. The key here is how to define an efficient neural network (NN) architecture. The entropy subnetwork is responsible for learning a probabilistic model on the quantized latent representation for entropy coding. Finding the correct entropy model is crucial for reducing bit rate overhead. Embodiments of this invention present a novel latent feature entropy modeling approach.

[0018] In reference [1], latent features From having a mean and standard deviation (or scale) Gaussian distribution modeling The mean and standard deviation (which can also be called scaling parameters) are latent variables derived from the previous reconstruction. Autoregressive context model parameters (127) and as auxiliary information encoded in the bitstream from the prior latent variable The obtained learning hyper-prior feature parameters (117) Jointly estimated. The distribution of latent variables is considered to be independently conditioned by the prior and context model.

[0019] Entropy skip As discussed in reference [3], in video coding, due to the use of reference frames, the entropy of the quantized latent variables is relatively small, and therefore the variance of their distribution function is relatively small. It is proposed to skip entropy coding and entropy decoding of certain elements and directly replace them with the peak values ​​of their probability distributions. This can save both bit rate and runtime in entropy coding, and there is almost no error expectation.

[0020] In reference [3], it is proposed to skip those with sufficiently high probability. latent variables Encoding of these latent variables (within ± 0.5 of their distribution center). Using the erf() function. (5) Given standard deviation The probabilities are calculated as follows: (6) Given this probability, a threshold can be applied. This determines whether to encode the element or skip it. This decision is described as... (7) in, It is the center (mean) of the Gaussian distribution. It is the standard deviation. ,in , N This represents the total number of latent variables. It can be observed that... yes Since it is a monotonic function, the error function calculation can be replaced instead with a function targeting... The specially selected threshold, denoted here, is This threshold is related to the target The selected value is associated with (e.g., )or (8) in, This indicates a quantization operation.

[0021] based on threshold In this embodiment, an improvement is proposed to the threshold proposed in reference [3] by considering the distribution of all latent variables. The proposed threshold does not depend on the distribution of all latent variables. Instead of an absolute threshold determined by probability, the skip threshold relies on a local threshold, encouraging the network to discard approximately 50% of latent variables and skip their encoding. This is done by calculating the average standard deviation term within the current set of latent variables and determining the skip threshold based on that value. Based on experiments, this results in approximately 60% to 70% of latent variables being skipped. This entropy skipping model can be expressed as: (9) Among them, for N One latent variable, .

[0022] For example, given a size of W × H Input video blocks and having M A CNN encoder network with one output channel. ,in,k Indicates the encoder's downsampling ratio, for example, W = H = 256, M = 64 or 192, and k = 16.

[0023] In alternative embodiments, according to each The absolute magnitude is used to determine which values ​​to skip. In this method, if... The absolute value is below a certain threshold Then its value will not be encoded into the bitstream, but will be replaced with some non-zero ε (e.g., Since each latent variable resides in the feature space, this method is similar to the DCT quantization commonly used in the JPEG standard. Alternatively, because... Since the value is not available in the decoder, the threshold can be defined based on the estimated mean. (10) Among them, if x If it is positive, then , and if x If it is negative, then And threshold Optimization can be achieved based on experimental results.

[0024] Entropy skipping in autoregressive models In autoregressive networks, such as Figure 1 The described autoregressive network, in addition to the hyperprior parameters (117), also conditions the entropy (or context) model of the latent variables (125) with the previously decoded latent variables. For example, in Figure 1 This is illustrated by the link from the AD block preceding the decoder (135) to the context model (125). Since the conditions in the entropy model utilize more context, the autoregressive model is significantly more efficient in compressing latent variables.

[0025] For this neural network model, directly applying the entropy skipping scheme of equation (9) would require calculating the entropy skipping threshold in an autoregressive manner (e.g., The mean is calculated using the standard deviation of the previously decoded variables. However, this method doesn't work in practice. During training, for computational efficiency reasons, the mean of all latent variables ( ) and scale (i.e., standard deviation) The autoregressive computation of the model will be performed in parallel using masked convolutions, which restrict the latent variables used in the convolutions to only those previously decoded. Since some of these latent variables will be skipped during latent variable encoding and the skipping decision depends on the scale, this leads to a chicken-and-egg problem, i.e., the calculation of the mean and scale depends on the knowledge of the scale. If this problem is not addressed, and all latent variables are used without considering any latent variable skipping when calculating the standard deviation (scale) and mean, the model may show a significant performance improvement during training, but will show lower encoding efficiency during inference. This is due to some information leakage that occurs in the training path of the network. The information leakage is due to the failure to replace the skipped latent variables with the mean when calculating the output context parameters (127) in the context model (125). Training ultimately uses information that is not available during inference. During inference, the decoder cannot access the skipped latent variables, and therefore it cannot reproduce the scale and mean used during encoding; furthermore, it cannot obtain the skipping decision calculated based on the scale. This leads to a divergence between the encoder and decoder.

[0026] In this embodiment, the problem is addressed by modifying the original neural network to first generate an initial set of means and scales based solely on prior information. Figure 2 A block diagram of the system is shown. Figure 1 In contrast, the new network includes an entropy skip encoder (205) and an entropy skip decoder (210). The new network also replaces the original Gaussian entropy parameter estimation block (130) with two similar blocks: an initial entropy parameter estimation block (215) and a refined entropy parameter estimation block (220). The initial entropy parameter estimation block (215) estimates the initial mean and scale to be used for entropy skipping (…). The output of the superdecoder (115) is also applied to a new refined entropy parameter estimation block (220) to estimate the parameters to be used for... through Refined mean and scale of arithmetic encoding (AE) and arithmetic decoding (AD) of latent variables. In the embodiments, without limitation, the following are now used: The method makes a skip decision, where the scale-mean is from the initial scale. (217) obtained, and estimated using the initial mean. The latent variables that have been skipped by entropy are replaced. The latent variable map obtained from the entropy skip encoder (205) is used in the context model block 125 to compute context parameters, which are then used in block 220 to compute the refined mean and scale. The refined mean and scale are then used to obtain the probability distribution for encoding the unskipped latent variables. At the decoder, the entropy skip decoder (210) provides the latent variable map (212) to the context modeler (125) to generate the context model during inference.

[0027] Experimental results confirm the benefits of this approach. The training and inference paths yield consistent results, and the resulting codec is significantly more efficient than the codec without entropy skipping. As discussed, the codec now has two entropy parameter networks, one generating the initial mean and scale, and the other generating the refined mean and scale. This network is trained by minimizing the rate-distortion loss in equation (3), where the rate R is the entropy of the autoencoder's latent variables under a probabilistic model using refined mean and scale parameters. Since the initial mean and scale do not directly affect the rate-distortion loss, the rate R is optimized to ensure that the initial entropy parameter network is also properly trained. R It has been modified to be a weighted average of the entropies of the autoencoder latent variables under a probabilistic model using both refined and initial means and scale parameters. For example, this weighting can be expressed as: (11) in, This represents the weight within the range [0,1]. It is the initial discrete entropy model, and It's a refined entropy model. It was found that the performance of the codec is affected by... The value is not particularly sensitive, and it has been found through experience that... The value performs well.

[0028] During inference, the decoder first decodes the latent variables and generates initial means and scales using the initial entropy parameter network (215). Latent variables are then skipped based on the initial scales. For example, using the model with equation (9), given the initial means and scales ( ), to encode each missing variable in the latent variable stream through Quantitative latent variable allocation The value of .

[0029] The bitstream contains only the encoded latent variables that have not been skipped by the encoder. An entropy skip decoder (210) extracts the arithmetically decoded latent variables from the bitstream and fills the skipped latent variables with corresponding initial mean estimates to produce the output latent variables decoded so far, which are used as context inputs by the context model (125). A refinement entropy parameter network (220) uses the output of the context model (125) and the super-prior parameters to generate a refinement mean and scale. The refinement mean and scale are then used to generate an entropy model, which is used by the arithmetic decoder (AD) to decode subsequent latent variables in the bitstream in an autoregressive manner.

[0030] Entropy skipping in multi-stage entropy models Due to runtime constraints, current state-of-the-art neural network-based compression models (references [4-5]) have opted for relaxed autoregressive entropy encoding methods. While autoregressive models offer the best performance, they cannot be parallelized. A popular alternative to entropy models is the multi-stage entropy model. These multi-stage models are similar to autoregressive entropy models but with subtle differences. Instead of encoding each latent element sequentially (conditional on previous latent elements), multi-stage models partition the latent variables into multiple blocks and encode them in parallel.

[0031] This partitioning strategy is often referred to as a two-stage checkerboard arrangement or a four-stage quadtree partitioning, but alternative partitioning trees with eight or more partitions can also be used. Checkerboard partitioning simply divides the latent tensor into odd-space elements and even-space elements, then splits the channels in half. This produces one partition with odd elements and half the channels, and another partition with the remaining half of the channels and even elements. The predicted... The selected latent variables are encoded. The first-stage entropy parameter is used. and processed latent variables Then it is used to calculate the second-stage entropy parameter for the remaining latent variables that have not yet been encoded. Quadtree partitioning is similar to the chessboard partitioning strategy, but it divides the channels into four partitions and spatially partitions the latent variables into a quadtree structure. More details can be found in references [4,5] and will be discussed later in this section.

[0032] Figure 3 An example block diagram of this system (300) is depicted. (Compared to...) Figure 1 In contrast, the output 107 of the quantizer (Q) is now fed into an optional temporal feature model block (305) and then into a checkerboard or quadtree entropy parameter block (310), replacing the context model (125) and the Gaussian entropy parameter block (130), and also eliminating the need for feedback from the decoder AD back to the context model (125).

[0033] The purpose of the temporal feature model block is to add temporal information (or modeling parameters), such as motion vector information, when encoding a video sequence. Therefore, although this block is not used for still images or the first frame of a video sequence, using it in subsequent frames of the video sequence can add additional modeling information, thereby improving the estimation of the entropy parameter.

[0034] In traditional single-frame entropy coding, given a series of quantized latent variables (hereinafter referred to as latent variables for simplicity) (405), the entropy parameters of the entire frame are calculated. μ and σ To generate an encoded bitstream. Figure 4AAn example of checkerboard partitioning is depicted, which allows for parallel entropy encoding processes on the "white" regions (407) and "black" regions (409) of the latent variables in frame (405). The depiction of only four latent variables in frame 405 is merely illustrative and not restrictive. Figure 4A What is described can be based solely on the white latent variable (407) and its corresponding... μ and σ The first-stage bitstream (410) is generated by the entropy parameter (412) of stage 1. The second-stage bitstream (430) can be generated based solely on the black latent variable (409); however, the conditions of the second-stage entropy parameter (422) are based on the first-stage entropy parameter (412), the super-latent variable, the time context (if available), and the previous latent variable (black or white).

[0035] Figure 4B An example of a two-stage entropy coding model using entropy skipping is depicted. As depicted, for stage one, given the entropy parameter ( μ , σ (412) and the corresponding threshold (e.g., see equation (9)), calculate the Boolean selection mask (435) in the black latent variables and white latent variables, thereby determining which white latent variables can be entropy skipped.

[0036] Typically, each latent variable has an associated mean and scaling parameter. In the embodiment, for each set of stage latent variables (e.g., in...), Figure 4B In the process, stage 1 consists of latent variables A and D, and stage 2 consists of latent variables B and C. The mean scale (variance) of all latent variables (either within that stage or across all latent variables) is taken. Next, if a latent variable in a stage has a scale higher than the mean scale, that latent variable is encoded; otherwise, it is skipped by entropy. Generally, entropy skipping is never performed on all latent variables in a given stage, but typically most elements within each stage are skipped. As an example, in... Figure 4B In stage 1, based on selection mask 435, latent variable D is entropy skipped, and latent variable A is entropy encoded. In stage 2, based on selection mask 440, only latent variable C is encoded, and latent variable B is entropy skipped.

[0037] The selection mask identifies to the encoder and decoder which values ​​are retained and which are skipped and need to be recalculated. Typically, in each stage, the selection mask value for a particular latent variable is set to zero if the scale value is less than the calculated mean of all variances of the elements in that stage; otherwise, the selection mask value is set to 1, indicating that it was not skipped by entropy. Note that the selection mask does not need to be sent from the encoder to the decoder. The decoder has all the information needed to calculate the mean and scale entropy parameters and the corresponding mask based on the super-latent variables and optional time context, as well as past decoded latent variables. The decoder only needs to replace the skipped latent variables with the estimated mean for each stage.

[0038] Extend this method to quadtree partitioning. Figure 5A A quadtree representation is depicted where the latent space (505) is divided into quadtree representations with quantized latent variable regions (or stages) 1, 2, 3, and 4. In this scenario, latent variable "1" is entropy-encoded first, latent variable "2" is entropy-encoded with updated context information, latent variable "3" is encoded with updated context information, and latent variable "4" is encoded last. Again, depicting only 16 latent variables (505) is illustrative and not restrictive. Once all four latent variable regions are encoded, four bitstreams can be transmitted. When using entropy skipping, the Boolean selection mask (e.g., 435, 440) needs to be adapted to support the new quadtree structure. Figure 5B An example of this selection mask is described, in which, unrestricted, two latent variables in stage "1" are entropy-encoded and the other two latent variables are skipped, or in the (row, column) representation: latent variables (1,1) and (3,3) are encoded, and latent variables (1,3) and (3,1) are skipped. For example, after encoding stage "1", the latent variables of stage "2" are encoded (e.g., latent variables (1,2), (3,2), and (3,4) are encoded (skipping latent variable (1,4)); then the latent variables of stage "3" are encoded (e.g., latent variable (2,1) is encoded (and latent variables (2,3), (4,1), and (4,3) are skipped); finally, all latent variables of stage "4" (e.g., latent variables (2,2), (2,4), (4,2), and (4,4)) are encoded (without entropy skipping).

[0039] In summary, in this type of neural network model, applying the entropy skipping scheme of equation (9) requires calculating the entropy skipping threshold for each stage in the partitioning strategy (e.g., This is due to the gradual refinement of the entropy parameters, which are refined at each stage and conditioned on the previous stage. Utilizing these new scale parameters, a new entropy skip threshold needs to be calculated; otherwise, there will be a mismatch between the actual entropy parameters and the initial entropy parameters used to calculate the selection mask. Figure 6 The process is outlined in the document.

[0040] However, the treatment of latent variables is the same as in the previously described model; that is, for the skipped elements, .

[0041] like Figure 6 The described multi-stage entropy skipping model begins in step 605, where the inputs are quantized latent and superlatent variables and an optional time context (if applicable). As an example, the first stage is denoted as stage [stage name missing]. j = 0 (total) K (Each stage). For example, the division of the chessboard squares. ，K = 2, while for quadtree partitioning ，K = 4 。

[0042] For the stage j = 0, in step 610, calculate the corresponding entropy parameter ( The calculated threshold is then used to select which latent variables to skip and the entropy skip threshold. Then, in step 615, a selection mask is generated based on the calculated threshold to select which latent variables to skip and to generate the encoded bitstream for this stage based on the non-skipped latent variables.

[0043] Subsequently, after the first phase, it is checked whether the entire potential space (all) has been examined. K Each stage is encoded. If not, a stage-specific bitstream is generated as follows: •Based on hyperlatent variables and ( And optionally, the corresponding entropy parameter is calculated based on past latent variables and time context (if available). (Step 620); then in step (625), ○Calculation stage j Update entropy skip threshold ○ Calculate the selection mask to define latent variables that can be skipped by entropy. ○ Use the non-skipped latent variable generation phase j Entropy-encoded bitstream Repeat steps 620 and 625 until all K Each stage is coded.

[0044] In the decoder, the process is exactly the same, except that the computed selection mask is used to regenerate the skipped quantized latent elements by simply assigning the average latent variable value of each stage to the skipped quantized latent elements. The received decoded bitstream and the reconstructed missing elements are then combined to form the final decoded bitstream.

[0045] Experimental results show that training with multi-stage entropy skipping improves the efficiency of the encoder and decoder. The network is trained by minimizing the rate-distortion loss in equation (3), but with slight differences. Before calculating the rate term defined in equation (2), the skipped elements need to be masked to avoid overestimating the rate. This mask... M The selection masks for each stage are combined into a single mask by simply adding them together. The rate term is then calculated as follows.

[0046] (12) References Each of the references listed in this article is cited in its entirety.

[0047] [1]D. Minnen, J. Balle ' And G. Toderici. “Joint autoregressive and hierarchical priors for learned image compression.” 32nd Neural Information Processing Systems Conference (NeurIPS 2018), Montreal, Canada, 2018. [2]D. Liu, Y. Li, J. Lin, H. Li, F. Wu, “Deep learning-based videocoding: A review and a case study.” arXiv:1904.12462v1, April 29, 2019. [3]Y. Shi, Y. Ge, J. Wang, J. Mao, “AlphaVC: High-Performance and Efficient Learned Video Compression,”arXiv:2207.14678, July 29, 2022. [4] Li, Jiahao, Bin Li and Yan Lu. “Hybrid spatial-temporal entropymodelling for neural video compression.” Proceedings of the 30th ACM International Conference on Multimedia [The 30th ACM [Proceedings of the International Multimedia Conference] 2022. [5] Li, Jiahao, Bin Li, and Yan Lu. “Neural video compression with diverse contexts.” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition [IEEE / CVF Computer Vision and Pattern Recognition] [Conference proceedings on pattern recognition] 2023. Example computer system implementation Embodiments of the present invention may be implemented using computer systems, systems configured with electronic circuit systems and components, integrated circuit (IC) devices (such as microcontrollers, field-programmable gate arrays (FPGAs) or other configurable or programmable logic devices (PLDs), discrete-time or digital signal processors (DSPs), application-specific integrated circuits (ASICs)), and / or means including one or more of such systems, devices, or components. The computer and / or IC may make, control, or execute instructions related to skipping entropy encoding of latent features in image and video coding, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to skipping entropy encoding of latent features in image and video coding as described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0048] Some embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors, such as those in a display, encoder, set-top box, transcoder, etc., can implement methods related to skipping entropy encoding of latent features in image and video encoding as described above by executing software instructions in a processor-accessible program memory. Embodiments of the present invention can also be provided in the form of a program product. The program product may include any non-transitory tangible medium carrying a set of computer-readable signals including instructions that, when executed by a data processor, cause the data processor to perform the methods of the present invention. The program product according to the present invention can take any of a variety of non-transitory and tangible forms. The program product may include, for example, physical media, such as magnetic data storage media including floppy disks and hard disk drives, optical data storage media including CD-ROMs and DVDs, electronic data storage media including ROMs and flash RAMs, etc. The computer-readable signals on the program product may optionally be compressed or encrypted. In the case of the components mentioned above (e.g., software modules, processors, components, devices, circuits, etc.), unless otherwise specified, references to these components (including references to “devices”) should be interpreted as including any component that performs the function of the described component as an equivalent of that component (e.g., functionally equivalent), including components that are structurally different from those that perform the functions in the illustrated exemplary embodiments of the invention.

[0049] Equivalents, extensions, alternatives and miscellaneous Therefore, example embodiments related to skipping entropy encoding of latent features in image and video coding are described. In the foregoing description, embodiments of the invention have been described with reference to numerous specific details, which may vary depending on the implementation. Therefore, the sole and exclusive indication of the invention and the applicant's inventive intent is the set of claims published in specific form according to this application, wherein such claim publication includes any subsequent corrections. Any definitions expressly set forth herein with respect to terms contained in such claims shall govern the meaning of such terms as used in the claims. Therefore, any limitations, elements, properties, characteristics, advantages, or attributes not expressly referenced in the claims should not in any way limit the scope of such claims. Therefore, this specification and drawings should be viewed in an illustrative rather than restrictive sense.

[0050] Various aspects of this disclosure can be understood from the following enumerated example embodiments (EEE): EEE1. A method for skipping entropy coding of quantized latent variables in a neural network for video coding, the method comprising: The video content is input into a neural network encoder to generate latent variables. ); The latent variables are quantified to generate quantized latent variables ( ); Estimate the mean of the latent variables ( ) and standard deviation ( );as well as Entropy coding is applied to the quantized latent variables, wherein, for the quantized latent variables ( ), if ,but And skip its entropy encoding, Otherwise, for the quantified latent variables ( Entropy encoding is performed to generate entropy-encoded latent variables, where, Indicates all The average of the values, where ,in, N This represents the total number of latent variables generated.

[0051] EEE2. A method for skipping entropy coding of quantized latent variables in a neural network for video coding, the method comprising: The video content is input into a neural network encoder to generate latent variables. ); The latent variables are quantified to generate quantized latent variables ( ); Estimate the mean of the latent variables ( );as well as Entropy coding is applied to the quantized latent variables, wherein, for latent variables... , if ,but And skip its entropy encoding, Otherwise, for the quantified latent variables ( Entropy encoding is performed to generate entropy-encoded latent variables, where, Represents the threshold, and It represents a small positive value that is greater than zero and less than 0.001.

[0052] EEE3. A method for skipping entropy coding of quantized latent variables in an autoregressive neural network for video coding, the method comprising: The video content is input into a neural network encoder to generate latent variables. ); The latent variables are quantified to generate quantized latent variables ( ); Estimate the initial mean of the latent variables ( ) and initial standard deviation ( ); The initial mean of the latent variables ( ) and initial standard deviation ( This is applied to the entropy skip encoder to determine the updated quantized latent variable value and whether to entropy encode the quantized latent variable value or skip the entropy encoding of the quantized latent variable. Estimate the refined mean of the latent variables based at least on latent variables that have undergone entropy skipping and those that have not. ) and refined standard deviation ( );as well as The refined mean of the latent variables ( ) and refined standard deviation ( Arithmetic encoding and arithmetic decoding of the latent variables to be entropy encoded.

[0053] EEE4. The method as described in EEE3, wherein determining the updated quantized latent variable value and determining whether to entropy encode the quantized latent variable value or skip entropy encoding includes: if ,but And skip its entropy encoding, Otherwise, for Entropy encoding is performed to generate entropy-encoded latent variables. in, Indicates all The average of the values, where ,in, N This represents the total number of latent variables generated.

[0054] EEE5. The method as described in EEE3, wherein determining the updated quantized latent variable value and determining whether to entropy encode the quantized latent variable value or skip entropy encoding includes: For latent variables , if ,but And skip its entropy encoding, Otherwise, for the quantified latent variables ( Entropy encoding is performed to generate entropy-encoded latent variables, where, Represents the threshold, and It represents a small positive value that is greater than zero and less than 0.001.

[0055] EEE6. A method for decoding an image using a neural network, the method comprising: Receive a bit stream that includes encoded latent variables and encoded superlatent variables; The latent variables are decoded to generate prior parameters and initial entropy parameters, wherein the initial entropy parameters include initial mean and initial scale; The initial scale is applied to the entropy skip decoder to identify skipped output latent variables, wherein the skipped output latent variables include latent variables that have not been entropy encoded by the encoder of the bitstream; The value of the skipped output latent variable is determined based on the initial mean; The skipped output latent variables and the super-prior parameters are applied to the context model to generate a refined entropy parameter that includes a refined mean and a refined scale. The refined entropy parameter is applied to decode the encoded latent variable and generate a non-skipped output latent variable; And generate an output decoded image based at least on the values ​​of the skipped output latent variables and the non-skipped output latent variables.

[0056] EEE7. A method for skipping entropy coding of quantized latent variables in a neural network for video coding, the method comprising: The video content is input into a neural network encoder to generate latent variables. ); The latent variables are quantified to generate quantized latent variables ( ); The quantified latent variables are divided into K A set of non-overlapping quantified latent variables; Based on the estimation of hyperlatent variables K Groups in groups j = 0 means of quantified latent variables ( ) and standard deviation ( ); The mean of the latent variables ( ) and standard deviation ( This is applied to the entropy skip encoder to determine the output latent variable value for the group j=0; For the above K In the groups j = 1 to K-1, for group j j Quantified latent variables in: Based on hyperpotential variables and ( The value is used to estimate the group. j The mean of the quantified latent variables mentioned in the text ( ) and standard deviation ( ); The mean of the latent variables ( ) and standard deviation ( The entropy skip encoder is applied to determine the group. j The output encoding latent variable value; and Based on the above K The output of each group encodes the latent variable values ​​to generate a bit stream of encoded and quantized latent variables.

[0057] EEE8. As described in EEE7, wherein, K = 2 or 4.

[0058] EEE9. The method as described in EEE7 or EEE8, wherein determining the output latent variable value and determining whether to entropy encode the quantized latent variable value or to skip entropy encoding includes: if ,but And skip its entropy encoding, Otherwise, for Entropy encoding is performed to generate entropy-encoded output latent variables. in, Representation group j All quantified latent variables in The average value.

[0059] EEE10. A method for decoding an image using a neural network, the method comprising: Receive a bitstream, the bitstream including coded latent variables and coded superlatent variables, wherein the coded latent variables represent those divided into... K Quantified latent variables of non-overlapping groups; For the group of encoded latent variables j =0: The latent variables are decoded to generate prior parameters and initial entropy parameters, wherein the initial entropy parameters include the initial mean ( ) and initial scale ( ); The initial scale is applied to the entropy skip decoder to identify skipped output latent variables, wherein the skipped output latent variables include latent variables that have not been entropy-encoded by the encoder of the bitstream; and Determine the group based on the initial mean. j = 0 for the skipped output latent variable value; For the group of encoded latent variables j ,in j = 1 to K-1: Based on the aforementioned hyperpotential variables and ( The value is used to estimate the group. j The mean of the quantified latent variables mentioned in the text ( ) and standard deviation ( );as well as Apply the estimated mean ( ) and the estimated standard deviation ( This is used to decode the encoded latent variables and generate skipped and non-skipped output latent variables. And at least based on all K The output decoded image is generated by taking the values ​​of the skipped and non-skipped output latent variables from each group.

[0060] EEE11. The method as described in EEE10, wherein determining the decoded skipped quantized latent variables and the decoded non-skipped quantized latent variables includes: if Then he skipped the right one. Entropy encoding, and , Otherwise, once Entropy encoding is performed, and its value is generated by the entropy decoder. in, Representation group j All quantified latent variables in The average value.

[0061] EEE12. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing the method according to any one of EEE1 to EEE11 using one or more processors.

[0062] EEE13. An apparatus comprising a processor and configured to perform the method according to any one of EEE1 to EEE11.

Claims

1. A method for skipping entropy encoding of quantized latent variables in a neural network for video coding, the method comprising: The video content is input into a neural network encoder to generate latent variables. ); The latent variables are quantified to generate quantized latent variables ( ); Estimate the mean of the latent variables ( ) and standard deviation ( );as well as Entropy coding is applied to the quantized latent variables, wherein, for the quantized latent variables ( ), if ,but And skip its entropy encoding, Otherwise, for the quantified latent variables ( Entropy encoding is performed to generate entropy-encoded latent variables, where, Indicates all The average of the values, where ,in, N This represents the total number of latent variables generated.

2. A method for skipping entropy encoding of quantized latent variables in a neural network for video coding, the method comprising: The video content is input into a neural network encoder to generate latent variables. ); The latent variables are quantified to generate quantized latent variables ( ); Estimate the mean of the latent variables ( );as well as Entropy coding is applied to the quantized latent variables, wherein, for latent variables... , if ,but And skip its entropy encoding, Otherwise, for the quantified latent variables ( Entropy encoding is performed to generate entropy-encoded latent variables, where, Represents the threshold, and It represents a small positive value that is greater than zero and less than 0.

001.

3. A method for skipping entropy encoding of quantized latent variables in an autoregressive neural network for video coding, the method comprising: The video content is input into a neural network encoder to generate latent variables. ); The latent variables are quantified to generate quantized latent variables ( ); Estimate the initial mean of the latent variables ( ) and initial standard deviation ( ); The initial mean of the latent variables ( ) and initial standard deviation ( This is applied to the entropy skip encoder to determine the updated quantized latent variable value and whether to entropy encode the quantized latent variable value or skip the entropy encoding of the quantized latent variable. Estimate the refined mean of the latent variables based at least on latent variables that have undergone entropy skipping and those that have not. ) and refined standard deviation ( );as well as The refined mean of the latent variables ( ) and refined standard deviation ( Arithmetic encoding and arithmetic decoding of the latent variables to be entropy encoded.

4. The method of claim 3, wherein, Determining the updated quantized latent variable values ​​and deciding whether to entropy encode the quantized latent variable values ​​or skip entropy encoding includes: if ,but And skip its entropy encoding, Otherwise, for Entropy encoding is performed to generate entropy-encoded latent variables. in, Indicates all The average of the values, where ,in, N This represents the total number of latent variables generated.

5. The method of claim 3, wherein, Determining the updated quantized latent variable values ​​and deciding whether to entropy encode the quantized latent variable values ​​or skip entropy encoding includes: For latent variables , if ,but And skip its entropy encoding, Otherwise, for the quantified latent variables ( Entropy encoding is performed to generate entropy-encoded latent variables, where, Represents the threshold, and It represents a small positive value that is greater than zero and less than 0.

001.

6. A method for decoding an image using a neural network, the method comprising: Receive a bit stream that includes encoded latent variables and encoded superlatent variables; The latent variables are decoded to generate prior parameters and initial entropy parameters, wherein the initial entropy parameters include initial mean and initial scale; The initial scale is applied to the entropy skip decoder to identify skipped output latent variables, wherein the skipped output latent variables include latent variables that have not been entropy encoded by the encoder of the bitstream; The value of the skipped output latent variable is determined based on the initial mean; The skipped output latent variables and the super-prior parameters are applied to the context model to generate a refined entropy parameter that includes a refined mean and a refined scale. The refined entropy parameter is applied to decode the encoded latent variable and generate a non-skipped output latent variable; And generate an output decoded image based at least on the values ​​of the skipped output latent variables and the non-skipped output latent variables.

7. A method for skipping entropy encoding of quantized latent variables in a neural network for video coding, the method comprising: The video content is input into a neural network encoder to generate latent variables. ); The latent variables are quantified to generate quantized latent variables ( ); The quantified latent variables are divided into K A set of non-overlapping quantified latent variables; Based on the estimation of hyperlatent variables K Groups in groups The mean of the quantified latent variables in ( ) and standard deviation ( ); The mean of the latent variables ( ) and standard deviation ( ) is applied to the entropy skip encoder to determine the group The output latent variable values; For the above K Groups in groups Up to K-1, targeting group j Quantified latent variables in: Based on hyperpotential variables and ( The value is used to estimate the group. j The mean of the quantified latent variables mentioned in the text ( ) and standard deviation ( ); The mean of the latent variables ( ) and standard deviation ( The entropy skip encoder is applied to determine the group. j The output encodes the latent variable values; as well as Based on the above K The output of each group encodes the latent variable values ​​to generate a bit stream of encoded and quantized latent variables.

8. The method of claim 7, wherein, K = 2 or 4.

9. The method of claim 7 or 8, wherein, Determining the output latent variable values ​​and deciding whether to entropy encode the quantized latent variable values ​​or skip entropy encoding includes: if ,but And skip its entropy encoding, Otherwise, for Entropy encoding is performed to generate entropy-encoded output latent variables. in, Representation group j All quantified latent variables in The average value.

10. A method for decoding an image using a neural network, the method comprising: Receive a bitstream, the bitstream including coded latent variables and coded superlatent variables, wherein the coded latent variables represent those divided into... K Quantified latent variables of non-overlapping groups; For the group of encoded latent variables : The latent variables are decoded to generate prior parameters and initial entropy parameters, wherein the initial entropy parameters include the initial mean ( ) and initial scale ( ); The initial scale is applied to the entropy skip decoder to identify skipped output latent variables, wherein the skipped output latent variables include latent variables that have not been entropy-encoded by the encoder of the bitstream; and Determine the group based on the initial mean. The value of the skipped output latent variable; For the group of encoded latent variables j ,in To K-1: Based on the aforementioned hyperpotential variables and ( The value is used to estimate the group. j The mean of the quantified latent variables mentioned in the text ( ) and standard deviation ( );as well as Apply the estimated mean ( ) and the estimated standard deviation ( This is used to decode the encoded latent variables and generate skipped and non-skipped output latent variables. And at least based on all K The output decoded image is generated by taking the values ​​of the skipped and non-skipped output latent variables from each group.

11. The method of claim 10, wherein, The determination of decoded skipped quantized latent variables and decoded non-skipped quantized latent variables includes: if Then he skipped the right one. Entropy encoding, and , Otherwise, once Entropy encoding is performed, and its value is generated by the entropy decoder. in, Representation group j All quantified latent variables in The average value.

12. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing the method as described in any one of claims 1 to 11 using one or more processors.

13. An apparatus comprising a processor and configured to perform the method according to any one of claims 1 to 11.

Images