Compression based on variable rate neural networks

Abstract

A mechanism for processing video data is disclosed. It is determined to size an image having a first size to create a resized image having a second size. A conversion between the visual media data and the bitstream is performed based on the resized image. The conversion includes applying a neural network-based coding and decoding model to the resized image to achieve variable rate neural network-based compression.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of International Patent Application No. PCT / CN2022 / 099107, filed on June 16, 2022. All of the above patent applications are incorporated herein by reference in their entireties. Technical Field

[0003] This patent document relates to processing digital images and videos. Background Art

[0004] Digital video accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth requirements for digital video usage are expected to continue to grow. Summary of the invention

[0005] A first aspect relates to a method for processing video data, comprising: determining to resize an image having a first size to create a resized image having a second size; and performing conversion between visual media data and a bitstream based on the resized image, wherein the conversion includes applying a neural network-based codec model to the resized image to achieve variable rate neural network-based compression.

[0006] A second aspect relates to an apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any of the preceding aspects.

[0007] A third aspect relates to a non-transitory computer-readable medium storing a bitstream of a video, wherein the bitstream is generated by a method performed by a video processing device, the method comprising: determining to resize an image having a first size to create a resized image having a second size; and generating the bitstream based on the determination, wherein the generating comprises applying a neural network-based codec model to the resized image to achieve variable rate neural network-based compression.

[0008] A fourth aspect relates to a method for storing a bitstream of a video, comprising: determining to resize an image having a first size to create a resized image having a second size; generating a bitstream based on the determination, wherein the generating comprises applying a neural network-based codec model to the resized image to achieve variable rate neural network-based compression; and storing the bitstream in a non-transitory computer-readable recording medium.

[0009] For clarity, any of the aforementioned embodiments may be combined with any one or more of the other aforementioned embodiments to form new embodiments within the scope of the present disclosure.

[0010]

[0011] These and other features will become more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more fully understand the present disclosure in conjunction with the accompanying drawings and detailed description, reference is now made to the following brief description, wherein like reference numerals represent like parts throughout the drawings.

[0012] Figure 1 is a schematic diagram showing an example transform coding scheme.

[0013] Figure 2 An example latent representation of an image is shown.

[0014] Figure 3 A diagram showing an example autoencoder implementing a hyper-prior model.

[0015] Figure 4 A schematic diagram illustrating an example combined model configured to jointly optimize a context model with a super-prior and an autoencoder.

[0016] Figure 5 An example method of applying padding to an image before and after resizing is shown.

[0017] Figure 6 Example methods of cropping an image before and after resizing are shown.

[0018] Figure 7 A schematic diagram showing an example framework of a variable rate neural network based codec with resizing capability.

[0019] Figure 8 is a block diagram of an example video processing system.

[0020] Fig. 9 is a block diagram of an example video processing device.

[0021] Fig.10 A flow chart of an example method for video processing.

[0022] Fig.11 is a block diagram illustrating an example video codec system.

[0023] Fig.12 is a block diagram showing an example of a video encoder.

[0024] Fig.13 is a block diagram showing an example of a video decoder.

[0025] Fig.14 is a schematic diagram of an example encoder.

[0026] Fig.15 A flow chart of an example method for video processing. DETAILED DESCRIPTION

[0027] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or yet to be developed. The present disclosure should in no way be limited to the illustrative implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the appended claims and their full scope of equivalents.

[0028] This patent document relates to neural network-based image and video compression. The described technology relates to a variable rate neural network-based image and / or video compression mechanism. In an example application scenario, a certain amount of rate is required for optimal transmission efficiency. The present application proposes a mechanism for achieving variable rate by resizing neural network-based image / video compression. The disclosed examples are designed for end-to-end neural network-based systems. The disclosed examples may also be applicable to hybrid compression frameworks in which neural network-based codec tools are integrated into the framework of other video codec standards, such as High Efficiency Video Codec (HEVC), Versatile Video Codec (VVC), or other video codec standards.

[0029] Deep learning is gaining ground in multiple fields, such as computer vision and image processing. Inspired by the successful application of deep learning techniques in computer vision, neural image / video compression techniques are being investigated for their application in image / video compression techniques. Neural networks are designed based on interdisciplinary research in neuroscience and mathematics. Neural networks show strong capabilities in nonlinear transformations and classification. An example neural network-based image compression algorithm achieves rate-distortion (RD) performance comparable to the Versatile Video Codec (VVC), a video codec standard developed by the Joint Video Experts Team (JVET) with experts from the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG). Neural network-based video compression is an actively developing research area, resulting in continuous improvements in neural image compression performance. However, neural network-based video codecs remain a largely unexplored discipline due to the inherent difficulty of the problems that neural networks solve.

[0030] Now let's discuss image and / or video compression. Image / video compression generally refers to a computing technique that compresses video images into binary codes for easy storage and transmission. The binary codes may or may not support lossless reconstruction of the original image / video. Codecs that do not lose data are called lossless compression, while codecs that allow for targeted loss of data are called lossy compression. Most codec systems use lossy compression because lossless reconstruction is unnecessary in most scenarios. Typically, the performance of image / video compression algorithms is evaluated based on the resulting compression ratio and reconstruction quality. The compression ratio is directly related to the number of binary codes produced by the compression, and the fewer the binary codes, the better the compression effect. The reconstruction quality is measured by comparing the reconstructed image / video with the original image / video, and the greater the similarity, the better the reconstruction quality.

[0031] Image / video compression techniques can be divided into video codec methods and neural network-based video compression methods. Video codec schemes use transform-based solutions, where statistical correlations in latent variables (such as discrete cosine transform (DCT) and wavelet coefficients) are used to carefully hand-design entropy codecs to model correlations in quantization mechanisms. Neural network-based video compression can be divided into neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded in the relevant video codec as a codec tool and is only part of the framework, while the latter is an independent framework developed based on neural networks and does not depend on the video codec.

[0032] A range of video codec standards have been developed to accommodate the growing demand for visual content delivery. The International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) has two expert groups, the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG). The International Telecommunication Union (ITU) Telecommunication Standardization Sector (ITU-T) also has a Video Codec Experts Group (VCEG) responsible for the standardization of image / video codec technologies. Influential video codec standards released by these organizations include the Joint Photographic Experts Group (JPEG), JPEG 2000, H.262, H.264 / Advanced Video Codec (AVC), and H.265 / High Efficiency Video Codec (HEVC). The Joint Video Experts Group (JVET), consisting of MPEG and VCEG, developed the Versatile Video Codec (VVC) standard. VVC is reported to reduce bitrate by an average of 50% compared to HEVC at the same visual quality.

[0033] Neural network-based image / video compression / codecs are also being developed. Example neural network codec network architectures are relatively shallow, and the performance of such networks is unsatisfactory. Neural network-based approaches benefit from the support of abundant data and powerful computing resources, and are therefore better utilized in various applications. Neural network-based image / video compression has shown promising improvements and has been proven to be feasible. However, this technology is far from mature and there are still many challenges to be addressed.

[0034] Neural networks, also known as artificial neural networks (ANNs), are computational models used in machine learning techniques. Neural networks are typically composed of multiple processing layers, and each layer consists of multiple simple but nonlinear basic computational units. One benefit of such deep networks is the ability to process data with multiple levels of abstraction and transform the data into different kinds of representations. The representations created by neural networks are not manually designed. Instead, the deep networks comprising the processing layers are learned from massive amounts of data using general machine learning procedures. Deep learning eliminates the necessity for handcrafted representations. Therefore, deep learning is considered to be very useful, especially for processing natural unstructured data such as sound and visual signals. The processing of such data has been a long-standing problem in the field of artificial intelligence.

[0035] Neural networks for image compression can be divided into two categories, including pixel probability models and autoencoder models. Pixel probability models use a predictive encoding and decoding strategy. Autoencoder models use a transformation-based solution. Sometimes, these two methods are combined together.

[0036] Now let's discuss pixel probability modeling. According to Shannon's information theory, the best method for lossless codec can achieve the minimum codec rate, which is expressed as -log2p(x), where p(x) is the probability of symbol x. Arithmetic coding is a lossless codec method that is considered to be one of the best methods. Given the probability distribution p(x), arithmetic coding makes the codec rate as close to the theoretical limit -log2p(x) as possible, without considering rounding errors. Therefore, the remaining problem is to determine the probability, which is very challenging for natural images / videos due to the dimensionality disaster. The dimensionality disaster refers to the problem that the data set becomes sparse due to the increase in dimensions, so as the number of dimensions increases, the amount of data needs to increase rapidly in order to effectively analyze and organize the data.

[0037] Following the predictive coding strategy, one way to model p(x) is to predict pixel probabilities one by one in raster scan order based on previous observations, where x is an image, which can be expressed as follows:

[0038] p(x)=p(x1)p(x2|x1)…p(x i |x1,…,x i-1 )…p(xm×n |x1,…,x m×n-1 ) (1)

[0039] Where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, the estimation of conditional probability is difficult. Therefore, a simplified approach is to limit the scope of the context of the current pixel as follows:

[0040] p(x)=p(x1)p(x2|x1)…p(x i |x i-k ,…,x i-1 )…p(x m×n |x m×n-k ,…,x m×n-1 ) (2)

[0041] where k is a predefined constant that controls the scope of the context.

[0042] It should be noted that the condition may also consider the sample values ​​of other color components. For example, when encoding and decoding red (R, green (G), and blue (B)) (RGB) color components, the R sample depends on the previously encoded and decoded pixels (including R, G, and / or B samples), and the current G sample may be encoded and decoded based on the previously encoded and decoded pixels and the current R sample. In addition, when encoding and decoding the current B sample, the previously encoded and decoded pixels and the current R and G samples may also be considered.

[0043] Neural networks can be designed for computer vision tasks and can also be effectively used for regression and classification problems. Therefore, given the context x1,x2,…,x i-1 In the case of i ). In the example neural network design, according to x i ∈{-1,+1}, the pixel probabilities are used for binary images. The Neural Autoregressive Distribution Estimator (NADE) is designed for pixel probability modeling. NADE is a feed-forward network with a single hidden layer. In another example, the feed-forward network can include connections that skip the hidden layer. In addition, parameters can also be shared. This neural network is used to perform experiments on the binarized Modified National Institute of Standards and Technology (MNIST) dataset. In one example, NADE is extended to a real-valued NADE (RNADE) model, where the probability p(x i |x1,…,x i-1) is obtained by a mixture of Gaussian distributions. The RNADE model feed-forward network also has a single hidden layer, but the hidden layer is rescaled to avoid saturation and uses rectified linear units (ReLU) instead of sigmoid. In another example, NADE and RNADE are improved by using reorganized pixel order and deeper neural networks.

[0044] Designing advanced neural networks plays an important role in improving pixel probability modeling. In the example neural network, a multidimensional long short-term memory (LSTM) is used. LSTM is used for probability modeling together with a mixture of conditional Gaussian scaled mixtures. LSTM is a special recurrent neural network (RNN) and can be used to model sequence data. The spatial variant of LSTM can also be used for images. Several different neural networks can be adopted, including recurrent neural networks (RNN) and convolutional neural networks (CNN), such as pixel RNN (PixelRNN) and pixel CNN (PixelCNN), respectively. In PixelRNN, two variants of LSTM are adopted, denoted as row LSTM and diagonal bidirectional LSTM (BiLSTM). Diagonal BiLSTM is specially designed for images. PixelRNN combines residual connections to help train deep neural networks with up to twelve layers. In PixelCNN, masked convolution is used to adjust the shape of the context. PixelRNN and PixelCNN are more dedicated to natural images. For example, PixelRNN and PixelCNN treat pixels as discrete values ​​(e.g., 0, 1, ..., 255) and predict a multinomial distribution of discrete values. In addition, PixelRNN and PixelCNN process color images in RGB color space. In addition, PixelRNN and PixelCNN work well on the large-scale image dataset ImageNet. In one example, Gated PixelCNN is used to improve PixelCNN. Gated PixelCNN achieves comparable performance to PixelRNN, but with much lower complexity. In one example, PixelCNN++ is used, which has the following improvements over PixelCNN: using discrete logistic mixed likelihood instead of 256-way multinomial distribution; using downsampling to capture structures at multiple resolutions; introducing additional shortcut connections to speed up training; using dropout for regularization; and, for one pixel, combining RGB. In another example, PixelSNAIL combines causal convolution with a self-attention mechanism.

[0045] Most of the above methods model the probability distribution directly in the pixel domain. Some designs also model the probability distribution as conditional based on explicit or latent representations. Such a model can be expressed as:

[0046]

[0047] Where h is an additional condition, and p(x)=p(h)p(x|h) indicates that modeling is divided into unconditional model and conditional model. The additional condition can be image label information or high-level representation.

[0048] An autoencoder is now described. An autoencoder is trained for dimensionality reduction and includes an encoding component and a decoding component. The encoding component converts a high-dimensional input signal into a low-dimensional representation. The low-dimensional representation may have a reduced spatial size but a greater number of channels. The decoding component recovers the high-dimensional input from the low-dimensional representation. The autoencoder enables automatic learning of representations without the need for handcrafted features, which is considered one of the most important advantages of neural networks.

[0049] Figure 1 is a schematic diagram showing an example transform coding scheme 100. The original image x is analyzed by the network g a The latent representation y is quantized q and compressed into bits. The number of bits R is used to measure the encoding and decoding rate. Then, the synthesis network g s Potential representation of quantization Perform inverse transform to obtain the reconstructed image The distortion D is calculated in the perceptual space by using the function g p Transform x and Get z and And compare them to get D.

[0050] Autoencoder networks can be applied to lossy image compression. The learned latent representation can be encoded from a well-trained neural network. However, adapting autoencoders to image compression is non-trivial, as the original autoencoders are not optimized for compression and therefore cannot be effectively used directly as trained autoencoders. In addition, there are other significant challenges. First, the low-dimensional representation should be quantized before being encoded. However, quantization is non-differentiable, which is required for backpropagation when training neural networks. Second, the goals in the compression scenario are different, as both distortion and rate need to be considered. Estimating this rate is challenging. Third, practical image coding and decoding schemes should support variable rates, scalability, encoding / decoding speed, and interoperability. Various schemes are being developed to address these challenges.

[0051] The example autoencoder for image compression using the example transform coding scheme 100 can be viewed as a transform coding strategy. a (x) transforms the original image x, where y is the latent representation to be quantized and decoded. The synthesis network transforms the quantized latent representation Inverse transform back to get the reconstructed image The framework is trained using the rate-distortion loss function, where D is x and The distortion between them, R is the quantized representation is the rate of computation or estimation, and λ is the Lagrange multiplier. D can be computed in the pixel domain or the receptive domain. Most example systems follow this archetype, and the difference between such systems may only be the network structure or the loss function.

[0052] In terms of network structure, RNN and CNN are the most widely used architectures. In the RNN-related category, the sample general framework for variable rate image compression uses RNN. The sample uses binary quantization to generate codes and does not consider the rate during training. The framework provides scalable encoding and decoding functions, in which RNNs with convolutional and deconvolutional layers perform well. Another example provides an improved version that compresses binary codes by upgrading the encoder with a neural network similar to PixelRNN. The performance is better than JPEG on the Kodak image dataset using the multi-scale structural similarity (MS-SSIM) evaluation metric. Another example further improves the RNN-based solution by introducing a hidden state startup mechanism. In addition, an SSIM weighted loss function is designed and a spatial adaptive bitrate mechanism is included. This example uses MS-SSIM as an evaluation metric on the Kodak image dataset and obtains better results than Better Portable Graphics (BPG). In another example, the system supports spatial adaptive bitrate by training an RNN that tolerates stop codes.

[0053] Another example proposes a general framework for rate-distortion optimized image compression. The example system uses multi-base quantization to generate integer codes and considers the rate during training. The loss is a joint rate-distortion cost, which can be a mean square error (MSE) or other metric. The example system adds random uniform noise during training to stimulate quantization and uses the differential entropy of the noise code as a surrogate for the rate. The example system uses generalized division normalization (GDN) as a network structure, which includes a linear mapping followed by a nonlinear parameter normalization. The effectiveness of GDN for image encoding and decoding is verified. Another example system includes an improved version that uses three convolutional layers, each followed by a downsampling layer and a GDN layer as a forward transform. Accordingly, the example version uses three layers of inverse GDN, each followed by an upsampling layer and a convolutional layer to simulate an inverse transform. In addition, an arithmetic coding method is designed to compress integer codes. It is reported that on the Kodak dataset, its performance is better than JPEG and JPEG2000 in terms of MSE. Another example improves the method by designing a scale hyper-prior in the autoencoder. The system will have subnet h a The potential representation y is transformed into z = h a(y), and z is quantized and sent as auxiliary information. Therefore, using subnet h s To achieve the inverse transformation, this subnetwork obtains the quantized auxiliary information Decode to quantized The standard deviation of is further used during arithmetic coding and decoding of . On the Kodak image set, this method is slightly worse than BGP in terms of peak signal-to-noise ratio (PSNR). Another example system further exploits the structure in the residual space by introducing an autoregressive model to estimate both the standard deviation and the mean. This example uses a Gaussian mixture model to further remove redundancy in the residual. The performance of the Kodak image set using PSNR as the evaluation metric is comparable to VVC.

[0054] Figure 2 An example latent representation of an image is shown. Figure 2 It includes an image 201 from the Kodak dataset, a visualization of the potential 202 representation y of the image 201, the standard deviation σ203 of the potential 202, and the potential y204 after the introduction of the super prior network. The super prior network includes a super encoder and a super decoder. Now we discuss the super prior model. In the transform coding method of image compression, such as Figure 1 As shown, the encoder subnetwork uses parameter analysis transformation The image vector x is transformed into a latent representation y, which is then quantized to form because is a discrete value, so Entropy coding and decoding techniques (such as arithmetic coding and decoding) can be used for lossless compression and sent as a bit sequence.

[0055] from Figure 2 It can be clearly seen that the potential 202 and standard deviation σ203 There is a significant spatial correlation between the elements of . It is noteworthy that their scales (standard deviation σ203) appear to be spatially coupled. An additional set of random variables can be introduced to capture spatial correlation and further reduce redundancy. In this case, the image compression network is Figure 3 shown.

[0056] Figure 3 300 is a diagram showing an example network architecture of an autoencoder implementing a super prior model. The upper side shows the image autoencoder network, and the lower side corresponds to the super prior subnet. The analysis transform and the synthesis transform are denoted as g a and g aQ represents quantization (Q), and AE and AD represent arithmetic encoder (AE) and arithmetic decoder (AD), respectively. The super prior model consists of two sub-networks, the super encoder (denoted by h) and the super decoder (denoted by h s denoted by ). The hyper-prior model generates a quantized hyper-latent It includes the potential Information related to the probability distribution of the sample points. Included in the bitstream and with are sent together to the receiver (decoder)

[0057] In the schematic diagram 300, the upper side of the model is the encoder g as described above. a and decoder g s The lower side is used to obtain The additional super encoder h a and super decoder h s In this architecture, the encoder subjects the input image x to g a , producing a response y with a spatially varying standard deviation. The response y is fed into h a , summarizing the distribution of standard deviations in z. z is then quantized is compressed and sent as side information. The encoder then uses the quantized vector to estimate the spatial distribution of the standard deviation σ, and use σ to compress and send the quantized image representation The decoder first recovers the compressed signal The decoder then uses h s to obtain σ, which provides the decoder with the correct probability estimate to successfully recover The decoder will then Feed g s to obtain the reconstructed image.

[0058] When the super encoder and super decoder are added to the image compression network, the quantized latent The spatial redundancy of is reduced. When using super encoder / super decoder, Figure 2 The potential y204 in corresponds to the quantized potential value. Compared with the standard deviation σ203, the spatial redundancy is significantly reduced because the samples of the quantized potential are less correlated.

[0059] Now let’s discuss the context model. Although the hyper-prior model improves the quantization potential , but additional improvements can be obtained by leveraging autoregressive models that quantize the latents from their causal context predictions, which can be referred to as context models. The term “autoregressive” indicates that the output of a process is later used as the input of this process. For example, the context model subnetwork generates a latent sample that is later used as input to obtain the next sample.

[0060] Figure 4 4 is a schematic diagram showing an example combined model configured to jointly optimize a context model with a hyper-prior and an autoencoder. The combined model jointly optimizes an autoregressive component that estimates a latent probability distribution based on their causal context (context model) as well as the hyper-prior and the underlying autoencoder. The real-valued latent representation is quantized (Q) to create a quantized latent and quantified super potential They are compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The dashed area corresponds to the components performed by a receiver (eg, a decoder) to recover the image from the compressed bitstream.

[0061] The example system utilizes a joint architecture where both the super-prior model sub-network (super-encoder and super-decoder) and the context model sub-network are utilized. The super-prior and context models are combined to learn the quantized latent The probability model is then used for entropy encoding and decoding. As shown in schematic diagram 400, the outputs of the context sub-network and the super decoder sub-network are combined by a sub-network called the entropy parameter, which generates the mean μ and scale (or variance) σ parameters of the Gaussian probability model. The quantized latent samples are then encoded into a bitstream with the help of the arithmetic encoder (AE) module using the Gaussian probability model. In the decoder, the arithmetic decoder (AD) module uses the Gaussian probability model to obtain the quantized latent samples from the bitstream.

[0062] In one example, the potential sample is modeled as a Gaussian distribution or a Gaussian mixture model (without limitation). In the example according to schematic diagram 400, the context model and the hyper-prior are jointly used to estimate the probability distribution of the potential sample. Since the Gaussian distribution can be defined by a mean and a variance (also known as sigma or scale), the joint model is used to estimate the mean and variance (expressed as μ and σ).

[0063] Now let's discuss image compression based on variable rate neural networks. Some methods can implement variable rate image compression using neural networks. In one example, a fully connected autoencoder paired with a recurrent neural network (RNN) with variable rate support can be used. To implement variable rate, the RNN can be trained only once, but multiple iterations can be used during the testing phase. In each iteration, the network encodes the residual between the output of the last iteration and the original image.

[0064] In another example, conditional autoencoders can be used to implement variable rate control. In the conditional autoencoder framework, two rate control parameters are used, including Lagrange multipliers and quantization bin size. The former is used to implement coarse rate adjustment, while the latter is used to adjust the rate at a finer level. Multiple combinations of these two parameters are used during the training phase.

[0065] In one example, a continuously variable rate neural network based image compression is achieved by employing a variable gain autoencoder. In this framework, a gain unit is placed after the output of the encoder, while an inverse gain unit is placed before the input of the decoder. The gain unit is a set of trainable vectors applied to feature maps before entropy encoding and decoding them.

[0066] In one example, a transform based on a wavelet-like neural network may be used. This example uses a trainable transform based on a wavelet-like neural network. This transform converts the image into coefficients without any information loss, providing the capabilities of both lossy and lossless codecs. In lossy codecs, the coefficients are optionally encoded.

[0067] We now discuss the use of neural networks for video compression. Similar to video codec technology, in neural network-based video compression, neural image compression serves as the basis for intra-frame compression. Therefore, the development of neural network-based video compression technology lags behind the development of neural network-based image compression because neural network-based video compression technology is more complex and therefore requires more efforts to address the corresponding challenges. Compared with image compression, video compression requires effective methods to eliminate redundancy between images. Inter-image prediction is the main step in these example systems. Motion estimation and compensation are widely adopted in video codecs, but are usually not implemented by trained neural networks.

[0068] Neural network-based video compression can be divided into two categories depending on the target scenario: random access and low latency. In the case of random access, the system allows decoding to start from any point in the sequence, usually dividing the entire sequence into multiple separate segments and allowing each segment to be decoded independently. In the case of low latency, the system aims to reduce the decoding time so that the temporally previous frame can be used as a reference frame for decoding subsequent frames.

[0069] Now let's discuss the low-latency system. An example system uses a video compression scheme with a trained neural network. The system first divides the video sequence frames into blocks, and each block is encoded or decoded according to the intra-frame codec mode or the inter-frame codec mode. If the intra-frame codec is selected, there is an associated autoencoder to compress the block. If the inter-frame codec is selected, motion estimation and compensation are performed, and the trained neural network is used for residual compression. The output of the autoencoder is directly quantized and encoded by the Huffman method.

[0070] Another neural network-based video codec scheme uses PixelMotionCNN. Frames are compressed in temporal order, and each frame is divided into blocks compressed in raster scan order. Each frame is first extrapolated with the previous two reconstructed frames. When a block is to be compressed, the extrapolated frame is fed into PixelMotionCNN along with the context of the current block to derive a potential representation. The residual is then compressed by a variable rate image scheme. The performance of this scheme is comparable to H.264.

[0071] Another example system uses an end-to-end neural network based video compression framework, where all modules are implemented with neural networks. The scheme accepts the current frame and the previously reconstructed frame as input. The optical flow is derived using a pre-trained neural network as motion information. The motion information is warped with a reference frame, and then a motion compensated frame is generated by the neural network. The residual and motion information are compressed using two independent neural autoencoders. The entire framework is trained with a single rate-distortion loss function. This example system achieves better performance than H.264.

[0072] Another example system adopts a video compression scheme based on advanced neural networks. This system inherits and extends the video encoding and decoding scheme with neural networks, with the following main features. First, the system uses only one autoencoder to compress motion information and residuals. Second, the system uses motion compensation with multiple frames and multiple optical flows. Third, the system uses online states that are learned and propagated through subsequent frames over time. This scheme achieves better performance than the HEVC reference software in MS-SSIM.

[0073] Another example system uses an extended end-to-end neural network-based video compression framework. In this example, multiple frames are used as references. The example system is thus able to provide a more accurate prediction of the current frame by using multiple reference frames and associated motion information. In addition, motion field prediction is employed to eliminate motion redundancy along the temporal channel. A post-processing network is also used to remove reconstruction artifacts from previous processing. The system performs significantly better than H.265 in terms of both PSNR and MS-SSIM.

[0074] Another example system uses scale-space flow instead of optical flow by adding a frame-based scale parameter. This example system can achieve better performance than H.264. Another example system uses multi-resolution representation based on optical flow. Specifically, the motion estimation network generates multiple optical flows with different resolutions and lets the network learn which one to choose under the loss function. The performance is slightly better than H.265.

[0075] We now discuss systems that employ random access. Another example system uses a neural network-based video compression scheme with frame interpolation. Key frames are compressed first with a neural image compressor, while the remaining frames are compressed in a hierarchical order. The system performs motion compensation in the perceptual domain by deriving feature maps at multiple spatial scales of the original frame and using motion to warp the feature maps. The result is used in an image compressor. The approach is comparable to H.264.

[0076] The example system uses an interpolation-based video compression approach. The interpolation model combines motion information compression and image synthesis. The same autoencoder is used for both images and residuals. Another example system uses a neural network-based video compression approach based on a variable autoencoder with a deterministic encoder. Specifically, the model includes an autoencoder and an autoregressive prior. Unlike previous methods, this system accepts a group of images (GOP) as input and incorporates a three-dimensional (3D) autoregressive prior by considering temporal correlations when encoding and decoding the latent representation. The system provides performance comparable to H.265.

[0077] Now let’s discuss the preliminary situation. Almost all natural images and / or videos are in digital format. Grayscale digital images can be Indicates that is a set of pixel values, m is the image height, and n is the image width. For example, is an example setup, in which case Therefore, a pixel can be represented by an 8-bit integer. An uncompressed grayscale digital image has 8 bits per pixel (bpp), while the number of bits for compression is certainly less.

[0078] Color images are usually represented in multiple channels to record color information. For example, in the RGB color space, an image can be represented by Representation, with three independent channels storing red, green, and blue information. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image has 24bpp. Digital images / videos can be represented in different color spaces. Neural network-based video compression schemes are mainly developed in the RGB color space, while video codecs typically use the YUV color space to represent video sequences. In the YUV color space, the image is decomposed into three channels, namely luminance (Y), blue-difference chrominance (Cb), and red-difference chrominance (Cr). Y is the luminance component, and Cb and Cr are the chrominance components. Since the human visual system is less sensitive to the chrominance components, Cb and Cr are usually downsampled for pre-compression, resulting in compression benefits for YUV.

[0079] A color video sequence consists of multiple color images (also called frames) that record scenes at different timestamps. For example, in the RGB color space, a color video can consist of X = {x0, x1, …, x t ,…,x T-1}, where T is the number of frames in the video sequence, If m=1080, n=1920, And the video has 50 frames per second (fps), then the data rate of this uncompressed video is 1920×1080×8×3×50=2,488,320,000 bits per second (bps). This will result in about 2.32 gigabits per second (Gbps), which will take up a lot of storage, so it should be compressed before transmitting over the Internet.

[0080] For natural images, lossless methods can usually achieve a compression ratio of about 1.5 to 3, which is obviously lower than the requirement for streaming. Therefore, lossy compression is used to achieve better compression ratios, but at the expense of inducing distortion. Distortion can be measured by calculating the mean square difference between the original image and the reconstructed image, for example based on MSE. For grayscale images, MSE can be calculated by the following formula.

[0081]

[0082] Therefore, the quality of the reconstructed image compared to the original image can be measured by the Peak Signal-to-Noise Ratio (PSNR):

[0083]

[0084] in yes The maximum value in , for example, for an 8-bit grayscale image is 255. There are other quality assessment metrics such as structural similarity (SSIM) and multi-scale SSIM (MS-SSIM).

[0085] To compare different lossless compression schemes, one can compare the compression ratio for a given resulting rate, and vice versa. However, to compare different lossy compression methods, the comparison must take into account both the rate and the quality of the reconstruction. For example, this can be achieved by calculating the relative rates for several different quality levels and then averaging these rates. The average relative rate is called Bjontegaard's delta rate (BD-rate). There are other aspects of evaluating image and / or video codec schemes, including encoding / decoding complexity, scalability, robustness, etc.

[0086] The following are example technical problems solved by the technical solutions of the present disclosure. The network architecture of the neural network-based codec should be changed to accommodate variable rate adjustment. The examples described above all use a dedicated network architecture, which makes it impossible to add variable rate adjustment when such a network has been well trained. In addition, the above network can only handle rate adjustment within a limited range. When the rate adjustment range is extended, there will be a significant rate-distortion performance degradation. For example, multiple models trained with different Lagrangian multipliers are still needed to achieve rate adjustment over a wide range using conditional autoencoders. In addition, using certain variable rate solutions, the training strategy may be more complicated. For example, using a conditional autoencoder requires that the network be trained with randomly sampled Lagrangian multipliers and box size pairs. The loss function is also the average loss of several iterations including multiple Lagrangian multipliers and box size pairs. These factors may hinder use in practical applications.

[0087] To solve the above and other problems, methods summarized as follows are disclosed. These items should be considered as examples to explain general concepts and should not be interpreted in a narrow way. In addition, these items can be used alone or in combination in any way. The technology described in this article provides an image and video compression method based on a variable rate neural network using resizing. The goal is to provide the flexibility of variable rates without changing the network structure, for example, so that such a mechanism can also be used when a fixed rate model has been trained. Additional benefits also include: no retraining is required, and rate adjustment can be achieved over a wide range. Specifically, the present disclosure describes adjusting the input image to different sizes for different target rates, rather than using different network architectures. In the following discussion, frames, pictures, and images can have the same meaning.

[0088] Example 1

[0089] In one example, a decoded image / video encoded at a first size may be resized (eg, upsampled or downsampled) to a second size.

[0090] Example 2

[0091] In one example, the image / video having the second size may be resized in a recovery manner (eg, down-sampled or up-sampled) before being encoded and decoded.

[0092] Example 3

[0093] In one example, the encoder / decoder uses at least one NN-based model.

[0094] Example 4

[0095] In one example, the size information may include width and height. In one example, the width and / or height may be counted in samples of a component (e.g., a luma component). In one example, the width and / or height may be counted in N (e.g., N=4 or 16) samples of a component (e.g., a luma component).

[0096] Example 5

[0097] In one example, size information of the first size and / or the second size may be signaled to the decoder. In one example, size information of the first size and the second size may be signaled independently. In one example, size information of the first size and the second size may be signaled in a predictive manner. In one example, size information of the first size may be signaled, and a ratio between the first size and the second size may be signaled. In one example, size information of the second size may be signaled, and a ratio between the first size and the second size may be signaled. In one example, the resized width and height may be proportional to their original width and height. In one example, the ratio of the width and height may be the same.

[0098] Figure 5 An example method 500 for applying padding to an image before and after resizing is shown. In one example, padding may be applied before / after resizing, such as Figure 5 shown. Figure 6 An example method 600 for cropping an image before and after resizing is shown. In one example, as shown in method 600, sample cropping can be applied before / after resizing. In one example, a signaling message can be signaled to indicate whether the image / video should be resized. In one example, size information of a first size can be signaled only when the image / video should be resized. In one example, size information of a second size can be signaled only when the image / video should be resized. In one example, a size ratio can be signaled only when the image / video should be resized.

[0099] Example 6

[0100] In one example, the size information may be signaled via at least one index. The index may refer to a predefined size, width, height, and / or ratio.

[0101] Example 7

[0102] In one example, how and / or whether to perform resizing may depend on the color component or color format (such as YUV444 or YUV 420 or RGB). In one example, different color components may have different resizing methods. For example, the luminance component may be encoded and decoded using resizing, but the chrominance components may be encoded and decoded without resizing. In one example, different components of a picture may be resized using different size parameters. For example, a first size parameter may be used to adjust the size of the luminance component, and a second size parameter may be used to adjust the size of the chrominance component. The picture may be a reconstructed output picture (output of the decoding process) or an input image. The size parameter may include a width value, a height value, or a combination of both.

[0103] Example 8

[0104] In one example, resizing is performed using a filter.

[0105] Example 9

[0106] In one example, the filter may be a separable filter, in which a horizontal filter and a vertical filter are applied to achieve resizing. The filter may be a lanczos filter. The filter may be a bilinear filter or a bicubic filter.

[0107] Example 10

[0108] In one example, the type of filter used for resizing may be indicated in the bitstream. The length of the filter may be indicated in the bitstream. The length of the filter may be determined by the number of taps of the filter. The length of the filter may be defined as the length of the filter in one of the horizontal or vertical directions. In one example, at least two different types of filters may be indicated in the bitstream. The first filter may be used for one component of the reconstructed output picture, and the second filter may be applied to a second component of the reconstructed output picture. The type of filter may determine the length of the filter, or the type of filter may determine the coefficients of the filter.

[0109] An index may be indicated in a bitstream to determine which filter in a set of filters is used for resizing. In one example, an indication may be obtained from a bitstream to determine which filter in a set of filters is used for resizing. The first value of the indication may indicate that a 3-tap filter (bicubic filter) is applied for resizing. The second value of the indication may indicate that an 8-tap filter is applied for resizing. Although 3 taps and 8 taps are provided as examples, the present disclosure is not limited to the examples provided. In another example, the indication may be used to determine which weight values ​​are used for filtering.

[0110] Example 11

[0111] In one example, the filter can be implemented as a convolutional layer or a deconvolutional layer of a neural network.

[0112] Example 12

[0113] In one example, a set of candidate models that may be used for encoding / decoding image / video signaling may depend on whether and / or how the image / video is resized before and / or after the encoding / decoding process.

[0114] Example 13

[0115] In one example, a model may be determined based on whether and / or how the image / video is resized before and / or after the encoding and decoding process.

[0116] Example 14

[0117] In one example, the multiple models may be determined based on whether and / or how the image / video size is adjusted before and / or after the encoding and decoding process. In one example, at least one of the multiple models may be selected and signaled to the decoder. In one example, at least one of the multiple models may be derived by the decoder. In one example, the multiple models may correspond to multiple bit rates.

[0118] Example 15

[0119] In one example, one or more models are pre-trained with different Lagrangian multipliers. Given a target rate, the original input image is resized to multiple sizes to obtain multiple candidates. Depending on the target bitrate range, only one model may be trained. Depending on the target bitrate range, multiple models may be trained.

[0120] The original input image may be downsampled. In one example, the resized width and height may be proportional to the original width and height. In one example, the resized width and height may not be proportional to its original width and height.

[0121] In one example, the original image may be upsampled. In one example, the resized width and height may be proportional to the original width and height. In one example, the resized width and height may not be proportional to its original width and height.

[0122] To obtain the resized candidates, different strategies can be used. In one example, the network can reduce or increase the width or height by a certain number of pixels. In one example, reduce or increase the total number of pixels. In one example, the network can reduce or increase the ratio of the resized size to the original image size. The ratio can be the width, height, or total number of pixels.

[0123] Some conditions that should be followed when obtaining candidates can be defined. In one example, 64 times the number of pixels along both width and height (this number depends on the network architecture, specifically, on the number of downsampling convolutions) are required to eliminate possible boundary artifacts. In one example, 32 times the number of pixels along both width and height (half the downsampling ratio of the convolution) are required to eliminate possible boundary artifacts.

[0124] Different filters can be used during upsampling or downsampling. In one example, filters such as lanczos filters, nearest neighbor filters, bicubic filters, bilinear filters, etc. can be used. In one example, a neural network-based filter can be used. For example, a deep filter for image super-resolution can be used. In one example, the encoder and decoder can use the same type of filter, such as both using lanczos filters. In one example, different types of filters can be used on the encoder side and the decoder side. For example, the encoder uses a lanczos filter, and the decoder uses a bicubic filter.

[0125] Example 16

[0126] In one example, additional parameters may be encoded in the bitstream. In one example, the resized width and height may be encoded into the bitstream. In one example, an index of the resized size may be encoded into the bitstream, and both the encoder and decoder may have a predefined list of resized sizes. In one example, at least one filter index may be encoded into the bitstream, and the decoder may have a list of predefined filter pairs.

[0127] Example 17

[0128] In one example, a rate-distortion optimization (RDO) process is used to select the one with the best rate-distortion (RD) performance from all candidates.

[0129] Example 18

[0130] In one example, during the RDO process, the anchor result can be used as a reference for calculating the RD loss. In one example, the results of a codec (e.g., Versatile Video Test Model (VTM), Better Portable Graphics (BPG), Joint Photographic Experts Group (JPEG), JPEG 2000, etc.) can be used as an anchor point. In one example, the anchor result can be the result of a model obtained by using a fixed quantization parameter (QP) without using any resizing strategy. In one example, the anchor result can be obtained by selecting a candidate with the best specific metric (e.g., Multi-Scale Structural Similarity (MS-SSIM) or Peak Signal-to-Noise Ratio (PSNR) for an image quality assessment model) for a certain rate point.

[0131] Example 19

[0132] In one example, in the RDO process, the RD loss can be a single metric or multiple metrics mixed in any manner.

[0133] Example 20

[0134] In one example, in the RDO process, the RD loss can be calculated at a single rate point or multiple rate points. In one example, there are four rate points in total. When calculating the RD loss, three rate points can be fixed. Then, the RDO process can traverse all candidates in the fourth rate point and finally select the best one as the fourth rate point. In one example, there are four rate points in total, and each rate point has ten candidates. The RDO process can traverse all combinations (10 4 combination) to obtain the combination with the minimum RD loss.

[0135] Example 21

[0136] In one example, the messages disclosed in this document may be signaled as a flag, a truncated unary code, a truncated binary code, an exponential Golomb code, a fixed length code, or a combination thereof.

[0137] Example 22

[0138] In one example, the messages disclosed in this document may be encoded or decoded using at least one arithmetic codec context. In one example, the messages disclosed in the document may be encoded or decoded using a bypass codec.

[0139] Example embodiments are now described. Example variable rate neural network-based solutions may include dedicated network architectures and may also require special training strategies. This may hinder the use of such networks in practical applications. In the present disclosure, resizing is used to achieve variable rate based neural network image and video compression. A set of pre-trained models trained with different Lagrangian multipliers are used. The training Lagrangian multiplier values ​​are determined by the target bit rate range. When the set of pre-trained models is ready, the input image is resized to different sizes, and the RD loss of each pre-trained model is used to select the candidate with the minimum RD loss. Examples are provided in the following subsections.

[0140] The example framework is as follows. Figure 7 Schematic diagram 700 shows an example framework of a variable rate neural network based codec with resizing function. A plurality of candidate images are obtained by resizing the original image. The rate distortion loss is used to select the one with the minimum RD loss.

[0141] Thus, an example of the disclosed examples is depicted in diagram 700. Let x be the original input image, y be the output of the analysis transformation, is the reconstructed image. The resized image from x with the (k+1)th pre-trained model is represented as Where k = 0, 1, ..., N-1 (N is the total number of pre-trained models), i = 0, 1, ..., R k -1(R k is the total number of resized candidates from the kth model). After obtaining all candidate models from all pre-trained models, the rate-distortion loss is used to select the one with the minimum loss. The resized size and filter type can be encoded into the bitstream.

[0142] Now let’s discuss variable rates with resizing. The first step is to encode a given image using all pre-trained models without resizing to obtain the base rate associated with each pre-trained model. The base rate is used to determine whether resizing should be used. For a particular pre-trained model, if the base rate is less than a lower bound, such as R t (1-s), then for this model, the original image x is not downsampled. In addition, the image can be upsampled only when the base rate of all pre-trained models is less than the lower limit. If the base rate of a model is greater than the upper limit, such as R t(1+s), the image is downsampled. In an implementation, there may be a threshold for excluding models whose base rate deviates too much from the target rate. For example, a model that produces more than 50% of the target rate is not used for resizing because this will introduce too much distortion, even if the resulting rate is in the desired range. It should be noted that following the predefined candidate generation rules, there may be multiple resized candidates for a specific pre-trained model. Some examples of implementing candidate generation are provided below.

[0143] In an example, the longer side (the width if width>height, otherwise the height) is reduced by a step size (e.g., 64) number of pixels each time, and the size of the other side is adjusted proportionally. In an example, the shorter side (the width if width<height, otherwise the height) is reduced by a step size (e.g., 64) number of pixels each time, and the size of the other side is adjusted proportionally. In an example, the total number of pixels can be reduced by a number (num) each time. For the above example, there can be multiple candidates in each step. For example, the width or height of the candidate can be rounded to the nearest integer, the width or height of the candidate can be rounded to a predetermined value (e.g., the integer closest to a multiple of 64), for example, the width of the candidate can be rounded to the nearest integer, and the height can be rounded to a predetermined integer value, or both the width and height of the candidate can be rounded to a predetermined integer value.

[0144] The downsampling or upsampling filter can be various filters, such as lanczos, bicubic, bilinear, nearest neighbor, etc. The downsampling or upsampling filter can also be a filter based on a neural network. In addition, the encoder side and the decoder side can be paired with different types of filters. For example, the encoder can use lanczos for downsampling, while the decoder uses bicubic for upsampling. In this implementation example, lanczos is used for the encoder side and the decoder side.

[0145] The selection of candidates is discussed. Given an image x, its target rate T, and a threshold s, the goal is to achieve a rate that deviates within the threshold range, which is subject to the rate-distortion loss. After obtaining the candidates of all pre-trained models. Rate-distortion optimization is used to select the best candidate for this specific target rate. This goal can be expressed as:

[0146]

[0147] Where k is the pre-trained model index and i is the candidate index. In this example, the Bjontegaard delta rate (BDRate) loss is used to measure the loss. The anchor result is used as a reference for calculating the BDRate loss. The anchor result can be obtained in different ways. In one example, the anchor result from another codec (such as H.265) can be used to compress the image and obtain performance at different rate points. In one example, the result can be obtained by selecting the best candidate on a certain evaluation metric under a rate constraint, such as MS-SSIM, Video Multi-Method Assessment Fusion (VMAF), PSNR, etc.

[0148] The full RDO process can be performed by traversing all possible combinations of candidates. However, this introduces a lot of computational complexity. For example, when there are 4 rate points, if there are 10 candidates per rate point, then there will be 10 4 As a practical alternative, it is much simpler to select the best candidate for a rate point. Once the best candidate for the current rate point is found, move to the next rate point and follow a similar process to find the best candidate. Using the same example, in this case, only 40 combinations need to be calculated.

[0149] refer to

[0150] [1] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learning image compression with discrete Gaussian mixture likelihood and attention module,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 7939–7948, 2020.

[0151] [2] B. Bross, J. Chen, S. Liu, and Y.-K. Wang, “Draft of Multifunctional Video (Draft 10)”, JVET-S2001, July 2020.

[0152] [3] R. D. Dony and S. Haykin, “A neural network approach to image compression,” Proceedings of the IEEE, vol. 83, no. 2, pp. 288–303, 1995.

[0153] [4] Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.

[0154] [5] G. Bjontegaard, “Computing the average PSNR difference between RD curves,” VCEG, Technical Report VCEG-M33, 2001.

[0155] [6] C.E. Shannon, “A Mathematical Theory of Communications,” Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948.

[0156] [7] H. Larochelle and I. Murray, “Neural Autoregressive Distribution Estimators,” in Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, 2011, pp. 29–37.

[0157] [8] K. Gregor and Y. LeCun, “Learning representations by maximizing compression,” arXiv preprint arXiv:1108.1169, 2011.

[0158] [9] B. Uria, I. Murray, and H. Larochelle, “RNADE: A real-valued neural autoregressive density estimator,” in NIPS, 2013, pp. 2175–2183.

[0159]

[10] L. Theis and M. Bethge, “Generative image modeling with spatial LSTMs,” in NIPS, 2015, pp. 1927–1935.

[0160]

[11] B. Uria, I. Murray, and H. Larochelle, “Deep and tractable density estimators,” in ICML, 2014, pp. 467–475.

[0161]

[12] A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixelated recurrent neural networks,” in ICML, 2016, pp. 1747–1756.

[0162]

[13] A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu, “Conditional image generation with a PixelCNN decoder,” in NIPS, 2016, pp. 4790–4798.

[0163]

[14] T. Salimans, A. Karpathy, X. Chen, and D.P. Kingma, “PixelCNN++: Improving PixelCNN via discretized logistic mixture likelihood and other modifications,” arXiv preprint arXiv:1701.05517, 2017.

[0164]

[15] X. Chen, N. Mishra, M. Rohaninejad, and P. Abbeel, “PixelSNAIL: An Improved Autoregressive Generative Model,” in ICML, 2018, pp. 863–871.

[0165]

[16] G.E. Hinton and R.R. Salakhutdinov, “Data dimensionality reduction using neural networks”, Science, vol. 313, no. 5786, pp. 504–507, 2006.

[0166]

[17] G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar, “Variable rate image compression using recurrent neural networks,” arXiv preprint arXiv:1511.06085, 2015.

[0167]

[18] G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell, “Full-resolution image compression using recurrent neural networks,” in CVPR, 2017, pp. 5306–5314.

[0168]

[19] N. Johnston, D. Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, S. Jin-Hwang, J. Shor, and G. Toderici, “Improving lossy image compression with priming and spatially adaptive bitrate for recurrent networks,” in CVPR, 2018, pp. 4385–4393.

[0169]

[20] M. Covell, N. Johnston, D. Minnen, S. J. Hwang, J. Shor, S. Singh, D. Vincent, and G. Toderici, “Target-quality image compression using recurrent convolutional neural networks,” arXiv preprint arXiv:1705.06687, 2017.

[0170]

[21] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” in PCS. IEEE, 2016, pp. 1–5.

[0171]

[22] J. Ballé, “Efficient nonlinear transforms for lossy image compression,” in PCS, 2018, pp. 248–252.

[0172]

[23] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” in International Conference on Learning Representations, 2017.

[0173]

[24] J. Ballé, D. Minnen, S. Singh, S. Hwang, and N. Johnston, “Variable image compression with scale hyper-priors,” in International Conference on Learning Representations, 2018.

[0174]

[25] D. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learning image compression,” arXiv.1809.02736.1, 2, 3, 4, 7.

[0175]

[26] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learning image compression using discrete Gaussian mixture likelihood and attention module,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[0176]

[27] Github repository “CompressAI: https: / / github.com / InterDigitalInc / CompressAI”, InterDigitalInc, accessed December 2020.

[0177]

[28] T. Chen, H. Liu, Q. Shen, T. Yue, X. Cao, and Z. Ma, “DeepCoder: Video Compression with Deep Neural Networks,” in VCIP. IEEE, 2017, pp. 1–4.

[0178]

[29] C.-Y. Wu, N. Singhal, and P. Krahenbuhl, “Video compression via image interpolation,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 416–431.

[0179]

[30] Z. Chen, T. He, X. Jin, and F. Wu, “Learning for video compression,” IEEE Transactions on Circuits and Systems for Video Technology, DOI: 10.1109 / TCSVT.2019.2892608, 2019.

[0180]

[31] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “DVC: ​​an end-to-end deep video compression framework,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[0181]

[32] O. Rippel, S. Nair, C. Lew, S. Branson, A. Anderson, and L. Bourdev, “Learned video compression,” in 2019 IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, South Korea, 2019, pp. 3453–3462. doi: 10.1109 / ICCV.2019.00355.

[0182]

[33] G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell, “Full-resolution image compression using recurrent neural networks,” in CVPR, 2017, pp. 5306–5314.

[0183]

[34] A. Habibian, T. Rozendaal, J. Tomczak, and T. Cohen, “Video compression using rate-distortion autoencoders,” in Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2019, pp. 7033–7042.

[0184]

[35] J. Lin, D. Liu, H. Li, and F. Wu, “M-LVC: Multi-frame prediction for learned video compression,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[0185]

[36] E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. J. Hwang, and G. Toderici, “Scale-space flows for end-to-end optimized video compression,” in 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 8500–8509. doi: 10.1109 / CVPR42600.2020.00853.

[0186]

[37] X. Hu, Z. Chen, D. Xu, G. Lu, W. Ouyang, and S. Gu, “Improving deep video compression via resolution-adaptive streaming codec,” in European Conference on Computer Vision (ECCV), 2020.

[0187]

[38] Abdelaziz Djelouah, Joaquim Campos, Simone Schaub-Meyer, and Christopher Schroers. Neural inter-frame compression for video coding. ICCV, pages 6421–6429, October 2019.

[0188]

[39] Toderici, G., O'Malley, S. M., Hwang, S. J., Vincent, D., Minnen, D., Baluja, S., Covell, M., and Sukthankar, R., 2015. Variable rate image compression using recurrent neural networks. arXiv preprint arXiv:1511.06085.

[0189]

[40] Y. Choi, M. El-Khamy, and J. Lee, Variable rate deep image compression using conditional autoencoders, in Proceedings of the IEEE / CVF International Conference on Computer Vision, 3146–3154, 2019.

[0190]

[41] Cui, Z., Wang, J., Bai, B., Guo, T., and Feng, Y., 2020. G-VAE: A framework for continuous variable rate deep image compression. arXiv preprint arXiv:2003.02012.

[0191]

[42] H. Ma, D. Liu, N. Yan, H. Li, and F. Wu, “Versatile image compression with end-to-end optimization using wavelet-like transform,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 3, pp. 1247–1263, March 1, 2022, doi: 10.1109 / TPAMI.2020.3026003.

[0192] Figure 8 4000. The system 4000 may include an input 4002 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10 bit multi-component pixel values), or may be received in a compressed or encoded format. The input 4002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, passive optical networks (PONs), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0193] System 4000 may include a codec component 4004 that can implement various codecs or encoding methods described in this document. Codec component 4004 can reduce the average bit rate of the video from input 4002 to the output of codec component 4004 to generate a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video transcoding technology. The output of codec component 4004 can be stored or sent via the connected communication, as represented by component 4006. The bitstream (or codec) representation of the storage or communication of the video received at input 4002 can be used by component 4008 to generate pixel values ​​or displayable video sent to display interface 4010. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations of the results of the inverse codec will be performed by the decoder.

[0194] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Displayport, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE) interface, etc. The techniques described in this document may be implemented in various electronic devices, such as mobile phones, laptop computers, smart phones, or other devices capable of digital data processing and / or video display.

[0195] Fig. 9 4106 is a block diagram of an example video processing device 4100. Device 4100 may be used to implement one or more of the methods described herein. Device 4100 may be implemented in a smart phone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. Device 4100 may include one or more processors 4102, one or more memories 4104, and video processing circuitry 4106. Processor(s) 4102 may be configured to implement one or more methods described in this document. Memory(s) 4104 may be used to store data and code that is used to implement the methods and techniques described herein. Video processing circuitry 4106 may be used to implement some of the techniques described in this document in hardware circuitry. In some embodiments, video processing circuitry 4106 may be at least partially included in processor 4102, such as a graphics coprocessor.

[0196] Fig.104200 is a flow chart of an example method 4200 for video processing. At step 4202, it is determined to resize an image to create a resized image, and a codec process is applied to the resized image. At step 4204, conversion between visual media data and a bitstream is performed based on the resized image. The conversion may include encoding at an encoder, decoding at a decoder, or a combination thereof.

[0197] It should be noted that method 4200 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory with instructions thereon, such as video encoder 4400, video decoder 4500, and / or encoder 4600. In this case, the instructions executed by the processor cause the processor to perform method 4200. In addition, method 4200 may be performed by a non-transitory computer-readable medium including a computer program product for use by a video codec device. The computer program product includes computer executable instructions stored on a non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs method 4200. In addition, the non-transitory computer-readable recording medium may store a bitstream of a video generated by method 4200 performed by a video processing device. In addition, method 4200 may be performed by an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory with instructions thereon. These instructions, when executed by the processor, cause the processor to perform method 4200.

[0198] Fig.11 43 is a block diagram illustrating an example video codec system 4300 that can utilize the techniques of the present disclosure. The video codec system 4300 may include a source device 4310 and a target device 4320. The source device 4310 generates encoded video data, which may be referred to as a video encoding device. The target device 4320 may decode the encoded video data generated by the source device 4310, which may be referred to as a video decoding device.

[0199] Source device 4310 may include a video source 4312, a video encoder 4314, and an input / output (I / O) interface 4316. Video source 4312 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system that generates video data, or a combination of these sources. Video data may include one or more pictures. Video encoder 4314 encodes video data from video source 4312 to generate a bitstream. The bitstream may include a sequence of bits that form a codec representation of video data. The bitstream may include a codec picture and associated data. The codec picture is a codec representation of a picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax elements. I / O interface 4316 includes a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be directly sent to target device 4320 via network 4330 via I / O interface 4316. The encoded video data may also be stored on storage medium / server 4340 for access by target device 4320.

[0200] Target device 4320 may include an I / O interface 4326, a video decoder 4324, and a display device 4322. I / O interface 4326 may include a receiver and / or a modem. I / O interface 4326 may obtain encoded video data from source device 4310 or storage medium / server 4340. Video decoder 4324 may decode the encoded video data. Display device 4322 may display the decoded video data to a user. Display device 4322 may be integrated with target device 4320, or may be outside target device 4320 configured to be connected to an external display device.

[0201] The video encoder 4314 and the video decoder 4324 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVM) standard, and other current and / or other standards.

[0202] Fig.12 is a block diagram showing an example of a video encoder 4400, which may be Fig.11 4300. Video encoder 4400 may be configured to perform any or all of the techniques of the present disclosure. Video encoder 4400 includes multiple functional components. The techniques described in the present disclosure may be shared between various components of video encoder 4400. In some examples, the processor may be configured to perform any or all of the techniques described in the present disclosure.

[0203] The functional components of the video encoder 4400 may include a segmentation unit 4401, a prediction unit 4402 (which may include a mode selection unit 4403, a motion estimation unit 4404, a motion compensation unit 4405, an intra-frame prediction unit 4406), a residual generation unit 4407, a transform processing unit 4408, a quantization unit 4409, an inverse quantization unit 4410, an inverse transform unit 4411, a reconstruction unit 4412, a buffer 4413 and an entropy coding unit 4414.

[0204] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In one example, the prediction unit 4402 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0205] Furthermore, some components, such as the motion estimation unit 4404 and the motion compensation unit 4405 , may be highly integrated but are represented separately in the example of the video decoder 4400 for purposes of explanation.

[0206] The segmentation unit 4401 may segment the picture into one or more video blocks. The video encoder 4400 and the video decoder 4500 may support various video block sizes.

[0207] The mode selection unit 4403 may select one of the intra or inter coding modes, for example, based on the error result, and provide the resulting intra or inter coding block to the residual generation unit 4407 to generate residual block data and to the reconstruction unit 4412 to reconstruct the coding block for use as a reference picture. In some examples, the mode selection unit 4403 may select a combination of intra and inter prediction (CIIP) mode, where the prediction is based on an inter prediction signal and an intra prediction signal. The mode selection unit 4403 may also select a resolution of motion vectors (e.g., sub-pixel or integer pixel precision) for blocks in the case of inter prediction.

[0208] To perform inter-frame prediction on the current video block, the motion estimation unit 4404 may generate motion information of the current video block by comparing one or more reference frames from the buffer 4413 with the current video block. The motion compensation unit 4405 may determine a predicted video block for the current video block based on the motion information and decoded samples of a picture from the buffer 4413 (rather than a picture associated with the current video block).

[0209] The motion estimation unit 4404 and the motion compensation unit 4405 may perform different operations for the current video block, for example, performing different operations depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0210] In some examples, the motion estimation unit 4404 may perform unidirectional prediction of the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference picture of list 0 or list 1. The motion estimation unit 4404 may then generate a reference index indicating a reference video block in the reference picture of list 0 or list 1 and a motion vector indicating a spatial displacement between the current video block and the reference video block. The motion estimation unit 4404 may output the reference index, the prediction direction indicator, and the motion vector as motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[0211] In other examples, the motion estimation unit 4404 may perform bidirectional prediction of the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference picture of list 0 and may also search for another reference video block of the current video block in the reference picture of list 1. The motion estimation unit 4404 may then generate a reference index indicating a reference video block in the reference picture of list 0 or list 1 and a motion vector indicating a spatial displacement between the reference video block and the current video block. The motion estimation unit 4404 may output the reference index and the motion vector of the current video block as motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0212] In some examples, motion estimation unit 4404 may output a full set of motion information for decoding processing by a decoder. In some examples, motion estimation unit 4404 may not output a full set of motion information for the current video. Instead, motion estimation unit 4404 may reference motion information of another video block to signal motion information of the current video block. For example, motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of an adjacent video block.

[0213] In one example, the motion estimation unit 4404 may indicate in a syntax structure associated with the current video block a value that indicates to the video decoder 4500 that the current video block has the same motion information as another video block.

[0214] In another example, the motion estimation unit 4404 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 4500 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0215] As discussed above, the video encoder 4400 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by the video encoder 4400 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0216] The intra prediction unit 4406 may perform intra prediction on the current video block. When the intra prediction unit 4406 performs intra prediction on the current video block, the intra prediction unit 4406 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0217] The residual generation unit 4407 may generate residual data of the current video block by subtracting the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample components of samples in the current video block.

[0218] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 4407 may not perform a subtraction operation.

[0219] The transform processing unit 4408 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0220] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0221] The inverse quantization unit 4410 and the inverse transform unit 4411 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 4412 may add the reconstructed residual video block to the corresponding samples from one or more predicted video blocks generated by the prediction unit 4402 to generate a reconstructed video block associated with the current block for storage in the buffer 4413.

[0222] After the reconstruction unit 4412 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0223] The entropy coding unit 4414 may receive data from other functional components of the video encoder 4400. When the entropy coding unit 4414 receives data, the entropy coding unit 4414 may perform one or more entropy coding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0224] Fig.13 is a block diagram showing an example of a video decoder 4500, which may be Fig.11 4300. Video decoder 4500 may be configured to perform any or all of the techniques of the present disclosure. In the example shown, video decoder 4500 includes multiple functional components. The techniques described in the present disclosure may be shared between various components of video decoder 4500. In some examples, the processor may be configured to perform any or all of the techniques described in the present disclosure.

[0225] In the example shown, the video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, the video decoder 4500 may perform a decoding process that is generally the inverse of the encoding process described with respect to the video encoder 4400.

[0226] The entropy decoding unit 4501 may retrieve a coded bitstream. The coded bitstream may include entropy coded video data (e.g., coded blocks of video data). The entropy decoding unit 4501 may decode the entropy coded video, and based on the entropy decoded video data, the motion compensation unit 4502 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 4502 may determine such information, for example, by performing AMVP and merge modes.

[0227] The motion compensation unit 4502 may generate a motion compensated block, possibly interpolated based on an interpolation filter. An identifier of the interpolation filter to be used with sub-pixel precision may be included in the syntax element.

[0228] The motion compensation unit 4502 may calculate interpolated values ​​of a sub-integer number of pixels of the reference block using the interpolation filter used by the video encoder 4400 during encoding of the video block. The motion compensation unit 4502 may determine the interpolation filter used by the video encoder 4400 based on received syntax information and use the interpolation filter to generate a prediction block.

[0229] The motion compensation unit 4502 can use some syntax information to determine: the size of the blocks used to encode (multiple) frames and / or (multiple) slices of the coded video sequence, the partitioning information describing how each macroblock of the pictures of the coded video sequence is partitioned, the mode indicating how to encode each partition, one or more reference frames (and reference frame lists) for each inter-frame coding block, and other information for decoding the coded video sequence.

[0230] The intra prediction unit 4503 may form a prediction block from spatially neighboring blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 4504 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 4501. The inverse transform unit 4505 applies an inverse transform.

[0231] The reconstruction unit 4506 may sum the residual block with the corresponding prediction block generated by the motion compensation unit 4502 or the intra prediction unit 4503 to form a decoded block. As desired, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 4507, which provides reference blocks for subsequent motion compensation / intra prediction and also produces a decoded video for presentation on a display device.

[0232] Fig.14 is a schematic diagram of an example encoder 4600. The encoder 4600 is suitable for implementing VVC technology. The encoder 4600 includes three loop filters, namely a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike the DF 4602 that uses a predefined filter, the SAO 4604 and the ALF 4606 use the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, using the auxiliary information of the codec to signal the offset and filter coefficients. The ALF 4606 is located at the last processing stage of each picture and can be regarded as a tool that attempts to capture and repair artifacts produced by previous stages.

[0233] The encoder 4600 also includes an intra prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610 configured to receive an input video. The intra prediction component 4608 is configured to perform intra prediction, while the ME / MC component 4610 is configured to perform inter prediction using a reference picture obtained from a reference picture buffer 4612. The residual block from the inter prediction or intra prediction is fed to a transform (T) component 4614 and a quantization (Q) component 4616 to generate quantized residual transform coefficients, which are fed to an entropy coding component 4618. The entropy coding component 4618 entropy codes and decodes the prediction result and the quantized transform coefficients and sends them to a video decoder (not shown). The quantization component output from the quantization component 4616 can be fed to an inverse quantization (IQ) component 4620, an inverse transform component 4622, and a reconstruction (REC) component 4624. The REC component 4624 is capable of outputting images to the DF 4602 , SAO 4604 , and ALF 4606 for filtering before these pictures are stored in the reference picture buffer 4612 .

[0234] Fig.15 4700 is a flow chart of an example method 4700 for video processing. At step 4702, a determination is made to resize an image having a first size to create a resized image having a second size. At step 4704, a conversion between visual media data and a bitstream is performed based on the resized image. The conversion includes applying a neural network-based codec model to the resized image to implement variable rate neural network-based compression. The conversion may include encoding at an encoder, decoding at a decoder, or a combination thereof.

[0235] It should be noted that method 4200 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory with instructions thereon, such as video encoder 4400, video decoder 4500, and / or encoder 4600. In this case, the instructions executed by the processor cause the processor to perform method 4200. In addition, method 4200 may be performed by a non-transitory computer-readable medium including a computer program product for use by a video codec device. The computer program product includes computer executable instructions stored on a non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs method 4200. In addition, the non-transitory computer-readable recording medium may store a bitstream of a video generated by method 4200 performed by a video processing device. In addition, method 4200 may be performed by an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory with instructions thereon. These instructions, when executed by the processor, cause the processor to perform method 4200.

[0236] A list of some example preferred solutions is provided below.

[0237] The following solutions show examples of the techniques discussed in this article.

[0238] 1. A method for processing video data (e.g., Fig.10 The method 4200 depicted in the claims includes: determining (4202) how to size an image to create a resized image, and applying a codec process to the resized image; and performing (4204) conversion between visual media data and a bitstream based on the resized image.

[0239] 2. The method of solution 1, wherein resizing comprises upsampling the image.

[0240] 3. A method according to any one of solutions 1-2, wherein the encoding and decoding process is performed by a neural network model, and wherein the resizing includes upsampling the image when the basic rate of the neural network model is less than a lower limit.

[0241] 4. A method according to any of solutions 1-3, wherein resizing includes downsampling the image.

[0242] 5. A method according to any one of solutions 1-4, wherein the encoding and decoding process is performed by a neural network model, and wherein the resizing includes downsampling the image when the basic rate of the neural network model is greater than a lower limit.

[0243] 6. A method according to any of solutions 1-5, wherein size information describing the resized image is included in the bitstream.

[0244] 7. The method according to any one of solutions 1-6, wherein the size information includes width, height, a ratio of height to width, the number of samples, the number of sample groups of predefined sizes, a size index, or a combination thereof.

[0245] 8. A method according to any one of solutions 1-7, wherein before resizing, sample padding or sample cropping is applied to the image.

[0246] 9. A method according to any one of solutions 1-8, wherein after resizing, sample padding or sample cropping is applied to the image.

[0247] 10. A method according to any of solutions 1-9, wherein the resizing is performed based on the color format or color components of the image.

[0248] 11. The method according to any of solutions 1-10, wherein the resizing is performed by a filter.

[0249] 12. A method according to any of solutions 1-11, wherein the filter used for resizing is indicated in the bitstream.

[0250] 13. A method according to any one of solutions 1-12, wherein the filter is a neural network filter, a lanczos filter, a bilinear filter, a bicubic filter or a combination thereof.

[0251] 14. The method of any of solutions 1-13, wherein the resizing is selected from a set of resizing candidates, and wherein the resizing candidate indicating the selected resizing is signaled in the bitstream.

[0252] 15. A method according to any one of solutions 1-14, wherein the coding process is applied by a coding model selected from a set of coding model candidates, and wherein the coding model candidate indicating the coding model is signaled in the bitstream.

[0253] 16. A method according to any one of solutions 1-15, wherein the set of codec model candidates includes codec models pre-trained with different Lagrangian multipliers.

[0254] 17. A method according to any of solutions 1-16, wherein the image is resized to a set of resizing candidates, each of which meets the target bit rate.

[0255] 18. A method according to any one of solutions 1-17, wherein the set of codec model candidates is selected to meet a target rate.

[0256] 19. A method according to any of solutions 1-18, wherein converting includes encoding the visual media data into a bitstream.

[0257] 20. A method according to any of solutions 1-19, wherein converting includes decoding the visual media data from a bitstream.

[0258] 21. An apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of solutions 1-20.

[0259] 22. A non-transitory computer-readable recording medium storing a bitstream of a video, wherein the bitstream is generated by a method performed by a video processing device, the method comprising: determining to resize an image to create a resized image, and applying codec processing to the resized image; and generating a bitstream based on the determination.

[0260] 23. A method for storing a bitstream of a video, comprising: determining to resize an image to create a resized image and applying a coding process to the resized image; generating a bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0261] The attached solution shows examples of the techniques discussed in this article.

[0262] 1. A method for processing video data, comprising: determining to resize an image having a first size to create a resized image having a second size; and performing conversion between visual media data and a bitstream based on the resized image, wherein the conversion includes applying a neural network-based codec model to the resized image to implement variable rate neural network-based compression.

[0263] 2. The method according to solution 1, wherein the resizing comprises upsampling the image or downsampling the image.

[0264] 3. A method according to any of solutions 1-2, wherein size information describing a first size of the image and a second size of the resized image is included in the bitstream.

[0265] 4. A method according to any one of solutions 1-3, wherein the size information includes width and height, and wherein the size information is indicated in samples of the component or in the number of samples (N) of the component.

[0266] 5. A method according to any one of solutions 1-4, wherein the first size and the second size are signaled independently, predictively, as a ratio between the first size and the second size, or a combination thereof.

[0267] 6. A method according to any of solutions 1-5, wherein the resized width of the resized image is proportional to the initial width of the image, and wherein the resized height of the resized image is proportional to the initial height of the image.

[0268] 7. A method according to any one of solutions 1-6, wherein the ratio of the resized width to the initial width is equal to the ratio of the resized height to the initial height.

[0269] 8. A method according to any one of solutions 1-7, wherein before resizing, sample padding or sample cropping is applied to the image.

[0270] 9. A method according to any one of solutions 1-8, wherein after resizing, sample padding or sample cropping is applied to the image.

[0271] 10. A method according to any of solutions 1-9, wherein a message is signaled in the bitstream to indicate whether the image should be resized.

[0272] 11. A method according to any one of solutions 1-10, wherein size information related to the first size, size information related to the second size, or a size ratio between the first size and the second size is signaled only when the image should be resized.

[0273] 12. A method according to any one of solutions 1-11, wherein the resizing includes: upsampling the image when the basic rate of the neural network model is less than a lower limit, or downsampling the image when the basic rate of the neural network model is greater than the lower limit.

[0274] 13. A method according to any of solutions 1-12, wherein the resizing is performed by a lanczos filter, a bilinear filter, a bicubic filter, a nearest neighbor filter or a combination thereof.

[0275] 14. A method according to any of solutions 1-13, wherein the same type of filter is used for resizing when encoding and decoding.

[0276] 15. A method according to any of solutions 1-14, wherein the size information is signaled using at least one index, and wherein the index indicates a predefined size, width, height or ratio.

[0277] 16. A method according to any one of solutions 1-15, wherein the use of resizing depends on the color component or color format, or wherein different color components are resized by different resizing processes, or wherein the luminance component is encoded and decoded by resizing, but the chrominance components are not encoded and decoded by resizing, or wherein different size parameters are used to adjust the sizes of different components, or wherein a first size parameter is used to adjust the size of the luminance component and a second parameter is used to adjust the size of the chrominance component, or wherein the image is a reconstructed output image or input image, or wherein the size parameter includes a width value, a height value, or a combination thereof.

[0278] 17. A method according to any one of solutions 1-16, wherein resizing is performed using a filter, wherein the filter is a separable filter including a horizontal filter and a vertical filter, or wherein the type of filter used for resizing is indicated in the bitstream, or wherein the length of the filter is indicated in the bitstream, the length of the filter is determined by the number of taps of the filter, or is defined as the length of the filter in the horizontal or vertical direction, or wherein at least two different types of filters can be indicated in the bitstream, wherein a first filter is applied to a first component of the reconstructed output image and a second filter is applied to a second component of the reconstructed output image, or wherein the type of filter is used to determine the length of the filter or the coefficients of the filter, or wherein an index is indicated in the bitstream to determine which filter of a set of filters is applied for resizing, or wherein an indication for specifying which filter of a set of filters is applied for resizing is included in the bitstream, wherein a first value of the indication indicates that a 3-tap filter is applied for resizing and a second value of the indication indicates that an 8-tap filter is applied for resizing, or wherein the indication specifies weight values ​​for filtering, or wherein the filter is implemented as a convolutional layer or a deconvolutional layer of a neural network.

[0279] 18. A method according to any of solutions 1-17, wherein a set of candidate models for encoding and decoding image signaling depends on whether the use of image resizing occurs before or after the encoding and decoding process.

[0280] 19. A method according to any one of solutions 1-18, wherein one or more neural network models are determined based on whether the use of image resizing occurs before or after the encoding and decoding process, or wherein at least one of the multiple models is selected and signaled in the bitstream, or wherein at least one of the multiple models is derived, or wherein the multiple models correspond to multiple bit rates.

[0281] 20. A method according to any one of solutions 1-19, wherein one or more models are pre-trained with different Lagrangian multipliers, or wherein the image is resized to multiple sizes to obtain multiple candidates for a given target rate, wherein one or more models are trained according to the target bit rate, or wherein the image is downsampled or upsampled, or wherein the resized width and the resized height are proportional to the original width and the original height, or wherein the resized width and the resized height are not proportional to the original width and the original height, or wherein changing the number of pixels of the width or height is used to obtain the resized candidates, or wherein changing the total number of pixels is used to obtain the resized candidates, or wherein changing the ratio of the resized size to the original image size is used to obtain the resized candidates, wherein the ratio is the width, height or the total number of pixels, or

[0282] Where images are required to comprise multiples of 32 or 64 pixels along width and height to remove boundary artifacts, or where different filters are used for upsampling and downsampling, or where neural network based filters are used, or where deep filters for image super-resolution are used, or where different types of filters are used for encoding and decoding, or where lanczos filters are used for encoding and bicubic filters are used for decoding.

[0283] 21. A method according to any one of solutions 1-20, wherein the resized width and the resized height are included in the bitstream, or wherein an index of the resized sizes corresponding to a predefined list of resized sizes is included in the bitstream, or wherein at least one filter index corresponding to a list of predefined filter pairs is included in the bitstream.

[0284] 22. A method according to any one of solutions 1-21, wherein a rate-distortion optimization (RDO) process is used to select a candidate with the best rate-distortion (RD) performance from the candidates, or wherein an anchor result is used as a reference for calculating the RD loss in the RDO process, wherein the result of the codec is used as the anchor result, or wherein the result of a model using a fixed quantization parameter (QP) without adjusting the size is used as the anchor result, or wherein the anchor result is obtained by selecting a candidate with the best specific metric for a certain rate point, or wherein the RD loss in the RDO process includes one or more metrics, or wherein the RD loss in the RDO process is calculated at one or more rate points, or wherein the RD loss is calculated using three fixed rates and then all candidates are analyzed using a set of fourth rate points to select the best fourth rate point, or wherein 4 rate points are used, 10 candidates are analyzed in each rate point, and 104 combinations are analyzed to obtain a combination with the minimum RD loss.

[0285] 23. A method according to any one of solutions 1-22, wherein the message is signaled as a flag, a truncated unary code, a truncated binary code, an exponential Golomb code, a fixed length code, or wherein the message is encoded or decoded using at least one arithmetic codec context or a bypass codec.

[0286] 24. An apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of solutions 1-23.

[0287] 25. A non-transitory computer-readable recording medium storing a bitstream of a video, wherein the bitstream is generated by a method performed by a video processing device, the method comprising: determining to resize an image having a first size to create a resized image having a second size; and generating the bitstream based on the determination, wherein the generating comprises applying a neural network-based codec model to the resized image to achieve variable rate neural network-based compression.

[0288] 26. A method for storing a bitstream of a video, comprising: determining to resize an image having a first size to create a resized image having a second size; generating a bitstream based on the determination, wherein the generating comprises applying a neural network-based codec model to the resized image to achieve variable rate neural network-based compression; and storing the bitstream in a non-transitory computer-readable recording medium.

[0289] In the solution described herein, an encoder can comply with the format rules by generating a codec representation according to the format rules. In the solution described herein, a decoder can use the format rules to parse syntax elements in the codec representation, knowing the presence and absence of syntax elements according to the format rules to produce decoded video.

[0290] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, during the conversion from a pixel representation of a video to a corresponding bitstream representation, a video compression algorithm may be applied, and vice versa. As defined by the syntax, the bitstream representation of the current video block may correspond to bits that are co-located or scattered at different positions within the bitstream, for example. For example, a macroblock may be encoded based on the error residual values ​​of the transform and the codec and also using bits in the header and other fields in the bitstream. In addition, during the conversion, the decoder may parse the bitstream based on this determination, knowing that some fields may or may not be present, as described in the above solution. Similarly, the encoder may determine whether to include or not include certain syntax fields, and generate the codec representation accordingly by including or excluding the syntax fields from the codec representation.

[0291] The disclosure and other schemes, examples, embodiments, modules and functional operations described in this document may be implemented in digital electronic circuits or in computer software, firmware or hardware, containing the structures disclosed in this document and the equivalents of their structures, or a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products encoded on a computer-readable medium, i.e., one or more computer program instruction modules, for execution by a data processing device or for controlling the operation of a data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composite that affects a machine-readable propagable signal, or a combination of one or more thereof. The term "data processing device" encompasses all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0292] A computer program (also referred to as a program, software, software application, script or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including stand-alone programs or modules, components, subroutines or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that stores other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines or code portions). A computer program can be deployed to execute on one computer or on multiple computers, which are located at one site or distributed across multiple sites and interconnected by a communication network.

[0293] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuits (e.g., field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs)), and the apparatus may be implemented as special purpose logic circuits (e.g., FPGAs or ASICs).

[0294] Processors suitable for the execution of computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (such as magnetic, magneto-optical disks or optical disks) for storing data, or operably coupled to receive data from a mass storage device (such as a magnetic, magneto-optical disk or optical disk) or transfer data to a mass storage device (such as a magnetic, magneto-optical disk or optical disk), or both. However, a computer does not have to have such devices. Computer-readable media suitable for storing computer program instructions and data contain all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices); disks (such as internal hard disks or removable disks); magneto-optical disks; and compact disk read-only memory (CD ROM) and digital versatile disk read-only memory (DVD-ROM) disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0295] Although this patent document contains many details, these details should not be interpreted as limitations on any subject matter or the scope of protection that can be claimed, but as descriptions of features of specific embodiments specified in specific technologies. In this patent document, certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments, respectively, or in various suitable sub-combinations. In addition, although features may be described as working in certain combinations and even initially required in the same manner as described above, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0296] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in the sequential order shown, or that all illustrated operations be performed, in order to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0297] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent document.

[0298] A first component is directly coupled to a second component when there are no intermediate components other than a line, trace, or another medium between the first component and the second component. A first component is indirectly coupled to a second component when there are intermediate components other than a line, trace, or another medium between the first component and the second component. The term "coupled" and its variations include direct coupling and indirect coupling. Unless otherwise specified, the use of the term "about" is meant to include a range of 10% of the subsequent value.

[0299] Although several embodiments are provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are considered to be illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0300] In addition, without departing from the scope of the present disclosure, the discrete or separate techniques, systems, subsystems, and methods described and illustrated in the various embodiments may be combined or integrated with other systems, modules, techniques, or methods. Other items shown or discussed as coupled may be directly connected, or may be indirectly coupled or communicated electrically, mechanically, or otherwise through some interface, device, or intermediate component. Other examples of changes, substitutions, and alterations may be determined by those skilled in the art, and these changes, substitutions, and alterations may be made without departing from the spirit and scope disclosed herein.

Claims

1. A method for processing video data, comprising: determining to resize an image having a first size to create a resized image having a second size; as well as performing conversion between visual media data and a bitstream based on the resized image, The conversion includes applying a neural network-based codec model to the resized image to achieve variable rate neural network-based compression.

2. The method according to claim 1, wherein: The resizing includes upsampling the image or downsampling the image.

3. The method according to any one of claims 1 to 2, wherein: Size information describing the first size of the image and the second size of the resized image is included in the bitstream.

4. The method according to any one of claims 1 to 3, wherein: The size information includes width and height, and wherein the size information is indicated in samples of a component or in the number of samples (N) of the component.

5. The method according to any one of claims 1-4, wherein the first size and the second size are signaled independently, predictively, as a ratio between the first size and the second size, or a combination thereof.

6. The method according to any one of claims 1 to 5, wherein: The resized width of the resized image is proportional to an initial width of the image, and wherein the resized height of the resized image is proportional to an initial height of the image.

7. The method according to any one of claims 1 to 6, wherein: The ratio of the resized width to the original width is equal to the ratio of the resized height to the original height.

8. The method according to any one of claims 1 to 7, wherein: Prior to the resizing, sample padding or sample cropping is applied to the image.

9. The method according to any one of claims 1 to 8, wherein: After the resizing, sample padding or sample cropping is applied to the image.

10. The method according to any one of claims 1 to 9, wherein: A message is signaled in the bitstream to indicate whether the picture should be resized.

11. The method according to any one of claims 1 to 10, wherein: Size information related to the first size, size information related to the second size, or a size ratio between the first size and the second size is signaled only when the image should be resized.

12. The method according to any one of claims 1 to 11, wherein the resizing comprises: The image is upsampled when the basic rate of the neural network model is less than a lower limit, or the image is downsampled when the basic rate of the neural network model is greater than the lower limit.

13. The method according to any one of claims 1 to 12, wherein: The resizing is performed by a lanczos filter, a bilinear filter, a bicubic filter, a nearest neighbor filter, or a combination thereof.

14. The method according to any one of claims 1 to 13, wherein: The resizing uses the same type of filters when encoding and decoding.

15. The method according to any one of claims 1 to 14, wherein: The size information is signaled with at least one index, and wherein the index indicates a predefined size, width, height or ratio.

16. The method according to any one of claims 1 to 15, wherein the use of the resizing depends on the color component or the color format, or wherein, The sizes of different color components are adjusted by different resizing processes, or wherein the luma component is encoded and decoded by the resizing, but the chroma components are not encoded and decoded by the resizing, or wherein different size parameters are used to adjust the sizes of different components, or wherein the luma component is adjusted by a first size parameter and the chroma components are adjusted by a second parameter, or wherein the image is a reconstructed output image or an input image, or wherein the size parameter includes a width value, a height value, or a combination thereof.

17. The method according to any one of claims 1 to 16, wherein: The resizing is performed using a filter, wherein the filter is a separable filter including a horizontal filter and a vertical filter, or wherein the type of the filter used for the resizing is indicated in the bitstream, or wherein the length of the filter is indicated in the bitstream, the length of the filter being determined by the number of taps of the filter or being defined as the length of the filter in the horizontal or vertical direction, or wherein at least two different types of filters may be indicated in the bitstream, wherein a first filter is applied to a first component of a reconstructed output image and a second filter is applied to a second component of the reconstructed output image, or wherein the type of the filter is used to determine the length of the filter or the coefficients of the filter, or wherein an index is indicated in the bitstream to determine which filter of a set of filters is applied for the resizing, or wherein an indication for specifying which filter of a set of filters is applied for the resizing is included in the bitstream, a first value of the indication indicating that a 3-tap filter is applied for the resizing and a second value of the indication indicating that an 8-tap filter is applied for the resizing, or wherein the indication specifies weight values ​​for filtering, or wherein the filter is implemented as a convolutional layer or a deconvolutional layer of a neural network.

18. The method according to any one of claims 1 to 17, wherein: The set of candidate models for encoding and decoding image signaling depends on whether the use of said resizing of said image occurs before or after the encoding and decoding process.

19. The method according to any one of claims 1 to 18, wherein: One or more neural network models are determined based on whether the use of the resizing of the image occurs before or after the encoding and decoding process, or wherein at least one model of a plurality of models is selected and signaled in the bitstream, or wherein at least one model of a plurality of models is derived, or wherein a plurality of models corresponds to a plurality of bit rates.

20. The method according to any one of claims 1 to 19, wherein: One or more models are pre-trained with different Lagrangian multipliers, or wherein the image is resized to multiple sizes to obtain multiple candidates for a given target rate, wherein one or more models are trained based on a target bit rate, or wherein the image is downsampled or upsampled, or wherein the resized width and resized height are proportional to the original width and original height, or wherein the resized width and resized height are not proportional to the original width and original height, or wherein the number of pixels of the width or height is changed to obtain the resized candidates, or wherein the total number of pixels is changed to obtain the resized candidates, or wherein the ratio of the resized size to the original image size is changed to obtain the resized candidates, wherein the ratio is the width, height or total number of pixels, or wherein the image is required to include a multiple of 32 or 64 pixels along the width and height to eliminate boundary artifacts, or wherein different filters are used for upsampling and downsampling, or wherein neural network based filters are used, or wherein deep filters for image super-resolution are used, or wherein different types of filters are used for encoding and decoding, or wherein a lanczos filter is used for encoding and a bicubic filter is used for decoding.

21. The method according to any one of claims 1 to 20, wherein: The resized width and the resized height are included in the bitstream, or wherein, Indices of resized sizes corresponding to a predefined list of resized sizes are included in the bitstream, or wherein at least one filter index corresponding to a predefined list of filter pairs is included in the bitstream.

22. The method according to any one of claims 1 to 21, wherein: A rate-distortion optimized (RDO) process is used to select a candidate with the best rate-distortion (RD) performance from the candidates, or wherein an anchor result is used as a reference for calculating the RD loss in the RDO process, wherein a result of a codec is used as the anchor result, or wherein a result of a model using a fixed quantization parameter (QP) without resizing is used as the anchor result, or wherein the anchor result is obtained by selecting a candidate with the best specific metric for a certain rate point, or wherein the RD loss in the RDO process includes one or more metrics, or wherein the RD loss in the RDO process is calculated at one or more rate points, or wherein the RD loss is calculated using three fixed rates and then all candidates are analyzed using a set of fourth rate points to select the best fourth rate point, or wherein 4 rate points are used, 10 candidates are analyzed in each rate point, and 10 are analyzed. 4 Combine to obtain the combination with the minimum RD loss.

23. The method of any one of claims 1-22, wherein the message is signaled as a flag, a truncated unary code, a truncated binary code, an exponential Golomb code, a fixed length code, or wherein, The message is encoded or decoded using at least one arithmetic codec context or a bypass codec.

24. An apparatus for processing video data, comprising: processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-23.

25. A non-transitory computer-readable recording medium storing a bit stream of a video, wherein: The bitstream is generated by a method performed by a video processing device, the method comprising: determining to resize an image having a first size to create a resized image having a second size; and generating a bitstream based on the determination, Wherein, the generating includes applying a neural network-based codec model to the resized image to achieve variable rate neural network-based compression.

26. A method for storing a bitstream of a video, comprising: determining to resize an image having a first size to create a resized image having a second size; generating a bitstream based on the determining, wherein the generating comprises applying a neural network-based codec model to the resized image to implement variable rate neural network-based compression; and The bit stream is stored in a non-transitory computer-readable recording medium.

Images