Method and device for visual data processing and medium

Abstract

Embodiments of the present disclosure provide a solution for visual data processing. In one embodiment, a method for visual data processing includes applying a neural network (NN)-based model including a first module for predictive fusion and a second module for a variance decoder utilizing hyper-priori information to video data for a conversion between visual data and a bitstream of visual data, the neural network (NN)-based model including a first module for predictive fusion and a second module for a variance decoder utilizing hyper-priori information, wherein at least one of a first module for predictive fusion and / or a second module for a variance decoder utilizing hyper-priori information is satisfied: the number of channels in a convolutional layer is less than or equal to a first threshold number, the total number of convolutional layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size; and performing the conversion based on a first module for predictive fusion and a second module for a variance decoder utilizing the hyper-priori information.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure generally relate to visual data processing techniques, and more particularly, to a neural network based image and video compression method with simplified variance decoder exploiting hyper-prior and prediction fusion network. BACKGROUND

[0002] In the past decade, deep learning has made rapid progress in various fields, especially in computer vision and image processing. Neural networks were originally invented with interdisciplinary research in neuroscience and mathematics. It shows strong ability in the context of nonlinear transformation and classification. In the past five years, neural network based image / video compression technology has made significant progress. It is reported that the latest neural network based image compression algorithm achieves rate-distortion (R-D) performance comparable to versatile video coding (VVC). With the continuous improvement of neural image compression performance, neural network based video compression has become an actively developing research field. However, the coding quality and coding efficiency of neural network based image / video coding are usually expected to be further improved. SUMMARY

[0003] Embodiments of the present disclosure provide a solution for visual data processing.

[0004] In a first aspect, a method for visual data processing is proposed. The method comprises: for conversion between visual data and a bitstream of the visual data, applying a neural network (NN) based model comprising a first module for prediction fusion and a second module for variance decoder exploiting hyper-prior to the visual data, wherein at least one of the following is satisfied for the first module for prediction fusion and / or the second module for variance decoder exploiting hyper-prior: a number of channels in a convolution layer is less than or equal to a first threshold number, a total number of convolution layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size; and performing the conversion based on the first module for prediction fusion and the second module for variance decoder exploiting hyper-prior. The method according to the first aspect of the present disclosure, compared with the traditional solution, the first module for prediction fusion can have one or more of the following: fewer convolution layers, fewer channels in each layer or smaller kernel size, and / or the second module for variance decoder exploiting hyper-prior can have one or more of the following: fewer convolution layers, fewer channels in each layer or smaller kernel size. In this way, the proposed method can advantageously simplify the module for prediction fusion and the module for variance decoder exploiting hyper-prior, thereby reducing the time consumed at this stage. As a result, the coding efficiency can be improved.

[0005] In a second aspect, an apparatus for visual data processing is presented. The apparatus includes a processor and a non-transitory memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0006] In a third aspect, a non-transitory computer-readable storage medium is presented. The non-transitory computer-readable storage medium stores instructions that cause a processor to perform the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect, another non-transitory computer-readable recording medium is presented. The non-transitory computer-readable recording medium stores a bitstream of visual data generated by a method performed by an apparatus for visual data processing. The method includes: applying a neural network (NN) based model including a first module for prediction fusion and a second module for a variance decoder with hyper-prior information to video data, wherein at least one of the following is satisfied for the first module for prediction fusion and / or the second module for the variance decoder with hyper-prior information: a number of channels in a convolution layer is less than or equal to a first threshold number, a total number of convolution layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size; and generating the bitstream based on the first module for prediction fusion and the second module for the variance decoder with hyper-prior information.

[0008] In a fifth aspect, a method for storing a bitstream of visual data is presented. The method includes: applying a neural network (NN) based model including a first module for prediction fusion and a second module for a variance decoder with hyper-prior information to video data, wherein at least one of the following is satisfied for the first module for prediction fusion and / or the second module for the variance decoder with hyper-prior information: a number of channels in a convolution layer is less than or equal to a first threshold number, a total number of convolution layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size; generating the bitstream based on the first module for prediction fusion and the second module for the variance decoder with hyper-prior information; and storing the bitstream in a non-transitory computer-readable recording medium.

[0009] This Summary is intended to introduce some of the concepts discussed below in the of the Invention section. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other objects, features and advantages of the example embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to the like elements throughout. In the example embodiments of the present disclosure, like reference numerals refer to like elements throughout.

[0011] Figure 1 A block diagram of an example visual data codec system is shown, in accordance with some embodiments of the present disclosure;

[0012] Figure 2 A typical transform coding scheme is shown;

[0013] Figure 3 Images from the Kodak dataset and different representations of the images are shown;

[0014] Figure 4 A network architecture of an autoencoder implementing a hyperprior model is shown;

[0015] Figure 5 A block diagram of a combined model is shown;

[0016] Figure 6 An encoding process of a combined model is shown;

[0017] Figure 7 A decoding process of a combined model is shown;

[0018] Figure 8 An example convolutional layer is shown;

[0019] Figure 9 An example convolutional layer with 2 groups is shown;

[0020] Figure 10 A decoder architecture with decoupled processing is shown;

[0021] Figure 11 An example prediction fusion network for luminance is shown;

[0022] Figure 12 An example variance decoder for luminance that utilizes hyperprior information is shown;

[0023] Figure 13 An example simplified prediction fusion network (5 layers, 32x) is shown;

[0024] Figure 14 An example simplified prediction fusion network (5 layers, 16x) is shown;

[0025] Figure 15 An example simplified prediction fusion network (4 layers, 32x) is shown;

[0026] Figure 16 An example simplified prediction fusion network (4 layers, 16x) is shown;

[0027] Figure 17 An example simplified prediction fusion network (6 layers, 32x) is shown;

[0028] Figure 18 A simplified prediction fusion network (6 layers, 16x) is shown as an example;

[0029] Figure 19 A simplified prediction fusion network (5 layers, 16x) is shown as an example;

[0030] Figure 20 A simplified variance decoder with hyper-prior information (4x3) is shown as an example;

[0031] Figure 21 A simplified variance decoder with hyper-prior information (5 layers, 4x3) is shown as an example;

[0032] Figure 22 A simplified variance decoder with hyper-prior information (5 layers, 4x4, upsample + conv + upsample) is shown as an example;

[0033] Figure 23 A simplified variance decoder with hyper-prior information (3x3 with Pixel Shuffle) is shown as an example;

[0034] Figure 24 A simplified variance decoder with hyper-prior information (4x3 + 3x4) is shown as an example;

[0035] Figure 25 A simplified variance decoder with hyper-prior information (4x3, 5 layers, C→3 / 2C→2C→3 / 2C→C) is shown as an example;

[0036] Figure 26 A simplified variance decoder with hyper-prior information (4x3, 5 layers, C→C→C→C→C) is shown as an example;

[0037] Figure 27 A simplified variance decoder with hyper-prior information (4x3, 4 layers, C→3 / 2C→3 / 2C→C) is shown as an example;

[0038] Figure 28 A simplified variance decoder with hyper-prior information (4x3, 4 layers, 3 / 2C→2C→3 / 2C→C) is shown as an example;

[0039] Figure 29 A simplified variance decoder with hyper-prior information (4x3, 4 layers, C→C→C→C) is shown as an example;

[0040] Figure 30A simplified variance decoder with hyper-prior information (4 layers, 2x2 pixel shuffle) is shown;

[0041] Figure 31 A simplified variance decoder with hyper-prior information (4 layers, 2x2 pixel shuffle) is shown;

[0042] Figure 32 A simplified variance decoder with hyper-prior information (4 layers, 2x2 pixel shuffle) is shown;

[0043] Figure 33 A simplified variance decoder with hyper-prior information (4 layers, 2x2 pixel shuffle) is shown;

[0044] Figure 34 A simplified variance decoder with hyper-prior information (4 layers, 2x2 pixel shuffle) is shown;

[0045] Figure 35 A simplified variance decoder with hyper-prior information (4 layers, 2x2 pixel shuffle) is shown;

[0046] Figure 36A A simplified prediction fusion network is shown, and Figure 36B A simplified variance decoder with hyper-prior information (3 layers) is shown;

[0047] Figure 37 A flowchart of a method for visual data processing according to an embodiment of the present disclosure is shown; and

[0048] Figure 38 A block diagram of a computing device in which various embodiments of the present disclosure can be implemented is shown.

[0049] In all of the drawings, like or similar reference numerals are generally employed to refer to like or similar elements throughout the drawings. DETAILED DESCRIPTION

[0050] The principles of the present disclosure will now be described with reference to some embodiments. It should be understood that the description of these embodiments is merely made for the purpose of illustrating and helping the skilled person to understand and implement the present disclosure, and does not imply any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various ways in addition to the ways described below.

[0051] In the following description and claims, unless otherwise defined, all scientific and technical terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0052] References in the present disclosure to “one embodiment,” “an embodiment,” “example embodiments,” etc., indicate that the embodiment described can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an example embodiment, it is submitted that it is within the knowledge of those skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0053] It should be understood that although the terms “first” and “second” etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the scope of the example embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed terms.

[0054] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes” and / or “including,” when used herein, specify the presence of stated features, elements and / or components, but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0055] Example Environment

[0056] Figure 1 is a block diagram illustrating an example visual data coding system 100 that can utilize the techniques of this disclosure. As shown, the visual data coding system 100 can include a source device 110 and a destination device 120. The source device 110 can also be referred to as a visual data encoding device, and the destination device 120 can also be referred to as a visual data decoding device. In operation, the source device 110 can be configured to generate encoded visual data, and the destination device 120 can be configured to decode the encoded visual data generated by the source device 110. The source device 110 can include a visual data source 112, a visual data encoder 114, and an input / output (I / O) interface 116.

[0057] The visual data source 112 can include a source such as a visual data capture device. Examples of visual data capture devices include, but are not limited to, an interface to receive visual data from a visual data provider, a computer graphics system to generate visual data, and / or combinations thereof.

[0058] The visual data can include one or more pictures or one or more images of a video. The visual data encoder 114 encodes the visual data from the visual data source 112 to generate a bitstream. The bitstream can include a sequence of bits that forms an encoded representation of the visual data. The bitstream can include encoded pictures and associated visual data. An encoded picture is an encoded representation of a picture. The associated visual data can include sequence parameter sets, picture parameter sets, and other syntax structures. The I / O interface 116 can include a modulator / demodulator and / or a transmitter. The encoded visual data can be transmitted directly to the destination device 120 via the I / O interface 116 over the network 130A. The encoded visual data can also be stored on a storage medium / server 130B for access by the destination device 120.

[0059] The destination device 120 can include an I / O interface 126, a visual data decoder 124, and a display device 122. The I / O interface 126 can include a receiver and / or a modem. The I / O interface 126 can retrieve the encoded visual data from the source device 110 or the storage medium / server 130B. The visual data decoder 124 can decode the encoded visual data. The display device 122 can display the decoded visual data to a user. The display device 122 can be integrated with the destination device 120 or can be external to the destination device 120 configured to interface with an external display device.

[0060] The visual data encoder 114 and the visual data decoder 124 can operate according to a visual data coding standard, such as a video coding standard or a still picture coding standard, and other existing and / or further standards.

[0061] Some example embodiments of the disclosure will be described in detail below. It should be understood that the use of section headings in this document is for ease of understanding only and is not to be construed as limiting the embodiments disclosed in that section to only the section. Also, while some embodiments are described with reference to multi-function video coding or other specific visual data codec, the disclosed techniques are applicable to other coding techniques as well. Further, while some embodiments describe encoding steps in detail, it is to be understood that corresponding decoding steps to undo the encoding would be implemented by a decoder. Also, the term "visual data processing" includes visual data encoding or compression, visual data decoding or decompression, and visual data transcoding, in which visual data is represented in one compression format to another compression format or at a different compression bit rate. 1. Preliminary Discussion

[0062] The present disclosure relates to a neural network based image and video compression method including an auto-regressive subnetwork and an entropy coding engine, where the entropy coding is performed independently of the auto-regressive subnetwork, i.e. a decoupled architecture. In the present disclosure, the decoupled architecture is simplified to reduce the decoding time complexity. Specifically, the variance decoder and the prediction fusion network that utilize hyper-prior information are simplified. 2. Further discussion

[0063] Deep learning is developing in various fields, such as computer vision and image processing. Inspired by the successful application of deep learning techniques in the field of computer vision, neural image / video compression techniques are being researched to be applied to image / video compression techniques. Neural networks are designed based on interdisciplinary research in neuroscience and mathematics. Neural networks show strong capabilities in the context of non-linear transformations and classification. An example neural network based image compression algorithm achieves comparable R-D performance to Versatile Video Coding (VVC), a video coding standard developed by the Joint Video Experts Team (JVET) with experts from the Motion Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG) using. Neural network based video compression is an actively developing research field, leading to continuous improvement in the performance of neural image compression. However, due to the inherent difficulty of the problems solved by neural networks, neural network based video coding is still a largely unexplored discipline. 2.1 Image / video compression

[0064] Image / video compression generally refers to a computational technique that compresses video images into binary codes for convenient storage and transmission. Binary codes can or can not support lossless reconstruction of the original image / video. Coding without data loss is called lossless compression, and coding that allows targeted data loss is called lossy compression. Most coding systems use lossy compression because in most cases lossless reconstruction is not necessary. In general, the performance of an image / video compression algorithm is evaluated based on the resulting compression ratio and reconstruction quality. Compression ratio is directly related to the number of binary codes resulting from compression, where fewer binary codes result in better compression. Reconstruction quality is measured by comparing the reconstructed image / video with the original image / video, where greater similarity results in better reconstruction quality.

[0065] Image / video compression techniques can be divided into video codec methods and neural network-based video compression methods. Video codec schemes employ transform-based solutions, in which statistical dependencies in latent variables such as discrete cosine transform (DCT) and wavelet coefficients are exploited to carefully handcraft entropy codes to model the dependencies in the quantization domain. Neural network-based video compression can be divided into neural network-based codec tools and end-to-end neural network-based video compression. The former is embedded into existing video codecs as a codec tool and serves only as part of the framework, while the latter is a separate framework developed based on neural networks and does not rely on video codecs.

[0066] A series of video codec standards have been developed to accommodate the growing demand for visual content transmission. The International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) has two expert groups, namely the Joint Photographic Experts Group (JPEG) and the Moving Picture Experts Group (MPEG). The International Telecommunication Union (ITU) Telecommunication Standardization Sector (ITU-T) also has a Video Coding Experts Group (VCEG) for the standardization of image / video coding technology. Influential video coding standards published by these organizations include Joint Photographic Experts Group (JPEG), JPEG 2000, H.262, H.264 / Advanced Video Coding (AVC), and H.265 / High Efficiency Video Coding (HEVC). The Versatile Video Coding (VVC) standard was developed by the Joint Video Experts Team (JVET) composed of MPEG and VCEG. Compared with HEVC, VVC reduces the bit rate by an average of 50% at the same visual quality.

[0067] Neural network-based image / video compression / coding is also under development. Example neural network codec network architectures are relatively shallow, and the performance of such networks is not satisfactory. Neural network-based methods benefit from the support of rich data and powerful computing resources, and are therefore better utilized in various applications. Neural network-based image / video compression has shown promising improvements and has been confirmed to be feasible. However, the technology is far from mature and many challenges should be addressed. 2.2 Neural Networks

[0068] Neural networks, also known as artificial neural networks (ANN), are computational models used in machine learning techniques. Neural networks are typically composed of multiple processing layers, and each layer is composed of multiple simple but non-linear basic computational units. One benefit of this deep network is the ability to process data with multiple levels of abstraction and transform the data into different kinds of representations. The representations created by neural networks are not manually designed. Instead, the deep network, including the processing layers, is learned from a large amount of data using general-purpose machine learning programs. Deep learning eliminates the necessity of hand-crafted representations. Therefore, deep learning is considered particularly suitable for processing native unstructured data, such as acoustic and visual signals. The processing of such data has been a long-standing difficulty in the field of artificial intelligence. 2.3 Neural networks for image compression

[0069] Neural networks for image compression can be divided into two categories, including pixel probability models and autoencoder models. Pixel probability models adopt a predictive coding strategy. Autoencoder models adopt a transform-based solution. Sometimes, the two methods are combined together. 2.3.1 Pixel probability modeling

[0070] According to Shannon's information theory, the best method for lossless coding can achieve the minimum coding rate, which is represented as -log2p(x), where p(x) is the probability of symbol x. Arithmetic coding is a lossless coding method that is considered one of the best methods. Given the probability distribution p(x), arithmetic coding makes the coding rate as close as possible to the theoretical limit -log2p(x) without considering the rounding error. Therefore, the remaining problem is to determine the probability, which is very challenging for natural images / videos due to the curse of dimensionality. The curse of dimensionality refers to the problem that increasing the dimensionality causes the dataset to become sparse, so the amount of data needs to be rapidly increased to effectively analyze and organize the data as the number of dimensions increases. Following the predictive coding strategy, one way to model p(x) is to predict the pixel probability one by one in the raster scan order based on previous observations, where x is an image, which can be represented as follows: p(x) = p(x1) p(x2 | x1)... p(x i | x1,..., x i-1 )... p(x m×n | x1,..., x m×n-1 ) (1) where m and n are the height and width of the image, respectively. The previous observation is also called the context of the current pixel. When the image is large, the estimation of the conditional probability can be difficult. Therefore, a simplified method is to limit the context range of the current pixel as follows: p(x) = p(x1) p(x2 | x1)... p(x i | x1,..., xi-k ,…,x i-1 )…p(x m×n |x m×n-k ,…,x m×n-1 ) (2) Where k is a predefined constant that controls the scope of the context.

[0071] It should be noted that this condition can also take into account the sample values ​​of other color components. For example, when encoding and decoding red (R), green (G), and blue (B) (RGB) color components, the R sample depends on the previously encoded and decoded pixels (including R, G, and / or B samples), and the current G sample can be encoded and decoded based on the previously encoded and decoded pixels and the current R sample. In addition, when encoding and decoding the current B sample, the previously encoded and decoded pixels as well as the current R and G samples can also be considered.

[0072] Neural networks can be designed for computer vision tasks and are also effective in regression and classification problems. Therefore, neural networks can be used to predict the probability of a given context x1, x2, ..., x i-1 In the case of estimating p(x i ). According to x i ∈{-1,+1}, pixel probabilities are used for binary images. A neural autoregressive distribution estimator (NADE) is designed for pixel probability modeling. NADE is a feedforward network with a single hidden layer. In another example, the feedforward network can include connections that skip the hidden layer. In addition, parameters can also be shared. Experiments have been performed on the binarized MNIST dataset. In the example, NADE is extended to a real-valued NADE (RNADE) model, where the probability p(x i |x1,…,x i-1 ) is derived using a Gaussian mixture. The RNADE model feed-forward network also has a single hidden layer, but the hidden layer is rescaled to avoid saturation and uses rectified linear units (ReLU) instead of sigmoids. In another example, NADE and RNADE are improved by reorganizing the order of pixels and using deeper neural networks.

[0073] Designing advanced neural networks plays an important role in improving the modeling of pixel probabilities. In an example neural network, a multi-dimensional long short-term memory (LSTM) is used. The LSTM is used in conjunction with a mixture of conditionally Gaussian scale mixture for the probability modeling. The LSTM is a special type of recurrent neural network (RNN) and can be used to model sequential data. Spatial variants of the LSTM can also be used for images later. Several different neural networks can be employed, including recurrent neural networks (RNNs) and convolutional neural networks (CNNs), such as PixelRNN and PixelCNN, respectively. In the PixelRNN, two variants of the LSTM are employed, referred to as row LSTM and diagonal bidirectional LSTM (BiLSTM). The diagonal BiLSTM is specifically designed for images. The PixelRNN incorporates residual connections to help train deep neural networks with up to twelve layers. In the PixelCNN, masked convolutions are used to adjust the shape of the context. The PixelRNN and the PixelCNN are more focused on natural images. For example, the PixelRNN and the PixelCNN treat pixels as discrete values (e.g., 0, 1, …, 255) and predict a multinomial distribution over the discrete values. In addition, the PixelRNN and the PixelCNN handle color images in the RGB color space. Further, the PixelRNN and the PixelCNN perform well on the large-scale image dataset ImageNet. In an example, a gated PixelCNN is used to improve the PixelCNN. The gated PixelCNN achieves comparable performance to the PixelRNN but with lower complexity. In an example, PixelCNN++ is employed, which makes the following improvements to the PixelCNN: uses a discretized logistic mixture likelihood instead of a 256-way multinomial distribution; uses down-sampling to capture structures at multiple resolutions; introduces additional shortcut connections to speed up training; employs dropout for regularization; and targets one pixel combination RGB. In another example, PixelSNAIL combines chance convolutions with self-attention.

[0074] Most of the above methods model the probability distribution directly in the pixel domain. Some designs also model the probability distribution as conditional based on explicit or latent representations. Such models can be expressed as: where h is an additional condition, and p(x) = p(h)p(x|h) indicates that the modeling is divided into an unconditional model and a conditional model. The additional condition can be image label information or a high-level representation. 2.3.2 Autoencoders

[0075] An autoencoder is now described. An autoencoder is trained for dimensionality reduction and includes an encoding component and a decoding component. The encoding component converts a high-dimensional input signal into a low-dimensional representation. The low-dimensional representation can have a reduced spatial dimension, but with a larger number of channels. The decoding component recovers the high-dimensional input from the low-dimensional representation. An autoencoder is able to automatically learn a representation and eliminates the need for handcrafted features, which is also considered one of the most important advantages of neural networks.

[0076] Figure 2 is a schematic diagram showing an example transform coding scheme. An original image x is transformed by an analysis network g a to achieve a latent representation y. The latent representation y is quantized (q) and compressed into bits. The number of bits R is used to measure the coding rate. The quantized latent representation is then transformed by a synthesis network g s to obtain a reconstructed image The distortion (D) is computed in the perceptual space by using a function g p to transform x and to produce z and which are compared to obtain D.

[0077] An autoencoder network can be applied to lossy image compression. The learned latent representation can be encoded from a well-trained neural network. However, it is not trivial to adapt an autoencoder to image compression because the original autoencoder is not optimized for compression and thus cannot be used effectively as a trained autoencoder directly. In addition, there are other major challenges. First, the low-dimensional representation should be quantized before being encoded. However, quantization is not differentiable, which is necessary in backpropagation when training a neural network. Second, the goal is different in the compression scenario because both distortion and bit rate need to be considered. Estimating the bit rate is challenging. Third, a practical image coding scheme should support variable rate, scalability, encoding / decoding speed, and interoperability. Various schemes are being developed for these challenges.

[0078] An example autoencoder for image compression using an example transform coding scheme can be considered as a transform coding strategy. An original image x is transformed with an analysis network y = g a (x), where y is a latent representation to be quantized and coded. A synthesis network transforms the quantized latent representation to obtain a reconstructed image The framework is trained with a rate-distortion loss function where D is the distortion between x and the quantized representation The computed or estimated rate, and λ is the Lagrange multiplier. D can be computed in the pixel domain or in the perceptual domain. Most example systems follow this prototype, and the differences between such systems can only be the network structure or the loss function.

[0079] In terms of network structure, RNNs and CNNs are the most widely used architectures. In the RNN-related category, an example general framework for variable-rate image compression uses RNNs. This example uses a binary quantization to generate codes, and does not consider rate during training. The framework provides scalable coding functionality, with RNNs with convolutional and deconvolutional layers performing well. Another example provides an improved version by upgrading the encoder with a neural network similar to PixelRNN to compress binary codes. Using the multi-scale structural similarity (MS-SSIM) evaluation metric, performance on the Kodak image dataset outperforms JPEG. Another example further improves the RNN-based solution by introducing hidden-state priming. In addition, an SSIM-weighted loss function is also designed, and includes a spatial-domain adaptive bit-rate mechanism. Using MS-SSIM as the evaluation metric, this example achieves better results than the better portable graphics (BPG) on the Kodak image dataset.

[0080] Another example proposes a general framework for rate-distortion optimized image compression. The example system uses multi-element quantization to generate integer codes, and considers bit rate during training. The loss is a joint rate-distortion cost, which can be mean squared error (MSE) or other metrics. The example system adds random uniform noise to simulate quantization during training, and uses the differential entropy of the noisy codes as a proxy for bit rate. The example system uses a generalized divisive normalization (GDN) as the network structure, which includes a linear mapping followed by a non-linear parametric normalization. The effectiveness of GDN on image coding is validated. Another example system includes an improved version of GDN, which uses three convolutional layers, each followed by a down-sampling layer and a GDN layer as the forward transform. Thus, this example version uses three layers of inverse GDN, each followed by an up-sampling layer and a convolutional layer to simulate the inverse transform. In addition, an arithmetic coding method is designed to compress the integer codes. Performance on the Kodak dataset is reported to outperform JPEG and JPEG 2000 in terms of MSE. Another example improves this method by designing a scaling hyper-prior into the autoencoder. The system transforms the latent representation y into z = h a (y), and z is quantized and transmitted as side information. Thus, the inverse transform is implemented with the subnetwork h a (y), and z is quantized and transmitted as side information. Thus, the inverse transform is implemented with the subnetwork h s (y), and z is quantized and transmitted as side information. Thus, the inverse transform is implemented with the subnetwork h (y), and z is quantized and transmitted as side information. Thus, the inverse transform is implemented with the subnetwork h (y), and z is quantized and transmitted as side information. Thus, the inverse transform is implemented with the subnetwork h further utilized during the arithmetic coding. On the Kodak image set, this method slightly underperforms BGP in terms of peak signal-to-noise ratio (PSNR). Another example system further explores the structure in the residual space by introducing an autoregressive model to estimate the standard deviation and mean. This example uses a Gaussian mixture model to further remove the redundancy in the residuals. Using PSNR as the evaluation metric, the performance on the Kodak image set is comparable to VVC. 2.3.3 Hyper-prior model

[0081] Figure 3 An example latent representation 300 of an image is shown. Figure 3 includes images from the Kodak dataset, a visualization of the latent representation y of the images, the standard deviation σ of the latent values, and the latent values y after introducing a hyper-prior network. The hyper-prior network includes an encoder and a decoder that utilize hyper-prior information. In the example shown, the hyper-prior network is trained jointly with the image compression network. Figure 2 In the transform coding method for image compression shown in FIG. 4, the encoder subnetwork uses a parametric analysis transform The image vector x is transformed into a latent representation y, which is then quantized to form Because is a discrete value, the can be losslessly compressed using an entropy coding technique such as arithmetic coding and transmitted as a bit sequence.

[0082] From the latent values and the standard deviation σ of Figure 3 it is evident that there is a significant spatial dependency between the elements of It is worth noting that their scales (standard deviations σ) seem to be coupled spatially. An additional set of random variables can be introduced to capture the spatial dependency and further reduce the redundancy. In this case, the image compression network 400 is as shown in FIG. 5. Figure 4

[0083] Figure 4 is a schematic diagram showing an example network architecture of an autoencoder implementing a hyper-prior model. The upper side shows the image autoencoder network, and the lower side corresponds to the hyper-prior subnetwork. The analysis transform and the synthesis transform are denoted as g a and g s . Q denotes quantization, and AE, AD denote the arithmetic encoder and the arithmetic decoder, respectively. The hyper-prior model includes two subnetworks, an encoder that utilizes hyper-prior information (denoted as h a ) and a decoder that utilizes hyper-prior information (denoted as h s ). The hyper-prior model generates quantized hyper-prior information latent values which include information related to the probability distribution of the samples of the quantized latent values . ​is included in the bitstream and is are transmitted together to the receiver (decoder).

[0084] In the schematic, the upper side of the model is the encoder g as discussed above a and decoder g s The lower side is used to obtain The additional encoder h using super prior information a and decoder h using super prior information s In this architecture, the encoder subjects the input image x to g a , producing a response y with a spatially varying standard deviation. The response y is fed to h a , summarize the distribution of standard deviations in z. Then z is quantized is compressed and transmitted as side information. The encoder then uses the quantized vector To estimate the spatial distribution of the standard deviation σ, and use σ to compress and transmit the quantized image representation The decoder first recovers the The decoder then uses h s to obtain σ, which provides the correct probability estimate to the decoder, thereby also successfully recovering The decoder will then Feed to g s to obtain the reconstructed image.

[0085] When an encoder utilizing super-prior information and a decoder utilizing super-prior information are added to an image compression network, the quantized potential value The spatial redundancy of is reduced. When using an encoder / decoder that utilizes super-prior information, Figure 3 The potential value y in corresponds to the quantized potential value. Compared with the standard deviation σ, the spatial redundancy is significantly reduced because the correlation of the samples of the quantized potential value is low. 2.3.4 Context Model

[0086] Although the hyper-prior model improves the quantified potential , but further improvement can be obtained by utilizing an autoregressive model that predicts the quantized potential value from its causal context (which can be called a context model).

[0087] The term "autoregressive" indicates that the output of a process is later used as the input of a process. For example, the context model subnetwork generates a sample of potential values, which is later used as the input to obtain the next sample.

[0088] Figure 5is a schematic diagram showing an example combined model 500 configured to jointly optimize a context model with a hyper-prior and an autoencoder. The combined model jointly optimizes an autoregressive component of the probability distribution of the latent values (context model) from the causal context of the latent values with a hyper-prior and the underlying autoencoder. The real-valued latent representation is quantized (Q) to create quantized latent values and quantized hyper-prior information latent values which are compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The dashed area corresponds to components performed by a receiver (e.g., decoder) to recover the image from the compressed bitstream.

[0089] The example system utilizes a joint architecture in which both the hyper-prior model subnetwork (encoder with hyper-prior information and decoder with hyper-prior information) and the context model subnetwork are utilized. The hyper-prior model and the context model are combined to learn a probability model over the quantized latent values which is then used for entropy coding. As shown in the schematic, the outputs of the context subnetwork and the decoder with hyper-prior information subnetwork are combined by a subnetwork called the entropy parameters, which generates the mean μ and scale (or variance) σ parameters for a Gaussian probability model. The Gaussian probability model is then used to encode the quantized latent values into a bitstream with the help of an arithmetic encoder (AE) module. In the decoder, the Gaussian probability model is utilized to obtain the quantized latent values from the bitstream by an arithmetic decoder (AD) module.

[0090] Figure 5 The design in FIGS. 1-4 corresponds to the example combined compression method. In this and the next section, the encoding and decoding processes are described separately.

[0091] In an example, the latent samples are modeled as a Gaussian distribution or a Gaussian mixture model (without limitation). According to Figure 5 the context model and the hyper-prior are jointly used to estimate the probability distribution of the latent samples. Since a Gaussian distribution can be defined by a mean and a variance (also called sigma or scale), the joint model is used to estimate the mean and the variance (denoted as μ and σ). 2.3.5 Encoding process using the joint autoregressive hyper-prior model

[0092] Figure 6 An example encoding process 600 is shown. The input image is first processed with an encoder subnetwork. The encoder transforms the input image into a transformed representation called the latent values, denoted by y. y is then input to a quantizer block denoted by Q to obtain quantized latent values are then encoded using an arithmetic encoding module denoted as AE. into a bitstream (bits1). The arithmetic encoding block converts each sample of the quantized latent value into a bitstream (bits1) in order.

[0093] The modules (encoder with hyperpriors, context, decoder with hyperpriors and entropy parameter subnetwork) are used to estimate the probability distribution of the samples of the quantized latent value The latent value y is input to the encoder with hyperpriors which outputs a hyperpriors latent value (denoted by z). The hyperpriors latent value is then quantized and a second bitstream (bits2) is generated using an arithmetic encoding (AE) module. The factorized entropy module generates the probability distribution used to encode the quantized hyperpriors latent value into a bitstream. The quantized hyperpriors latent value includes information about the probability distribution of the quantized latent value

[0094] The entropy parameter subnetwork generates the probability distribution estimate used to encode the quantized latent value The information generated by the entropy parameter usually includes a mean μ and a scale (or variance) σ parameter which are used together to obtain a Gaussian probability distribution. A Gaussian distribution of a random variable x is defined as where the parameter μ is the mean or expectation of the distribution (also the median and mode) and the parameter σ is its standard deviation (or variance or scale). To define a Gaussian distribution, the mean and variance values need to be determined. The entropy parameter module is used to estimate the mean and variance values.

[0095] The subnetwork (decoder with hyperpriors) generates part of the information used by the entropy parameter subnetwork, another part of the information is generated by an autoregressive module called context module. The context module uses the samples that have already been encoded by the arithmetic encoding (AE) module to generate information about the probability distribution of the samples of the quantized latent value. The quantized latent value is usually a matrix composed of many samples. Depending on the dimensions of the matrix the samples can be indicated using indices such as or The samples are usually encoded by the AE one by one using a raster scan order. In a raster scan order, the rows of the matrix are processed from top to bottom, where the samples in a row are processed from left to right. In this case (where the AE encodes the samples into a bitstream using a raster scan order), the context module uses the samples that have been encoded before in the raster scan order to generate information about the sample The information generated by the context module and the decoder with hyperpriors is combined by the entropy parameter module to generate the probability distribution estimate used to encode the quantized latent value ​The probability distribution is encoded into a bitstream (bits1).

[0096] Finally, the first and second bitstreams are transmitted to the decoder as a result of the encoding process. It is important to note that other names can be used for the above modules.

[0097] In the above description, Figure 6 All elements in the are collectively referred to as an encoder. The analytical transform that converts the input image into a latent representation is also referred to as an encoder (or autoencoder). 2.3.6 Decoding process using the joint autoregressive hyperprior model

[0098] Figure 7 An example decoding process is shown.

[0099] In the decoding process, the decoder first receives the first bitstream (bits1) and the second bitstream (bits2) generated by the corresponding encoder. bits2 is first decoded by an arithmetic decoding (AD) module using the probability distribution generated by the factorized entropy subnetwork. The factorized entropy module typically uses a predetermined template to generate the probability distribution, for example, using predetermined mean and variance values in the case of a Gaussian distribution. The output of the arithmetic decoding process of bits2 is which is the quantized hyperprior information latent value. The AD process recovers the AE process applied in the encoder. The processes of AE and AD are lossless, which means that the quantized hyperprior information latent value generated by the encoder

[0100] After obtaining , it is processed by the decoder with hyperprior information, and the output of the decoder with hyperprior information is fed to the entropy parameter module. The three subnetworks employed in the decoder (context, decoder with hyperprior information, and entropy parameter) are the same as the three subnetworks in the encoder. Therefore, the same probability distribution as in the encoder can be obtained in the decoder, which is essential for losslessly reconstructing the quantized latent value from the encoder . As a result, the same version of the quantized latent value

[0101] After the probability distribution (e.g., mean and variance parameters) is obtained by the entropy parameter subnetwork, the arithmetic decoding module decodes the samples of the quantized latent value one by one from the bitstream bits1. From a practical point of view, autoregressive models (context models) are inherently serial, and therefore techniques such as parallelization cannot be used to speed them up. Finally, the fully reconstructed quantized latent value is input to the synthesis transform (in Figure 7The module denoted as decoder) to obtain a reconstructed image.

[0102] In the above description, Figure 7 All elements in the decoder are collectively referred to as the decoder. The synthesis transform that converts the quantized latent values to the reconstructed image is also referred to as the decoder (or auto-decoder). 2.4 Neural networks for video compression

[0103] Similar to video codec technology, neural image compression is the basis for intra-frame compression in neural video compression. Therefore, the development of neural video compression technology lags behind the development of neural image compression because neural video compression technology has higher complexity and therefore requires more effort to address the corresponding challenges. Compared to image compression, video compression requires efficient methods to eliminate inter-frame redundancy. Inter-frame prediction is a major step in these example systems. Motion estimation and compensation are widely used in video codecs, but are usually not implemented through trained neural networks.

[0104] Neural video compression can be divided into two categories according to the target scenario: random access and low latency. In the case of random access, the system allows decoding to start from any point in the sequence, usually dividing the entire sequence into multiple separate segments, and allowing each segment to be decoded independently. In the case of low latency, the system aims to reduce the decoding time, whereby the frames preceding in the time domain can be used as reference frames to decode the subsequent frames. 2.4.1 Low latency

[0105] The example system first divides the video sequence frames into blocks, and each block is coded according to an intra-coding mode or an inter-coding mode. If intra-coding is selected, there is an associated auto-encoder to compress the block. If inter-coding is selected, motion estimation and compensation are performed, and a trained neural network is used for residual compression. The output of the auto-encoder is directly quantized and coded by the Huffman method.

[0106] Another neural network-based video codec scheme employs a Pixel Motion CNN (PixelMotion CNN). The frames are compressed in time domain order, and each frame is divided into blocks, which are compressed in raster scan order. Each frame is first extrapolated with the two preceding reconstructed frames. When a block is to be compressed, the extrapolated frames are fed into the PixelMotion CNN along with the context of the current block to derive a latent representation. Then, the residual is compressed by a variable rate image scheme. The performance of this scheme is comparable to H.264.

[0107] Another example system employs an end-to-end neural network based video compression framework, where all modules are implemented with neural networks. The scheme takes the current frame and the previously reconstructed frames as input. The optical flow is derived with a pre-trained neural network as motion information. The motion information is warped with the reference frame, followed by a neural network that generates the motion compensated frame. The residual and the motion information are compressed with two separate neural autoencoders. The whole framework is trained with a single rate-distortion loss function. The example system achieves better performance than H.264.

[0108] Another example system employs an advanced neural network based video compression scheme. This system inherits and extends the video codec scheme with neural networks with the following main features. First, the system uses only one autoencoder to compress the motion information and the residual. Second, the system uses motion compensation with multiple frames and multiple optical flows. Third, the system uses online states that are learned and propagated through subsequent frames over time. This scheme achieves better performance than the HEVC reference software in MS-SSIM.

[0109] Another example system uses an extended end-to-end neural network based video compression framework. In this example, multiple frames are used as reference. The example system is thus able to provide more accurate prediction of the current frame by using multiple reference frames and associated motion information. In addition, motion field prediction is deployed to remove motion redundancy along the temporal channel. A post-processing network is also used to remove reconstruction artifacts from the previous process. The performance of this system is significantly better than H.265 in both PSNR and MS-SSIM.

[0110] Another example system uses scale space flow to replace optical flow by adding a scale parameter. This example system can achieve better performance than H.264. Another example system uses a multi-resolution representation for optical flow. Specifically, the motion estimation network produces multiple optical flows with different resolutions, and lets the network learn which one to choose under a loss function. The performance is better than H.265. 2.4.2 Random Access

[0111] Frame interpolation is employed in another example. Key frames are first compressed with a neural image compressor, and the remaining frames are compressed in a hierarchical order. The system performs motion compensation in the perceptual domain by deriving feature maps of the original frames at multiple spatial scales and warping the feature maps using motion. The result is used for the image compressor. This method is comparable to H.264.

[0112] Example systems use methods for interpolation-based video compression. The interpolation model combines motion information compression and image synthesis. The same autoencoder is used for both the image and the residual. Another example system employs a neural network-based video compression method based on a variational autoencoder with a deterministic encoder. Specifically, the model includes an autoencoder and an autoregressive prior. Unlike previous methods, this system takes a group of pictures (GOP) as input and incorporates a three-dimensional (3D) autoregressive prior by considering temporal correlation when coding the latent representation. The system provides performance comparable to H.265. 2.5 Preliminaries

[0113] Almost all natural images and / or videos are in digital format. A grayscale digital image can be represented by where D is a set of pixel values, m is the image height, and n is the image width. For example, is an example setup, and in this case Thus, a pixel can be represented by an 8-bit integer. An uncompressed grayscale digital image has 8 bits per pixel (bpp), while a compressed bit is certainly less.

[0114] Color images are typically represented in multiple channels to record color information. For example, in the RGB color space, an image can be represented by where the three separate channels store red, green, and blue information. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image has 24 bpp. Digital images / videos can be represented in different color spaces. Neural network-based video compression schemes are mostly developed in the RGB color space, while video codecs typically use the YUV color space to represent video sequences. In the YUV color space, an image is decomposed into three channels, i.e., luminance (Y), blue-difference chroma (Cb), and red-difference chroma (Cr). Y is the luminance component, and Cb and Cr are the chrominance components. The compression benefit of YUV is that Cb and Cr are typically downsampled to achieve pre-compression because the human visual system is less sensitive to the chrominance components.

[0115] A color video sequence is composed of multiple color images (also called frames) to record the scene at different timestamps. For example, in the RGB color space, a color video can be represented by X = {x0, x1,..., xT}, where T is the number of frames in the video sequence, and t xTis a color image at time T. T-1 If m = 1080, n = 1920, ​And if the video has 50 frames per second (fps), the data rate of this uncompressed video is 1920x1080x8x3x50 = 2,488,320,000 bits per second (bps). This results in approximately 2.32 Giga bits per second (Gbps), which uses a large amount of storage and should be compressed before transmission over the Internet.

[0116] Generally, lossless methods can achieve a compression ratio of about 1.5 to 3 for natural images, which is significantly lower than the streaming requirement. Therefore, lossy compression is adopted to achieve a better compression ratio, but at the cost of producing distortion. Distortion can be measured by computing the mean squared error between the original image and the reconstructed image, e.g., based on MSE. For a grayscale image, MSE can be computed with the following equation. Correspondingly, the quality of the reconstructed image compared to the original image can be measured by the peak signal-to-noise ratio (PSNR): where is the maximum value in, e.g., 255 for an 8-bit grayscale image. There are other quality assessment metrics, such as structural similarity (SSIM) and multi-scale SSIM (MS-SSIM).

[0117] To compare different lossless compression schemes, one can compare the compression ratio for a given rate, or vice versa. However, to compare different lossy compression methods, the comparison must take both the rate and the reconstruction quality into account. For example, this can be achieved by computing the relative rate at several different quality levels, and then averaging the rates. The average relative rate is called the Bjontegaard’s difference rate (BD rate). There are other aspects to evaluate image and / or video coding schemes, including encoding / decoding complexity, scalability, robustness, etc. 2.5.1 Grouped Convolution

[0118] The core of grouped convolution (GC) is to divide the filters (learned weights) into groups and apply convolution using the filters of each group, then stitch the output feature maps. It is used to reduce the memory and computation complexity. For example, we define the input feature map size as c1xHxW, where c1 is the number of channels of the input feature map. Then the filter size is c2xc1xh1xw1, where c2 is the number of output channels, h1 and w1 are the filter kernel height and width, respectively.

[0119] In a normal convolution operation as shown in Figure 8 , the number of parameters of the filter is c2xc1xh1xw1.

[0120] Figure 8 An example convolutional layer is shown, where the two smaller blocks represent learned parameters and the two larger blocks represent feature maps.

[0121] Figure 9 An example convolutional layer with 2 groups (grouped convolution) is shown. Now, each filter in the grouped convolutional layer is exactly half the depth, i.e., half the parameters and half the computation operations.

[0122] Figure 9 An example of grouped convolution with 2 groups is shown. The operation becomes equivalent to having two convolutional layers side-by-side, each seeing half the input channels and producing half the output channels, both of which are subsequently concatenated. The convolutional filter depth is reduced to half the size but with 2 groups, and each group includes channels. In this case, the total number of filter parameters becomes which is half the number of parameters compared to normal convolution as shown in Figure 9 . Therefore, if the number of groups is N and the convolution kernel size is unchanged, the number of parameters is 3. Technical problems solved by the disclosed technical solutions 3.1 Core problem

[0123] Figure 10 An example decoder architecture with decoupled processing is shown. As shown in Figure 10 , the decoder with decoupled processing decouples the arithmetic decoding process (the process of receiving bits and generating ) from the autoregressive loop (including the combination module of the decoder with hyper-prior information, the masked convolution, and the prediction fusion network). In this architecture, the autoregressive loop consumes a lot of time because it repeats the process until all elements are decoded. To reduce the decoding time complexity, it is of great significance to simplify this part. Since the masked convolution network only includes one convolutional layer, it is difficult to further simplify, so it is more effective to simplify the prediction fusion network (represented in bold outline in Figure 10 ).

[0124] In addition, the variance decoder network with hyper-prior information (represented in bold outline in Figure 10 ) used to obtain the Gaussian variance parameter σ includes 6 convolutional layers. Simplification can also be applied to this network to reduce the number of arithmetic operations and decoding time.

[0125] In short, the present disclosure proposes a simplification scheme for the prediction fusion network and the variance decoder with hyper-prior information as shown in bold in Figure 10 , aiming to reduce the decoding time or the number of arithmetic operations (e.g., measured by kMac / pixel). 3.1 Details of the problem

[0126] Figure 11 An example prediction fusion network for luminance is shown. Figure 12 An example variance decoder exploiting super-prior information for luminance is shown. Figure 11 to Figure 12 The design of the prediction fusion network and the variance decoder exploiting super-prior information for the luminance component are shown, respectively. The network follows the same architecture for the chrominance components but generally has a reduced (e.g., half) number of channels for each convolutional layer. Prediction fusion network The prediction fusion network comprises 6 convolutional layers with 1x1 kernels. It accepts two input sources from the mask convolutional network and the decoder exploiting super-prior information and generates the mean parameter μ. The number of input channels and the number of output channels are 4C and C, respectively, where C is an integer indicating the base unit for the number of channels when designing the neural network. The number of channels for each convolutional layer is listed in Table 1. Table 1. Number of channels for the prediction fusion network. The prediction fusion network can be simplified from two aspects. 1) Reduce the number of channels for certain convolutional layer(s). 2) Reduce the total number of convolutional layers. Variance decoder exploiting super-prior information The variance decoder exploiting super-prior information takes as input the quantized super-prior information latent values and generates the Gaussian variance parameter σ. The number of channels for both the input and the output is C, however, the spatial dimension of the output is larger than that of the input, i.e., from w6xh6to w4xh4. The spatial dimension change is done by the first two convolutional layers. Figure 12 Details of the variance decoder exploiting super-prior information are shown, which comprises 6 convolutional layers. The number of channels, kernel size, and spatial dimension change for each convolutional layer are listed in Table 2. It can be observed that the first convolutional layer performs a spatial dimension change with a larger kernel size (5x5), while the following convolutional layers are responsible for refining the output with 3x3 kernel size. From the perspective of coding gain and decoding complexity, this design is not proven to be optimal. Table 2. Details of the variance decoder exploiting super-prior information. The variance decoder exploiting super-prior information can be simplified from the following aspects. 1) Reduce the number of channels for certain convolutional layer(s). 2) Reduce the total number of convolutional layers. 3) reduce the core size. 4. List of solutions and embodiments The following detailed aspects should be considered illustrative of examples explaining the general concepts. The examples should not be construed as being limited in any way. Furthermore, the examples can be combined in any way. 4.1 Specific examples The goal of the present disclosure is to simplify the prediction fusion network and the variance decoder exploiting hyper-prior information such that the decoding time or kMac / pixel can be reduced. In designing the network, a typical value of C is 128. The following examples use C = 128 to explain the proposed solutions. However, different values of C can be used in the following designs. Prediction fusion network As shown in Table 1, the number of channels is always a multiple of (i.e., 64). Therefore, for the chroma components, the number of channels is always a multiple of (i.e., 32). From the implementation efficiency point of view, it is recommended to make the number of channels a power of 2. An example design can be implemented by reducing the number of channels of the luma and chroma prediction fusion networks by and respectively, as shown in Table 3. Table 3. Example of the number of channels for the simplified prediction fusion network (16x) (Example 1.1) Alternatively, the number of channels for the chroma prediction fusion network can also be a multiple of (i.e., 8). The detailed number of channels for the luma and chroma prediction fusion networks is listed in Table 4. Table 4. Example of the number of channels for the simplified prediction fusion network (8x) (Example 1.2) Alternatively, as shown in Figure 5 , the number of channels is only reduced on the first convolutional layer, from to for the luma and from to for the chroma. Table 5. Example of the number of channels for the simplified prediction fusion network (reduce # channels) (Example 1.3) Based on Example 1.3, it is also possible to make the number of channels for the chroma a multiple of (similar to Example 1.1). The details are listed in Table 6. Table 6. Example of number of channels for a simplified prediction fusion network (16x) (Example 1.4) Based on Example 1.3, the number of channels for chroma can also be a multiple of (32x) (similar to Example 1.2). Details are shown in Table 7. Table 7. Example of number of channels for a simplified prediction fusion network (8x) (Example 1.5) Alternatively, the total number of convolutional layers can be reduced. When we reduce the number of convolutional layers from 6 to 5, the following examples can be implemented as shown in Tables 8-10. Table 8. Example of number of channels for a simplified prediction fusion network (reduce # layers) (Example 1.6) Table 9. Example of number of channels for a simplified prediction fusion network (16x) (Example 1.7) Table 10. Example of number of channels for a simplified prediction fusion network (8x) (Example 1.8) As an alternative to Table 8, the number of channels can also be designed in a more uniform way, i.e., reducing from 4C to C. This example is listed in Table 11. Table 11. Example of number of channels for a simplified prediction fusion network (Example 1.9) Tables 8-11 reduce the total number of layers from 6 to 5. In a more aggressive way, the number of layers can be reduced from 6 to 4. A simple design is listed in Table 12, where the number of channels for these 4 layers gradually reduces from 4C to C. Table 12. Example of number of channels for a simplified prediction fusion network (Example 1.10) Similarly, the number of channels (for the chroma network) can be a multiple of (32x), (16x), or (8x), as shown in Tables 13-15. Table 13. Example of number of channels for a simplified prediction fusion network (32x) (Example 1.11) Table 14. Example of number of channels for a simplified predictive fusion network (16x) (Example 1.12) Table 15. Example of number of channels for a simplified predictive fusion network (8x) (Example 1.13) Variance decoder with hyper-prior information As shown in Table 2, the variance decoder with hyper-prior information for luma or chroma comprises 6 convolutional layers, where the first 2 convolutional layers use 5x5 kernels and are responsible for the up-sampling, and the last 4 convolutional layers use 3x3 kernels and have no spatial dimension change. A simplification can be made to reduce the kernel size from 5x5 to 4x3 or 3x4 or one following the other. Details are shown in Tables 16-18. Table 16. Example of simplified variance decoder with hyper-prior information (4x3) (Example 2.1) Table 17. Example of simplified variance decoder with hyper-prior information (3x4) (Example 2.2) Table 18. Example of simplified variance decoder with hyper-prior information (3x4) (Example 2.3) As an alternative to Table 18, an implementation with changed order can also be designed, i.e. 4x3 followed by 3x4. In addition to Tables 16-18, a simplification can also be made to reduce the kernel size from 5x5 to 4x4. And the 4x4 up-sampling convolution is implemented with a 2D convolution and a pixel shuffle layer instead of a 2D transpose convolution. For example in PyTorch, it is implemented with torch.nn.Conv2d() followed by torch.nn.PixelShuffle(). An example is shown in detail in Table 19. Table 19. Example of simplified variance decoder with hyper-prior information (4x4, pixel shuffle) (Example 2.4) As an alternative to Table 19, the kernel size can be 2x2, 3x3, 3x4, 4x3 or one following the other. Tables 16-19 change the up-sampling convolutional layer kernel size or its implementation. Alternatively, we can change the number of channels. In one example, we can reduce the number of channels of the chroma with hyper-prior variance decoder to (16x) or (8x) times. Details are shown in Tables 20-21. Table 20. Example of simplified chroma with hyper-prior variance decoder (16x) (Example 2.5) Table 21. Example of simplified chroma with hyper-prior variance decoder (8x) (Example 2.6) In addition to reducing the kernel size (Tables 16-19) or reducing the number of channels (Tables 20-21), the number of convolutional layers can also be reduced. In the original design, there are six convolutional layers, the present invention proposes to reduce to 5 or 4 as shown in Tables 22-26. Table 22. Example of simplified chroma with hyper-prior variance decoder (5 layers) (Example 2.7) Table 23. Example of simplified chroma with hyper-prior variance decoder (5 layers) (Example 2.8) Table 24. Example of simplified chroma with hyper-prior variance decoder (4 layers) (Example 2.9) Table 25. Example of simplified chroma with hyper-prior variance decoder (4 layers) (Example 2.10) Table 26. Example of simplified chroma with hyper-prior variance decoder (4 layers) (Example 2.11) Alternatively, the most simple design in practice can be a 4-layer design combined with 2x2 convolution + pixel shuffle as shown in Tables 27 and 28. A simpler solution can be designed using group convolution as shown in Table 29. In Table 29, it shows a 4-layer architecture where the first 2 layers are 2x2 convolution + pixel shuffle for up-sampling and the last 2 layers are 3x3 group convolution layers with 2 groups. Alternatively, group convolution can be applied to only the third or fourth layer. In addition, the number of groups can vary in any case. Table 27. Example of simplified variance decoder with super-prior information (4 layers, 2x2) (Example 2.12) Table 28. Example of simplified variance decoder with super-prior information (4 layers, 2x2) (Example 2.13) Table 29. Example of simplified variance decoder with super-prior information (4 layers, 2x2 group convolution) In the table GC indicates "group convolution". (Example 2.14) In addition to the designs listed above, the proposed designs can be combined in any way. Specifically, a. In the designs for the prediction fusion network (described in Tables 3-15), changing the number of channels (Table 3-7) can be combined in any way with reducing the number of convolution layers (Tables 8-15). b. In the designs for the variance decoder with super-prior information (described in Tables 16-26), changing the kernel size / implementation (Tables 16-19), changing the number of channels (Tables 20-21), and reducing the number of layers (Tables 22-26) can be combined in any way. c. In the designs of Tables 3-26, the activation function LeakyReLU can be replaced by ReLU or any other kind of activation function. When quantization is needed, qLeakyReLU (i.e., quantized LeakyReLU) can be replaced by qReLU (i.e., quantized ReLU). d. Any design of the prediction fusion network (Tables 3-15 or one of item a) can be combined with any design of the variance decoder with super-prior information (Tables 16-26 or one of item b). 4.2 Further details of the examples 1. The prediction fusion network can be modified for simplicity purposes in the following ways. a) In one example, the number of channels for each convolution layer in the chroma prediction fusion network can be adjusted or reduced to a multiple of or and the number of channels for each layer of the luma prediction fusion network is twice that in the chroma prediction fusion network. b) In one example, the number of layers can be reduced from 6 to 5, 4, or 3. c) In item b), the number of channels for each layer can be adjusted to fit the design. d) In one example, whether the number of channels has been changed can be signaled with a syntax element in the bitstream. e) In one example, multiple designs of the prediction fusion network can exist in the software and a syntax element is signaled in the bitstream to indicate which one is used. f) In one example, the modified prediction fusion network can be applied to the luma or chroma components, or both. 2. The variance decoder with hyper-prior information can be modified for simplicity purposes in the following ways. a) In one example, the 5x5 convolution kernel (for upsampling) can be reduced to 4x3 or 4x3 or one following the other. b) In one example, the 5x5 convolution kernel implemented with 2D transposed convolution can be replaced by a 4x4 convolution kernel implemented with 2D convolution layer + pixel shuffle layer. c) In one example, two consecutive 5x5 upsampling layers can be replaced by three layers with a regular convolution layer in the middle, i.e. 5x5 upsampling + convolution + 5x5 upsampling. d) In one example, the number of channels of each convolution layer of the variance decoder with hyper-prior information for chroma can be adjusted to be a multiple of or and the number of channels of the variance decoder with hyper-prior information for luma is twice that of the variance decoder with hyper-prior information for chroma. e) In one example, the number of layers can be reduced from 6 to 5, 4 or 3. f) In item e), it can be accompanied by an adjustment of the number of channels using the rules defined in item a). g) In one example, a flag can be signaled in the bitstream to indicate whether the modified variance decoder with hyper-prior information or the original variance decoder with hyper-prior information is used. h) In one example, multiple modified designs of the variance decoder with hyper-prior information can be included in the decoder and a syntax can be used to indicate which one is used. i) In one example, the modified variance decoder with hyper-prior information can be applied to luma or chroma or both. Further examples 1. Whether and / or how the methods disclosed above can be signaled at block level / sequence level / picture group level / picture level / slice level / tile group level, such as in the coding structure of CTU / CU / TU / PU / CTB / CB / TB / PB, or sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header. 2. Whether and / or how the above disclosed method can be applied can depend on the coded information, such as block size, color format, single tree partitioning / dual tree partitioning, color component, slice / picture type. 3. The proposed method disclosed herein can be used in other coding tools that require chroma fusion. 4. The above disclosed syntax element can be binarized as a flag, a fixed length code, an EG(x) code, a unary code, a truncated unary code, a truncated binary code, etc. It can be signed or unsigned. 5. The above disclosed syntax element can be coded with at least one context model. Or it can be bypass coded. 6. The above disclosed syntax element can be signaled in a conditional way. a. SE is signaled only when the corresponding function is applicable. b. SE is signaled only when the dimension (width and / or height) of the block satisfies a condition. 7. The above disclosed syntax element can be signaled at block level / sequence level / group of pictures level / picture level / slice level / tile group level, such as in the coding structure of CTU / CU / TU / PU / CTB / CB / TB / PB, or sequence header / picture header / SPS / VPS / DPS / DCI / PPS / APS / slice header / tile group header. 4.3 Benefits of the example

[0127] According to the present disclosure, the simplified prediction fusion network and the simplified variance decoder with hyper-prior information will consume less time or require less number of arithmetic operations in the decoding time, e.g. measured in kMac / pixel. 5 Embodiment

[0128] For the simplified prediction fusion network, the following implementations are deployed. Note that 32x or 16x indicates that the number of channels in the chroma network is a multiple of 32 or 16 (assuming C=128). In the following figures, the luminance network architecture is shown. The chroma network follows the same design but with half the number of channels. 1) 6 layers, 32x 2) 6 layers, 16x 3) 5 layers, 32x 4) 5 layers, 16x 5) 4 layers, 32x 6) 4 layers, 16x

[0129] Figure 13 to Figure 19 Implementation details are shown (for the luminance component). Figure 13A simplified predictive fusion network (5 layers, 32x) is shown as an example. Figure 14 A simplified predictive fusion network (5 layers, 16x) is shown as an example. Figure 15 A simplified predictive fusion network (4 layers, 32x) is shown as an example. Figure 16 A simplified predictive fusion network (4 layers, 16x) is shown as an example. Figure 17 A simplified predictive fusion network (6 layers, 32x) is shown as an example. Figure 18 A simplified predictive fusion network (6 layers, 16x) is shown as an example. Figure 19 A simplified predictive fusion network (4 layers, 16x) is shown as an example. Compared to the example in Figure 14 The first layer in this design has 11 / 4C channels.

[0130] It should be noted that for the designs shown in Figure 13-19 The LeakyReLU activation layer can be replaced by a ReLU activation layer for the designs shown in Figure 23 and Figure 29 to Figure 31 The upsampling factor is 2, i.e. both the horizontal and vertical spatial dimensions of the feature maps are doubled after the pixel shuffle layer. In the following items 4) and 11) - 14), the description of “number of channels” refers to “after the pixel shuffle layer”.

[0131] Figure 20 A simplified variance decoder exploiting super-prior information (4x3) is shown as an example. Figure 21 A simplified variance decoder exploiting super-prior information (5 layers, 4x3) is shown as an example. Figure 22 A simplified variance decoder exploiting super-prior information (5 layers, 4x4, upsample + conv + upsample) is shown as an example. Figure 23 A simplified variance decoder exploiting super-prior information (3x3 with pixel shuffle) is shown as an example. Figure 24 A simplified variance decoder exploiting super-prior information (3x3 with pixel shuffle, 5 layers) is shown as an example. Figure 25 A simplified variance decoder exploiting super-prior information (4x3, 5 layers, C→3 / 2C→2C→3 / 2C→C) is shown as an example. Figure 26 A simplified variance decoder exploiting super-prior information (4x3, 5 layers, C→C→C→C→C) is shown as an example. Figure 27 A simplified variance decoder exploiting super-prior information (4x3, 4 layers, C→3 / 2C→3 / 2C→C) is shown as an example. Figure 28An example simplified variance decoder with hyper-prior information utilization (4x3, 4 layers, 3 / 2C→2C→3 / 2C→C) is shown. Figure 29 An example simplified variance decoder with hyper-prior information utilization (4x3, 4 layers, C→C→C→C) is shown. Figure 30 An example simplified variance decoder with hyper-prior information utilization (5 layers, pixel shuffle, C→3 / 2C→5 / 3C→3 / 2C→C) is shown. Figure 31 An example simplified variance decoder with hyper-prior information utilization (5 layers, pixel shuffle, CCCC) is shown. Figure 32 An example simplified variance decoder with hyper-prior information utilization (4 layers, pixel shuffle, C→3 / 2C→3 / 2C→C) is shown. Figure 33 An example simplified variance decoder with hyper-prior information utilization (4 layers, pixel shuffle, C-C-C-C) is shown. Figure 34 An example simplified variance decoder with hyper-prior information utilization (4 layers, 2x2 pixel shuffle) is shown. Figure 35 An example simplified variance decoder with hyper-prior information utilization (4 layers, 2x2 pixel shuffle, group convolution) is shown.

[0132] The following simplifications are deployed for a variance decoder with hyper-prior information utilization. 1) Change 5x5 kernel to 4x3. 2) Change 5x5 kernel to 4x3 and change 6 layers to 5 layers with number of channels C→3 / 2C→5 / 2C→3 / 2C→C. 3) Change 5x5 kernel to 4x4, change 6 layers to 5 layers, and rearrange from up+up+conv to up+conv+up. 4) Change 5x5 transpose convolution layer to 3x3 convolution layer with pixel shuffle, change 6 layers to 4 layers with number of channels 2C→3 / 2C→3 / 2C→C. 5) Change two consecutive layers with 5x5 kernel to two layers with 4x3 and 3x4 kernel respectively. 6) Change 5x5 convolution kernel to 4x3 and change 6 layers to 5 layers with number of channels C→3 / 2C→2C→3 / 2C→C. Figure 25 7) Change 5x5 convolution kernel to 4x3 and change 6 layers to 5 layers with number of channels C→C→C→C→C. Figure 26 8) Change 5x5 convolution kernel to 4x3 and change 6 layers to 4 layers with number of channels C→3 / 2C→3 / 2C→C. Figure 27 ​​​9) Change 5x5 kernel to 4x3 and 6 layers to 4 layers with number of channels 3 / 2C→2C→3 / 2C→C. Figure 28 10) Change 5x5 kernel to 4x3 and 6 layers to 4 layers with number of channels C→C→C→C. Figure 29 11) Change 5x5 transpose convolution layer to 3x3 convolution layer with pixel shuffle, 6 layers to 5 layers with number of channels C→3 / 2C→5 / 2C→3 / 2C→C. Figure 30 12) Change 5x5 transpose convolution layer to 3x3 convolution layer with pixel shuffle, 6 layers to 5 layers with number of channels C→C→C→C→C. Figure 31 13) Change 5x5 transpose convolution layer to 3x3 convolution layer with pixel shuffle, 6 layers to 4 layers with number of channels C→3 / 2C→3 / 2C→C. Figure 32 14) Change 5x5 transpose convolution layer to 3x3 convolution layer with pixel shuffle, 6 layers to 4 layers with number of channels C→C→C→C. Figure 33 15) 4 layers architecture, first 2 layers are 2x2 + pixel shuffle for up-sampling and last 2 layers are 3x3 convolution layers. Figure 34 16) Based on 15), last 2 layers are 3x3 group convolution with 2 groups. Figure 35 17) Change 5x5 kernel to 3x3 and 1x1 and 6 layers to 4 layers with number of channels C→C→16C. Figure 36B

[0133] The specific design (for the luminance component) is shown in Figure 13 to Figure 24

[0134] It should be noted that in the design shown in Figure 20 to Figure 33 the activation layer qLeakyReLU (i.e. quantized LeakyReLU) can be replaced by qReLU (i.e. quantized ReLU).

[0135] ​​​​​​​​​​More details of embodiments of the present disclosure will be described below, which are related to neural network based visual data coding. As used herein, the term “visual data” can refer to video, image, picture in video, or any other visual data suitable to be coded. As used herein, the terms “(first) module for prediction fusion” and “prediction fusion network” can be used interchangeably. The terms “second module for variance decoder with super-prior information” and “variance decoder with super-prior information” can be used interchangeably.

[0136] As discussed above, in existing designs, the auto-regressive loop in neural network (NN) based model includes a context model network, a prediction fusion network, and a variance decoder with super-prior information. The prediction fusion network and the variance decoder with super-prior information can consume a large amount of time during the auto-regressive process. This leads to an increase in the time required for the entire coding process, resulting in a decrease in coding efficiency.

[0137] To solve the above problems and some other problems not mentioned, a visual data processing solution as described below is disclosed. Embodiments of the present disclosure should be considered as examples to explain the general concepts and should not be interpreted in a narrow way. In addition, these embodiments can be applied individually or combined in any way.

[0138] Figure 37 A flowchart of a method 3600 for visual data processing according to some embodiments of the present disclosure is shown. The method 3600 is implemented during conversion between visual data and a bitstream of visual data.

[0139] At block 3610, for conversion between visual data and a bitstream of visual data, a neural network (NN) based model including a first module for prediction fusion and a second module for a variance decoder with super-prior information is applied to the video data. In some embodiments, at least one of the following can be satisfied for the first module for prediction fusion: a number of channels in a convolutional layer is less than or equal to a first threshold number, a number of convolutional layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size. Alternatively or additionally, at least one of the following can be satisfied for the second module for a variance decoder with super-prior information: a number of channels in a convolutional layer is less than or equal to a first threshold number, a total number of convolutional layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size. For example, the first threshold number can be 6. For example, the second threshold number can be a multiple of (e.g., 64). As another example, the threshold size can be 5x5.

[0140] ​At block 3620, a conversion between the visual data and a bitstream of the visual data is performed based on the first module for prediction fusion and the second module for variance decoder with super-prior information. In some embodiments, the conversion can include encoding the visual data into the bitstream. Additionally or alternatively, the conversion can include decoding the visual data from the bitstream. For example, Figure 9 The decoding model shown in FIG. 3B can be used to decode the visual data from the bitstream.

[0141] In some embodiments, the number of convolution layers in the first module for prediction fusion is one of: 6, 5, 4, or 3. In some other embodiments, the number of channels for each layer in the first module for prediction fusion is adjustable.

[0142] In some embodiments, the bitstream includes a syntax element indicating the number of channels for each layer in the first module for prediction fusion. In some other embodiments, a plurality of first modules for prediction fusion are stored, and wherein the bitstream includes a syntax element indicating which first module for prediction fusion is used in the conversion. In some embodiments, the first module for prediction fusion is applied to one or more of: a luma component or a chroma component.

[0143] In some embodiments, the first module for prediction fusion includes a chroma prediction fusion network and a luma prediction fusion network, the chroma prediction fusion network having convolution layers, wherein each convolution layer has a number of channels that is a multiple of or twice the number of channels in the chroma prediction fusion network, wherein C is an integer. For example, C can be 128. Alternatively, C can be 64. It should be noted that C can be any suitable number.

[0144] In some embodiments, for each convolution layer in the luma prediction fusion network, the number of channels is equal to one of: or C; and for each convolution layer in the chroma prediction fusion network, the number of channels is equal to one of: or An example design can be implemented by reducing the number of channels for the luma and chroma prediction fusion networks by and respectively, as shown in Table 3.

[0145] In some embodiments, for each convolution layer in the luma prediction fusion network, the number of channels is equal to one of: or C, and for each convolution layer in the chroma prediction fusion network, the number of channels is equal to one of: or For example, the number of channels of the chroma prediction fusion network can also be a multiple of (i.e., 8). The detailed number of channels of the luma and chroma prediction fusion networks are listed in Table 4.

[0146] In some embodiments, the number of channels in the first convolutional layer in the luma prediction fusion network is and the number of channels in the first convolutional layer in the chroma prediction fusion network is For example, as shown in Figure 5 the number of channels is only reduced on the first convolutional layer, from to for luma and from to

[0147] In some embodiments, for each convolutional layer in the luma prediction fusion network, the number of channels is equal to one of: or C; and for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of: or For example, the number of channels for chroma can be a multiple of (similar to Example 1.1), which is shown in Table 6.

[0148] In some embodiments, for each convolutional layer in the luma prediction fusion network, the number of channels is equal to one of: or C, and for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of: or For example, the number of channels for chroma can be a multiple of (similar to Example 1.2), which is shown in Table 7.

[0149] In some embodiments, the total number of convolutional layers can be reduced. For example, the number of convolutional layers in the luma prediction fusion network is 5 (i.e., from 6 to 5), and the number of convolutional layers in the chroma prediction fusion network is 5 (i.e., from 6 to 5).

[0150] In some embodiments, for each convolutional layer in the luma prediction fusion network, the number of channels is equal to one of: 3C, 2C, or C, and for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of: C, or Examples are shown in Table 8. In some other embodiments, for each convolutional layer in the luma prediction fusion network, the number of channels is equal to one of the following: or C, and for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or Examples are shown in Table 9. In some further embodiments, for each convolutional layer in the luma prediction fusion network, the number of channels is equal to one of the following: or C, and for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or Examples are shown in Table 10.

[0151] In some embodiments, for each convolutional layer in the luma prediction fusion network, the number of channels is equal to one of the following: or C, and for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: C, or For example, the number of channels can also be designed in a more evenly decreasing manner, i.e., from 4C to C, which is shown in Table 11.

[0152] In some embodiments, the number of layers can be reduced from 6 to 4. A simple design is listed in Table 12, where the number of channels of these 4 layers gradually decreases from 4C to C. For example, the number of convolutional layers in the luma prediction fusion network is 4, and the number of convolutional layers in the chroma prediction fusion network is 4.

[0153] In some embodiments, for each convolutional layer in the luma prediction fusion network, the number of channels is equal to one of the following: 4C, 3C, 2C, or C, and for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: 2C, C, or Examples are shown in Table 12.

[0154] In some embodiments, for each convolutional layer in the luma prediction fusion network, the number of channels is equal to one of the following: or C, and for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or For example, the number of channels (of the chroma network) can be a multiple of (32x) as shown in Table 13.

[0155] In some embodiments, the number of channels is equal to one of the following for each convolutional layer in the luminance prediction fusion network: or C, and the number of channels is equal to one of the following for each convolutional layer in the chrominance prediction fusion network: or For example, the number of channels (of the chrominance network) can be a multiple of (16x), as shown in Table 14.

[0156] In some embodiments, the number of channels is equal to one of the following for each convolutional layer in the luminance prediction fusion network: or C, and the number of channels is equal to one of the following for each convolutional layer in the chrominance prediction fusion network: or For example, the number of channels (of the chrominance network) can be a multiple of (8x), as shown in Table 15.

[0157] In some embodiments, the second module for the variance decoder utilizing hyper-prior information includes a 4x3 convolution kernel, or a 3x4 convolution kernel, or a combination thereof. For example, a simplification can be made to reduce the kernel size from 5x5 to 4x3 or 3x4 or one following the other, which is shown in Tables 16-18. Alternatively, an implementation with changed order can also be designed, i.e., 4x3 followed by 3x4.

[0158] In some embodiments, a simplification can be made to reduce the kernel size from 5x5 to 4x4. And the 4x4 up-sampling convolution can be implemented with a 2D convolution and a pixel shuffle layer instead of a 2D transpose convolution. For example, in PyTorch, it is implemented with torch.nn.Conv2d() followed by torch.nn.PixelShuffle(). For example, the second module for the variance decoder utilizing hyper-prior information includes a 4x4 convolution kernel implemented with a two-dimensional (2D) convolution layer plus a pixel shuffle layer. An example is shown in Table 19. Alternatively, the kernel size can be 3x3, 3x4, 4x3, or one following the other.

[0159] In some embodiments, the second module for the variance decoder utilizing hyper-prior information includes a 5x5 up-sampling layer followed by a convolutional layer followed by a 5x5 up-sampling layer. In some other embodiments, the number of convolutional layers in the second module for the variance decoder utilizing hyper-prior information is 6, 5, 4, or 3. In some further embodiments, the number of channels in the second module for the variance decoder utilizing hyper-prior information is adjustable.

[0160] In some embodiments, the bitstream includes a flag indicating whether a second module for a variance decoder with hyper-prior information is used in the conversion. In some other embodiments, a plurality of second modules for a variance decoder with hyper-prior information are stored, and wherein the bitstream includes a syntax element indicating which second module for a variance decoder with hyper-prior information is used in the conversion. In some further embodiments, the second module for a variance decoder with hyper-prior information is applied to at least one of: a luma component or a chroma component.

[0161] In some embodiments, the number of channels in the second module for a variance decoder with hyper-prior information can be changed. For example, the second module for a variance decoder with hyper-prior information includes a chroma variance decoder with hyper-prior information and a luma variance decoder with hyper-prior information, the chroma variance decoder with hyper-prior information has convolutional layers, where each convolutional layer has a number of channels that is a multiple of or twice the number of channels in the chroma variance decoder with hyper-prior information.

[0162] In some embodiments, the number of channels for each convolutional layer in the luma variance decoder with hyper-prior information is equal to C, and C, respectively; and the number of channels for each convolutional layer in the chroma variance decoder with hyper-prior information is equal to and For example, as shown in Figure 20 the number of channels for the chroma variance decoder with hyper-prior information can be a multiple of (16x).

[0163] In some embodiments, the number of channels for each convolutional layer in the luma variance decoder with hyper-prior information is equal to C, and C, respectively; and the number of channels for each convolutional layer in the chroma variance decoder with hyper-prior information is equal to and For example, as shown in Figure 21 the number of channels for the chroma variance decoder with hyper-prior information can be a multiple of (8x).

[0164] In some embodiments, the number of convolutional layers can also be reduced in addition to reducing the kernel size (Tables 16-19) or reducing the number of channels (Tables 20-21). For example, the number of convolutional layers can be changed from 6 to 5 or to 4.

[0165] In some embodiments, the number of channels for each convolutional layer in the variance decoder utilizing super-prior information for luma is equal to: C, 2C, and C, respectively; and the number of channels for each convolutional layer in the variance decoder utilizing super-prior information for chroma is equal to: C, and Examples are shown in Table 22. In some other embodiments, the number of channels for each convolutional layer in the variance decoder utilizing super-prior information for luma is equal to: C, and C, respectively; and the number of channels for each convolutional layer in the variance decoder utilizing super-prior information for chroma is equal to: and Examples are shown in Table 23.

[0166] In some embodiments, the number of channels for each convolutional layer in the variance decoder utilizing super-prior information for luma is equal to: C, 2C, 2C, and C, respectively; and the number of channels for each convolutional layer in the variance decoder utilizing super-prior information for chroma is equal to: C, C, and Examples are shown in Table 24.

[0167] In some embodiments, the number of channels for each convolutional layer in the variance decoder utilizing super-prior information for luma is equal to: C, C, C, and C, respectively; and the number of channels for each convolutional layer in the variance decoder utilizing super-prior information for chroma is equal to: and Examples are shown in Table 25 and Table 26.

[0168] In some embodiments, the second module for the variance decoder utilizing super-prior information includes 4 convolutional layers, where the kernel size is 2x2 and has pixel shuffle. Examples can be shown in Table 27 and Table 28.

[0169] In some embodiments, the second module for the variance decoder utilizing super-prior information includes 4 layers. Examples are shown in Table 29. In some embodiments, the first two layers of the 4 layers are 2x2 convolutional layers implemented with a pixel shuffler (e.g., as shown in Figure 34 Alternatively or additionally, the last two layers of the 4 layers are 3x3 group convolutional layers grouped as 2 (e.g., as shown in Figure 35 .

[0170] In some embodiments, the second module for a variance decoder that utilizes super-prior information includes 4 convolutional layers with kernel size 3x3 and with pixel shuffle. The number of channels in each layer after the pixel shuffle layer can be: C, C, C, and C, respectively. An example is shown in Figure 33 .

[0171] In some embodiments, the second module for a variance decoder that utilizes super-prior information includes 4 convolutional layers with kernel size 4x3 and with pixel shuffle. The number of channels in each layer after the pixel shuffle layer can be: C, , and C, respectively.

[0172] In some embodiments, the second module for a variance decoder that utilizes super-prior information includes 4 convolutional layers including an up-sampling layer with kernel size 5x5, followed by a convolutional layer, followed by an up-sampling layer with kernel size 5x5.

[0173] In some embodiments, the second module for a variance decoder that utilizes super-prior information includes 4 convolutional layers with kernel size 3x3 and with pixel shuffle. The number of channels in each layer after the pixel shuffle layer can be: 2C, , and 2C, respectively.

[0174] In some embodiments, the second module for a variance decoder that utilizes super-prior information includes 5 convolutional layers with kernel size 4x3. The number of channels in each layer after the pixel shuffle layer can be: C, 2C, , and C, respectively. An example is shown in Figure 25 .

[0175] In some embodiments, the second module for a variance decoder that utilizes super-prior information includes 5 convolutional layers with kernel size 4x3. The number of channels in each layer after the pixel shuffle layer can be: C, C, C, C, and C, respectively. An example is shown in Figure 26 .

[0176] In some embodiments, the second module for a variance decoder that utilizes super-prior information includes 4 convolutional layers with kernel size 4x3. The number of channels in each layer after the pixel shuffle layer can be: C, , and C, respectively. An example is shown in Figure 27 .

[0177] In some embodiments, the second module for a variance decoder that utilizes super-prior information includes 4 convolutional layers with kernel size 4x3. The number of channels in each layer after the pixel shuffle layer can be: 2C, , and C, respectively. An example is shown inFigure 28 is shown in

[0178] In some embodiments, the second module for the variance decoder that utilizes super-prior information includes 4 convolutional layers with kernel size 4x3. The number of channels in each layer after the pixel shuffle layer can be C, C, C, and C, respectively. An example is shown in Figure 29

[0179] In some embodiments, the second module for the variance decoder that utilizes super-prior information includes 5 convolutional layers with kernel size 3x3 and with a pixel shuffler. The number of channels in each layer after the pixel shuffle layer can be C, C, C, C, and C, respectively. An example is shown in Figure 30

[0180] In some embodiments, the second module for the variance decoder that utilizes super-prior information includes 5 convolutional layers with kernel size 3x3 and with a pixel shuffler. The number of channels in each layer after the pixel shuffle layer can be C, C, C, C, and C, respectively. An example is shown in Figure 31

[0181] In some embodiments, the second module for the variance decoder that utilizes super-prior information includes 4 convolutional layers with kernel size 3x3 and with a pixel shuffler. The number of channels in each layer after the pixel shuffle layer can be C, C, C, and C, respectively. An example is shown in Figure 32

[0182] In some embodiments, the second module for the variance decoder that utilizes super-prior information includes 3 convolutional layers with kernel size 3x3 and / or lxl and a pixel shuffling layer, and where the number of channels in each layer after the pixel shuffling layer are C, C, and 16C, respectively.

[0183] In some embodiments, the first module for the prediction fusion includes one of: a LeakyReLU activation layer or a ReLU activation layer. In some other embodiments, the second module for the variance decoder that utilizes super-prior information includes one of: a quantized LeakyReLU activation layer or a quantized ReLU activation layer.

[0184] ​​​​​​It should be noted that the embodiments of the present disclosure may be implemented individually or in any suitable manner. For example, Tables 3 to 15 may be combined in any suitable manner. For example, the number of one or more channels shown in Tables 3-7 may be combined in any suitable manner, and the number of one or more reduced convolutional layers in Tables 8-15 may be combined in any suitable manner. In addition, the number of one or more channels from Tables 3-7 may be combined with the convolutional layers shown in Tables 8-15. For example, Figure 36A As shown, the first module for prediction may include three 1×1 convolutions with a stride of 1. The nonlinear activation may be ReLU. The number of tensor channels in the hidden layer may be gradually reduced from 3C to C / 2: 3C→9C / 4→7C / 4→C / 2. For inputs from different groups, prediction fusion can use the same network structure to process the data, while the weights of the network are separate for different inputs. For example, the number of channels is equal to one of the following: 3C, or And for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or In some other example embodiments, Tables 16 to 26 may be combined in any suitable manner. For example, one or more kernel sizes / implementations shown in Tables 16-19, one or more numbers of channels shown in Tables 20-21, and the number of layers shown in Tables 22-26 may be combined in any suitable manner. Alternatively or additionally, one or more kernel sizes / implementations in Tables 16-19 may be combined in any suitable manner, one or more numbers of channels shown in Tables 20-21 may be combined in any manner, and the number of layers shown in Tables 22-26 may be combined in any manner. For example, the second module of the variance decoder for utilizing super-prior information may include 3 convolutional layers, where one of the convolutional layers has a kernel size of 3×3, and another of the convolutional layers has a kernel size of 1×1, and pixel shuffling. For example, as Figure 36B As shown, the second module of the variance decoder for utilizing super-prior information can start with a convolutional layer with a kernel size of 1×1 and stride 1, followed by a rectified linear unit. This is followed by a stride 1 convolutional layer with a kernel size of 3×3, followed by a rectified linear unit. The next stride 1 convolutional layer (kernel size 1×1) increases the number of channels to 16C. This is followed by pixel shuffling (stride 4), which brings the number of channels back to C. A pruning layer (stride 4, depth 5) ensures that the output tensor has the shape [C, h_4, w_4].

[0185] In some embodiments, an indication of whether and / or how the first module for prediction fusion and the second module for variance decoder with hyper-prior information are applied to the video data is indicated at one of: sequence level, picture group level, picture level, slice level, or tile group level. In some other embodiments, an indication of whether and / or how the first module for prediction fusion and the second module for variance decoder with hyper-prior information are applied to the video data is indicated in one of: sequence header, picture header, sequence parameter set (SPS), video parameter set (VPS), dependent parameter set (DPS), decoding capability information (DCI), picture parameter set (PPS), adaptation parameter set (APS), slice header, or tile group header. In some further embodiments, an indication of whether and / or how the first module for prediction fusion and the second module for variance decoder with hyper-prior information are applied to the video data is in one of: prediction block (PB), transform block (TB), coding block (CB), prediction unit (PU), transform unit (TU), coding unit (CU), coding tree block (CTB), coding tree unit (CTU).

[0186] In some embodiments, the method 3600 further includes determining, based on coded information of the video data, whether and / or how the first module for prediction fusion and the second module for variance decoder with hyper-prior information are applied to the video data, the coded information including at least one of: block size, color format, single tree partitioning and / or dual tree partitioning, color component, slice type, or picture type.

[0187] In some embodiments, the video data is applied with other coding tools that require chroma fusion.

[0188] In some embodiments, a syntax element (SE) is binarized into one of: a flag, a fixed length code, an EG(x) code, a unary code, a truncated unary code, or a truncated binary code. In some embodiments, the SE is signed or unsigned.

[0189] In some embodiments, the SE is coded with at least one context model. In some other embodiments, the SE is bypass coded.

[0190] In some embodiments, the SE is signaled in a conditional manner. For example, the SE is signaled only when a corresponding function is applicable. As another example, the SE is signaled only when dimensions including width and / or height of the video data satisfy a condition.

[0191] In some embodiments, the SE is indicated at one of a sequence level, a group of pictures level, a picture level, a slice level, or a tile group level. In some other embodiments, the SE is indicated at one of a prediction block (PB), a transform block (TB), a coding block (CB), a prediction unit (PU), a transform unit (TU), a coding unit (CU), a coding tree block (CTB), or a coding tree unit (CTU).

[0192] In view of the above, the solution according to some embodiments of the disclosure can advantageously improve coding efficiency and coding quality.

[0193] According to further embodiments of the disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream generated by a method for visual data processing, the method including: applying a neural network (NN)-based model including a first module for prediction fusion and a second module for a variance decoder utilizing super-prior information to video data, wherein at least one of the following is satisfied for the first module for prediction fusion and / or the second module for the variance decoder utilizing super-prior information: a number of channels in a convolution layer is less than or equal to a first threshold number, a total number of convolution layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size; and generating the bitstream based on the first module for prediction fusion and the second module for the variance decoder utilizing super-prior information.

[0194] According to still further embodiments of the disclosure, a method for storing a bitstream of a video is provided. The method includes: applying a neural network (NN)-based model including a first module for prediction fusion and a second module for a variance decoder utilizing super-prior information to video data, wherein at least one of the following is satisfied for the first module for prediction fusion and / or the second module for the variance decoder utilizing super-prior information: a number of channels in a convolution layer is less than or equal to a first threshold number, a total number of convolution layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size; generating the bitstream based on the first module for prediction fusion and the second module for the variance decoder utilizing super-prior information; and storing the bitstream in a non-transitory computer-readable recording medium.

[0195] Implementations of the disclosure can be described according to the following clauses, which features can be combined in any reasonable manner.

[0196] Item 1. A method for video processing, comprising: applying, for a conversion between visual data and a bitstream of the visual data, a neural network (NN) based model comprising a first module for prediction fusion and a second module for variance decoder with super-prior information to video data, wherein at least one of the following is satisfied for the first module for prediction fusion and / or the second module for variance decoder with super-prior information: a number of channels in a convolution layer is less than or equal to a first threshold number, a number of convolution layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size; and performing the conversion based on the first module for prediction fusion and the second module for variance decoder with super-prior information.

[0197] Item 2. The method of item 1, wherein a number of convolution layers in the first module for prediction fusion is one of: 6, 5, 4, or 3.

[0198] Item 3. The method of any of items 1-2, wherein a number of channels is adjustable for each layer in the first module for prediction fusion.

[0199] Item 4. The method of any of items 1-3, wherein the bitstream includes a syntax element indicating the number of channels for each layer in the first module for prediction fusion.

[0200] Item 5. The method of any of items 1-4, wherein a plurality of first modules for prediction fusion are stored, and wherein the bitstream includes a syntax element indicating which first module for prediction fusion is used in the conversion.

[0201] Item 6. The method of any of items 1-5, wherein the first module for prediction fusion is applied to one or more of: a luma component or a chroma component.

[0202] Item 7. The method of any of items 1-6, wherein the first module for prediction fusion includes a chroma prediction fusion network having convolution layers, wherein each convolution layer has a number of channels that is a multiple of or and a luma prediction fusion network having convolution layers, wherein each convolution layer has a number of channels that is twice the number of channels in the chroma prediction fusion network, wherein C is an integer.

[0203] Item 8. The method of item 7, wherein for each convolution layer in the luma prediction fusion network, the number of channels is equal to one of: or C; and for each convolution layer in the chroma prediction fusion network, the number of channels is equal to one of:

[0204] Item 9. The method of item 7, wherein the number of channels is equal to one of: or C for each convolutional layer in the luma prediction fusion network, and the number of channels is equal to one of: or

[0205] Item 10. The method of item 7, wherein the number of channels in the first convolutional layer in the luma prediction fusion network is and the number of channels in the first convolutional layer in the chroma prediction fusion network is

[0206] Item 11. The method of item 7, wherein the number of channels is equal to one of: or C for each convolutional layer in the luma prediction fusion network, and the number of channels is equal to one of: or

[0207] Item 12. The method of item 7, wherein the number of channels is equal to one of: or C for each convolutional layer in the luma prediction fusion network, and the number of channels is equal to one of: or

[0208] Item 13. The method of any one of items 1-12, wherein the number of convolutional layers in the luma prediction fusion network is 5, and the number of convolutional layers in the chroma prediction fusion network is 5.

[0209] Item 14. The method of item 13, wherein the number of channels is equal to one of: 3C, 2C, or C for each convolutional layer in the luma prediction fusion network, and the number of channels is equal to one of: C, or

[0210] Item 15. The method of item 13, wherein the number of channels is equal to one of: or C for each convolutional layer in the luma prediction fusion network, and the number of channels is equal to one of: or​

[0211] Item 16. The method of item 13, wherein the number of channels is equal to one of the following for each convolutional layer in the luma prediction fusion network: or C, and the number of channels is equal to one of the following for each convolutional layer in the chroma prediction fusion network: or

[0212] Item 17. The method of item 13, wherein the number of channels is equal to one of the following for each convolutional layer in the luma prediction fusion network: 2C, or C, and the number of channels is equal to one of the following for each convolutional layer in the chroma prediction fusion network: C, or

[0213] Item 18. The method of any one of items 1-12, wherein the number of convolutional layers in the luma prediction fusion network is 4, and the number of convolutional layers in the chroma prediction fusion network is 4.

[0214] Item 19. The method of item 18, wherein the number of channels is equal to one of the following for each convolutional layer in the luma prediction fusion network: 4C, 3C, 2C, or C, and the number of channels is equal to one of the following for each convolutional layer in the chroma prediction fusion network: 2C, C, or

[0215] Item 20. The method of item 18, wherein the number of channels is equal to one of the following for each convolutional layer in the luma prediction fusion network: or C, and the number of channels is equal to one of the following for each convolutional layer in the chroma prediction fusion network: or

[0216] Item 21. The method of item 18, wherein the number of channels is equal to one of the following for each convolutional layer in the luma prediction fusion network: or C, and the number of channels is equal to one of the following for each convolutional layer in the chroma prediction fusion network: or

[0217] Item 22. The method of item 18, wherein the number of channels is equal to one of the following for each convolutional layer in the luma prediction fusion network: or C, and for each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of: or

[0218] Clause 23. The method of any of clauses 7-22, wherein C is equal to 128.

[0219] Clause 24. The method of any of clauses 1-23, wherein the second module for the hyper- prior informed variance decoder comprises a 4x3 convolutional kernel, or a 3x4 convolutional kernel, or a combination thereof.

[0220] Clause 25. The method of any of clauses 1-24, wherein the second module for the hyper- prior informed variance decoder comprises a 4x4 convolutional kernel implemented with a two- dimensional (2D) convolutional layer plus a pixel shuffle layer.

[0221] Clause 26. The method of any of clauses 1-25, wherein the second module for the hyper- prior informed variance decoder comprises a 5x5 up-sampling layer followed by a convolutional layer followed by a 5x5 up-sampling layer.

[0222] Clause 27. The method of any of clauses 1-26, wherein the number of convolutional layers in the second module for the hyper-prior informed variance decoder is 6, 5, 4, or 3.

[0223] Clause 28. The method of any of clauses 1-27, wherein the number of channels in the second module for the hyper-prior informed variance decoder is adjustable.

[0224] Clause 29. The method of any of clauses 1-28, wherein the bitstream includes a flag indicating whether the second module for the hyper-prior informed variance decoder is used in the conversion.

[0225] Clause 30. The method of any of clauses 1-29, wherein a plurality of second modules for the hyper-prior informed variance decoder are stored, and wherein the bitstream includes a syntax element indicating which second module for the hyper-prior informed variance decoder is used in the conversion.

[0226] Clause 31. The method of any of clauses 1-30, wherein the second module for the hyper- prior informed variance decoder is applied to at least one of: a luma component or a chroma component.

[0227] Item 32. The method according to any one of Items 1 to 31, wherein the second module for the variance decoder utilizing super-prior information comprises a chroma variance decoder utilizing super-prior information and a luma variance decoder utilizing super-prior information, wherein the chroma variance decoder utilizing super-prior information has a convolutional layer, wherein the number of channels of each convolutional layer is or The luminance variance decoder using super-prior information has a convolutional layer, wherein the number of channels of each convolutional layer is twice the number of channels in the chrominance variance decoder using super-prior information.

[0228] Item 33. The method of Item 32, wherein the number of channels of each convolutional layer in the variance decoder utilizing super-prior information for luminance is equal to: C, and C; and the number of channels of each convolutional layer in the variance decoder using super prior information for chrominance is equal to: and

[0229] Item 34. The method of Item 32, wherein the number of channels of each convolutional layer in the variance decoder utilizing super-prior information for luminance is equal to: C, and C; and the number of channels of each convolutional layer in the variance decoder using super prior information for chrominance is equal to: and

[0230] Item 35. The method of Item 32, wherein the number of channels of each convolutional layer in the variance decoder utilizing super-prior information for luminance is equal to: C, 2C, and C; and the number of channels of each convolutional layer in the variance decoder using super prior information for chrominance is equal to: C. and

[0231] Item 36. The method of Item 32, wherein the number of channels of each convolutional layer in the variance decoder utilizing super-prior information for luminance is equal to: C, and C; and the number of channels of each convolutional layer in the variance decoder using super prior information for chrominance is equal to: and

[0232] Item 37. The method of item 32, wherein the number of channels for each convolutional layer in the variance decoder with hyper-prior information for luma are equal to: C, 2C, 2C, and C, respectively; and the number of channels for each convolutional layer in the variance decoder with hyper-prior information for chroma are equal to: C, C, and C, respectively. C, C, and

[0233] Item 38. The method of item 32, wherein the number of channels for each convolutional layer in the variance decoder with hyper-prior information for luma are equal to: C, C, C, and C, respectively; and the number of channels for each convolutional layer in the variance decoder with hyper-prior information for chroma are equal to: C, C, and C, respectively. and

[0234] Item 39. The method of any one of items 1-38, wherein the second module for the variance decoder with hyper-prior information comprises 4 convolutional layers, wherein the kernel size is 2x2 and has a pixel shuffle.

[0235] Item 40. The method of any one of items 1-38, wherein the second module for the variance decoder with hyper-prior information comprises 4 convolutional layers, wherein the first two of the 4 convolutional layers are convolutional layers with a kernel size of 2x2 and implemented with a pixel shuffler, and / or the last two of the 4 convolutional layers are group convolutional layers with a group size of 2 and a kernel size of 3x3.

[0236] Item 41. The method of any one of items 1-31, wherein the second module for the variance decoder with hyper-prior information comprises 4 convolutional layers, wherein the kernel size is 3x3 and has a pixel shuffler, and wherein the number of channels after the pixel shuffle layer are: C, C, C, and C, respectively.

[0237] Item 42. The method of any one of items 1-31, wherein the second module for the variance decoder with hyper-prior information comprises 4 convolutional layers, wherein the kernel size is 4x3 and has a pixel shuffle, and wherein the number of channels in each layer after the pixel shuffle layer are: C, and C, respectively.

[0238] Item 43. The method of any one of items 1-31, wherein the second module for the variance decoder with hyper-prior information comprises 4 convolutional layers, the 4 convolutional layers comprising an up-sampling layer with a kernel size of 5x5, followed by a convolutional layer, followed by an up-sampling layer with a kernel size of 5x5.

[0239] Item 44. The method of any one of items 1-31, wherein the second module for the variance decoder that utilizes super-prior information comprises 4 convolutional layers with kernel size of 3x3 and with pixel shuffle, and wherein the number of channels in each layer after the pixel shuffle layer are: 2C, and 2C.

[0240] Item 45. The method of any one of items 1-31, wherein the second module for the variance decoder that utilizes super-prior information comprises 5 convolutional layers with kernel size of 4x3, and wherein the number of channels in each layer are: C, 2C, and C.

[0241] Item 46. The method of any one of items 1-31, wherein the second module for the variance decoder that utilizes super-prior information comprises 5 convolutional layers with kernel size of 4x3, and wherein the number of channels in each layer are: C, C, C, C, and C.

[0242] Item 47. The method of any one of items 1-31, wherein the second module for the variance decoder that utilizes super-prior information comprises 4 convolutional layers with kernel size of 4x3, and wherein the number of channels in each layer are: C, and C.

[0243] Item 48. The method of any one of items 1-31, wherein the second module for the variance decoder that utilizes super-prior information comprises 4 convolutional layers with kernel size of 4x3, and wherein the number of channels in each layer are: 2C, and C.

[0244] Item 49. The method of any one of items 1-31, wherein the second module for the variance decoder that utilizes super-prior information comprises 4 convolutional layers with kernel size of 4x3, and wherein the number of channels in each layer after the pixel shuffle layer are: C, C, C, and C.

[0245] Item 50. The method of any one of items 1-31, wherein the second module for the variance decoder that utilizes super-prior information comprises 5 convolutional layers with kernel size of 3x3 and with a pixel shuffler, and wherein the number of channels in each layer after the pixel shuffle layer are: C, and C.

[0246] Item 51. The method of any of items 1-31, wherein the second module for the variance decoder exploiting super-prior information comprises 5 convolutional layers with kernel size 3x3 and having a pixel shuffler, and wherein the number of channels in each layer after the pixel shuffling layer are C, C, C, C, and C, respectively.

[0247] Item 52. The method of any of items 1-31, wherein the second module for the variance decoder exploiting super-prior information comprises 4 convolutional layers with kernel size 3x3 and having a pixel shuffler, and wherein the number of channels in each layer after the pixel shuffling layer are C, and C, respectively.

[0248] Item 53. The method of any of items 1-31, wherein the second module for the variance decoder exploiting super-prior information comprises 3 convolutional layers with kernel size 3x3 and / or 1x1 and a pixel shuffling layer, and wherein the number of channels in each layer after the pixel shuffling layer are C, C, and 16C, respectively.

[0249] Item 54. The method of any of items 1-53, wherein the first module for prediction fusion comprises one of: a LeakyReLU activation layer or a ReLU activation layer, and / or wherein the second module for the variance decoder exploiting super-prior information comprises one of: a quantized LeakyReLU activation layer or a quantized ReLU activation layer.

[0250] Item 55. The method of any of items 1-54, wherein an indication of whether and / or how to apply the first module for prediction fusion and the second module for the variance decoder exploiting super-prior information to video data is indicated at one of: a sequence level, a group of pictures level, a picture level, a slice level, or a tile group level.

[0251] Item 56. The method of any of items 1-54, wherein an indication of whether and / or how to apply the first module for prediction fusion and the second module for the variance decoder exploiting super-prior information to video data is indicated in one of: a sequence header, a picture header, a sequence parameter set (SPS), a video parameter set (VPS), a dependent parameter set (DPS), a decoding capability information (DCI), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, or a tile group header.

[0252] Item 57. The method according to any one of items 1-54, wherein an indication of whether and / or how to apply the first module for prediction fusion and the second module for variance decoder with super-prior information to the video data is in one of: a prediction block (PB), a transform block (TB), a coding block (CB), a prediction unit (PU), a transform unit (TU), a coding unit (CU), a coding tree block (CTB), a coding tree unit (CTU).

[0253] Item 58. The method according to any one of items 1-57, further comprising determining, based on coded information of the video data, whether and / or how to apply the first module for prediction fusion and the second module for variance decoder with super-prior information to the video data, the coded information comprising at least one of: block size, color format, single tree partitioning and / or dual tree partitioning, color component, slice type or picture type.

[0254] Item 59. The method according to any one of items 1-58, wherein the video data is applied together with other coding tools requiring chroma fusion.

[0255] Item 60. The method according to any one of items 1-59, wherein a syntax element (SE) is binarized into at least one of: a flag, a fixed length code, an EG(x) code, a unary code, a truncated unary code or a truncated binary code.

[0256] Item 61. The method according to item 60, wherein the SE is signed or unsigned.

[0257] Item 62. The method according to any one of items 1-61, wherein the SE is coded with at least one context model, or wherein the SE is bypass coded.

[0258] Item 63. The method according to any one of items 1-61, wherein the SE is signaled in a conditional manner.

[0259] Item 64. The method according to item 63, wherein the SE is signaled only when a corresponding functionality is applicable, or wherein the SE is signaled only when a dimension comprising a width and / or height of the video data satisfies a condition.

[0260] Item 65. The method according to any one of items 1-62, wherein the SE is indicated at one of: a sequence level, a group of pictures level, a picture level, a slice level or a tile group level.

[0261] Item 66. The method of any of items 1-64, wherein the SE is indicated at one of a prediction block (PB), a transform block (TB), a coding block (CB), a prediction unit (PU), a transform unit (TU), a coding unit (CU), a coding tree block (CTB), or a coding tree unit (CTU).

[0262] Item 67. The method of any of items 1-66, wherein the converting comprises encoding the visual data into the bitstream.

[0263] Item 68. The method of any of items 1-66, wherein the converting comprises decoding the visual data from the bitstream.

[0264] Item 69. An apparatus for visual data processing comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any of items 1-68.

[0265] Item 70. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method of any of items 1-68.

[0266] Item 71. A non-transitory computer-readable recording medium storing a bitstream of visual data generated by a method performed by an apparatus for visual data processing, wherein the method comprises: applying a neural network (NN)-based model comprising a first module for prediction fusion and a second module for a variance decoder with hyper-prior information to the video data, wherein at least one of the following is satisfied for the first module for prediction fusion and / or the second module for the variance decoder with hyper-prior information: a number of channels in a convolution layer is less than or equal to a first threshold number, a number of convolution layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size; and generating the bitstream based on the first module for prediction fusion and the second module for the variance decoder with hyper-prior information.

[0267] Item 72. A method for storing a bitstream of visual data, comprising: applying a neural network (NN) based model including a first module for predictive fusion and a second module for a variance decoder with super-prior information to video data, wherein at least one of the following is satisfied for the first module for predictive fusion and / or the second module for the variance decoder with super-prior information: a number of channels in a convolution layer is less than or equal to a first threshold number, a number of convolution layers is less than or equal to a second threshold number, or a kernel size is less than or equal to a threshold size; generating the bitstream based on the first module for predictive fusion and the second module for the variance decoder with super-prior information; and storing the bitstream in a non-transitory computer-readable recording medium. Example device

[0268] Figure 37 A block diagram of a computing device 3700 in which various embodiments of the present disclosure can be implemented is shown. The computing device 3700 can be implemented as or included in the source device 110 (or the visual data encoder 114) or the destination device 120 (or the visual data decoder 124).

[0269] It should be understood that, Figure 37 The computing device 3700 shown in FIG. 13 is for purposes of illustration and explanation only and is not intended to imply any limitation on the functionality and scope of embodiments of the present disclosure.

[0270] As Figure 37 shown, the computing device 3700 includes a general-purpose computing device 3700. The computing device 3700 can include at least one or more processors or processing units 3710, a memory 3720, a storage unit 3730, one or more communication units 3740, one or more input devices 3750, and one or more output devices 3760.

[0271] In some embodiments, the computing device 3700 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, and including accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 3700 can support any type of interface to the user (such as a "wearable" circuit device, etc.).

[0272] The processing unit 3710 may be a physical processor or a virtual processor and may implement various processes based on a program stored in the memory 3720. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of the computing device 3700. The processing unit 3710 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0273] The computing device 3700 typically includes various computer storage media. Such media can be any media accessible by the computing device 3700, including but not limited to volatile media and non-volatile media, or removable media and non-removable media. The memory 3720 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM) or flash memory) or any combination thereof. The storage unit 3730 can be any removable or non-removable medium and can include machine-readable media, such as memory, a flash drive, a disk or other media that can be used to store information and / or visual data and can be accessed in the computing device 3700.

[0274] The computing device 3700 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Figure 37 Although not shown, a magnetic disk drive for reading from and / or writing to a removable non-volatile magnetic disk, and an optical disk drive for reading from and / or writing to a removable non-volatile optical disk may be provided. In this case, each drive may be connected to a bus (not shown) via one or more visual data medium interfaces.

[0275] The communication unit 3740 communicates with another computing device via a communication medium. Additionally, the functions of the components of the computing device 3700 can be implemented by a single computing cluster or multiple computing machines in communication with one another via a communication connection. Thus, the computing device 3700 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other common network nodes. The communication unit 3740 can include one or more wired and / or wireless receivers and transmitters.

[0276] The input device 3750 can be one or more of a variety of input devices, such as a mouse, a keyboard, a trackball, a voice input device, or the like. The output device 3760 can be one or more of a variety of output devices, such as a display, a speaker, a printer, or the like. By means of the communication unit 3740, the computing device 3700 can also communicate with one or more external devices (not shown), such as a storage device and a display device, and one or more devices that enable a user to interact with the computing device 3700, or any devices (e.g., a network card, a modem, etc.) that enable the computing device 3700 to communicate with one or more other computing devices, if desired. Such communication can be via an input / output (I / O) interface (not shown).

[0277] In some embodiments, some or all of the components of the computing device 3700 can not be integrated in a single device, but can also be arranged in a cloud computing architecture. In a cloud computing architecture, the components can be provided remotely and work together to implement the functionality described in this disclosure. In some embodiments, cloud computing provides computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system or of the hardware and / or software resources being accessed. In various embodiments, cloud computing uses suitable protocols and / or interfaces to provide services via the wide area network, such as the Internet. For example, a cloud computing provider provides applications through the wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and corresponding visual data can be stored on servers at remote locations. The computing resources in a cloud computing environment can be consolidated or distributed at locations in remote visual data centers. Cloud computing infrastructure can provide services through shared visual data centers, although they appear as a single point of access for users. Thus, a cloud computing architecture can be used to provide the components and functionality described herein from a service provider at a remote location. Alternatively, they can be provided from a conventional server, or installed directly or otherwise on a client device.

[0278] The computing device 3700 can be used to implement visual data encoding / decoding in embodiments of the present disclosure. The memory 3720 can include one or more visual data coding modules 3725 having one or more program instructions. These modules can be accessible and executable by the processing unit 3710 to perform the functions of the various embodiments described herein.

[0279] In example embodiments performing visual data encoding, the input device 3750 can receive visual data as input 3770 to be encoded. The visual data can be processed by, for example, the visual data coding module 3725 to generate an encoded bitstream. The encoded bitstream can be provided as output 3780 via the output device 3760.

[0280] In example embodiments performing visual data decoding, the input device 3750 can receive an encoded bitstream as input 3770. The encoded bitstream can be processed by, for example, the visual data coding module 3725 to generate decoded visual data. The decoded visual data can be provided as output 3780 via the output device 3760.

[0281] While the present disclosure has been particularly shown and described with reference to the preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application as defined by the appended claims. Such changes are intended to fall within the scope of the present application. Accordingly, the foregoing description of embodiments of the application is not intended to be limiting.

Claims

1. A method for video processing, comprising: For conversion between visual data and a bitstream of the visual data, a neural network (NN)-based model including a first module for predictive fusion and a second module for a variance decoder utilizing super-prior information is applied to the video data, wherein at least one of the following conditions is satisfied for the first module for predictive fusion and / or the second module for the variance decoder utilizing super-prior information: the number of channels in a convolutional layer is less than or equal to a first threshold number, the number of convolutional layers is less than or equal to a second threshold number, or the kernel size is less than or equal to a threshold size; as well as The conversion is performed based on the first module for prediction fusion and the second module for variance decoder utilizing super-prior information.

2. The method of claim 1, wherein the number of the convolutional layers in the first module for predicting fusion is one of the following: 5, 4 or 3.

3. The method according to any one of claims 1 to 2, wherein the number of the channels for each layer in the first module for prediction fusion is adjustable. 4 . The method according to claim 1 , wherein the bitstream comprises a syntax element indicating the number of channels for each layer in the first module used for prediction fusion.

5. The method according to any one of claims 1 to 4, wherein a plurality of first modules for predictive fusion are stored, and wherein the bitstream comprises a syntax element indicating which first module for predictive fusion is used in the conversion.

6. The method according to any one of claims 1 to 5, wherein the first module for predictive fusion is applied to one or more of the following: a luma component or a chroma component.

7. The method according to any one of claims 1 to 6, wherein the first module for prediction fusion comprises a chroma prediction fusion network and a luma prediction fusion network, wherein the chroma prediction fusion network has a convolutional layer, wherein the number of channels of each convolutional layer is or The luma prediction fusion network has a convolutional layer, wherein each convolutional layer has a number of channels that is twice the number of channels in the chroma prediction fusion network, where C is an integer.

8. The method of claim 7, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: or C; and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or 9. The method of claim 7, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or 10. The method according to claim 7, wherein the number of channels in the first convolutional layer in the brightness prediction fusion network is And the number of channels in the first convolutional layer in the chroma prediction fusion network is 11. The method of claim 7, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: or C; and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or 12. The method of claim 7, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or 13 . The method according to claim 1 , wherein the number of the convolutional layers in the luma prediction fusion network is 5, and the number of the convolutional layers in the chroma prediction fusion network is 5.

14. The method of claim 13, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: 3C, 2C, or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: C. or 15. The method of claim 13, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or 16. The method of claim 13, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or 17. The method of claim 13, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: 2C, or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: C. or 18. The method according to any one of claims 1 to 12, wherein the number of the convolutional layers in the luma prediction fusion network is 4, and the number of the convolutional layers in the chroma prediction fusion network is 4.

19. The method of claim 18, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: 4C, 3C, 2C, or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: 2C, C or 20. The method of claim 18, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or 21. The method of claim 18, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or 22. The method of claim 18, wherein for each convolutional layer in the brightness prediction fusion network, the number of channels is equal to one of the following: or C, and For each convolutional layer in the chroma prediction fusion network, the number of channels is equal to one of the following: or 23. The method according to any one of claims 7 to 22, wherein C is equal to 128.

24. The method according to any one of claims 1 to 23, wherein the second module for the variance decoder utilizing super-prior information comprises a 4x3 convolution kernel, or a 3x4 convolution kernel, or a combination thereof.

25. The method according to any one of claims 1 to 24, wherein the second module for the variance decoder utilizing super-prior information comprises a 4x4 convolution kernel implemented using a two-dimensional (2D) convolution layer plus a pixel shuffling layer.

26. The method according to any one of claims 1 to 25, wherein the second module for the variance decoder utilizing super-prior information comprises a 5x5 upsampling layer followed by a convolutional layer followed by a 5x5 upsampling layer.

27. The method according to any one of claims 1 to 26, wherein the number of the convolutional layers in the second module of the variance decoder for utilizing super-prior information is one of the following: 5, 4 or 3.

28. The method according to any one of claims 1 to 27, wherein the number of channels in the second module of the variance decoder for utilizing super-prior information is adjustable.

29. The method according to any one of claims 1 to 28, wherein the bitstream comprises a flag indicating whether the second module for a variance decoder utilizing super-a priori information is used in the conversion.

30. The method according to any one of claims 1 to 29, wherein a plurality of second modules for a variance decoder utilizing super a priori information are stored, and wherein the bitstream comprises a syntax element indicating which second module for a variance decoder utilizing super a priori information is used in the conversion.

31. The method according to any one of claims 1 to 30, wherein the second module for a variance decoder utilizing super-a priori information is applied to at least one of the following: a luma component or a chroma component.

32. The method according to any one of claims 1 to 31, wherein the second module for the variance decoder using super-prior information comprises a chroma variance decoder using super-prior information and a luminance variance decoder using super-prior information, wherein the chroma variance decoder using super-prior information has a convolutional layer, wherein the number of channels of each convolutional layer is or The luminance variance decoder using super-prior information has a convolutional layer, wherein the number of channels of each convolutional layer is twice the number of channels in the chrominance variance decoder using super-prior information.

33. The method according to claim 32, wherein the number of channels of each convolutional layer in the variance decoder using super-prior information for the brightness is respectively equal to: C, and C; and The number of channels of each convolutional layer in the variance decoder using super-prior information for the chrominance is respectively equal to: and 34. The method according to claim 32, wherein the number of channels of each convolutional layer in the variance decoder using super-prior information for the brightness is respectively equal to: C, and C; and The number of channels of each convolutional layer in the variance decoder using super-prior information for the chrominance is respectively equal to: and 35. The method according to claim 32, wherein the number of channels of each convolutional layer in the variance decoder using super-prior information for the brightness is respectively equal to: C, 2C, and C; and The number of channels of each convolutional layer in the variance decoder using super-prior information for the chrominance is respectively equal to: C. and 36. The method according to claim 32, wherein the number of channels of each convolutional layer in the variance decoder using super-prior information for the brightness is respectively equal to: C, and C; and The number of channels of each convolutional layer in the variance decoder using super-prior information for the chrominance is respectively equal to: and 37. The method of claim 32, wherein the number of channels of each convolutional layer in the variance decoder utilizing super-prior information for the luminance is equal to: C, 2C, 2C, and C, respectively; and The number of channels of each convolutional layer in the variance decoder using super-prior information for the chrominance is respectively equal to: C, C and 38. The method of claim 32, wherein the number of channels of each convolutional layer in the variance decoder utilizing super-prior information for the luminance is equal to: C, C, C, and C, respectively; and The number of channels of each convolutional layer in the variance decoder using super-prior information for the chrominance is respectively equal to: and 39. The method according to any one of claims 1 to 38, wherein the second module for the variance decoder utilizing super-prior information comprises 4 convolutional layers with a kernel size of 2x2 and pixel shuffling.

40. A method according to any one of claims 1 to 38, wherein the second module of the variance decoder for utilizing super-prior information includes 4 convolutional layers, wherein the first two convolutional layers of the 4 convolutional layers are convolutional layers with a kernel size of 2x2 and are implemented using a pixel shuffler, and / or the last two convolutional layers of the 4 convolutional layers are group convolutional layers with a group size of 2 and a kernel size of 3x3.

41. The method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information comprises 4 convolutional layers with a kernel size of 3x3 and a pixel shuffler, and wherein the number of channels after the pixel shuffle layer is: C1, C2, C3, and C4, respectively.

42. The method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information comprises 4 convolutional layers with a kernel size of 4x3 and pixel shuffling, and wherein the number of channels in each layer after the pixel shuffling layer is respectively: C, and C.

43. A method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information includes 4 convolutional layers, and the 4 convolutional layers include an upsampling layer with a kernel size of 5x5, followed by a convolutional layer, followed by an upsampling layer with a kernel size of 5x5.

44. The method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information comprises 4 convolutional layers with a kernel size of 3x3 and pixel shuffling, and wherein the number of channels in each layer after the pixel shuffling layer is respectively: 2C, and 2C.

45. The method according to any one of claims 1 to 31, wherein the second module for the variance decoder utilizing super-prior information comprises 5 convolutional layers with a kernel size of 4x3, and wherein the number of channels in each layer is respectively: C, 2C, and C.

46. ​​A method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information comprises 5 convolutional layers with a kernel size of 4x3, and wherein the number of channels in each layer is: C1, C2, C3, C4, and C5, respectively.

47. The method according to any one of claims 1 to 31, wherein the second module for the variance decoder utilizing super-prior information comprises 4 convolutional layers with a kernel size of 4x3, and wherein the number of channels in each layer is respectively: C, and C.

48. The method according to any one of claims 1 to 31, wherein the second module for the variance decoder utilizing super-prior information comprises 4 convolutional layers with a kernel size of 4x3, and wherein the number of channels in each layer is respectively: 2C, and C.

49. A method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information comprises 4 convolutional layers with a kernel size of 4x3, and wherein the number of channels in each layer after the pixel shuffling layer is: C, C, C and C, respectively.

50. The method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information comprises 5 convolutional layers with a kernel size of 3x3 and a pixel shuffler, and wherein the number of channels in each layer after the pixel shuffle layer is respectively: C, and C.

51. A method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information includes 5 convolutional layers with a kernel size of 3x3 and a pixel shuffler, and wherein the number of channels in each layer after the pixel shuffle layer is: C, C, C, C and C, respectively.

52. The method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information comprises 4 convolutional layers with a kernel size of 3x3 and a pixel shuffler, and wherein the number of channels in each layer after the pixel shuffle layer is respectively: C, and C.

53. A method according to any one of claims 1 to 31, wherein the second module of the variance decoder for utilizing super-prior information includes 3 convolutional layers with kernel sizes of 3x3 and / or 1x1 and a pixel shuffling layer, and wherein the number of channels in each layer after the pixel shuffling layer is: C, C and 16C, respectively.

54. A method according to any one of claims 1 to 52, wherein the first module for predicting fusion comprises one of the following: a LeakyReLU activation layer or a ReLU activation layer, and / or wherein the second module for a variance decoder utilizing super-prior information comprises one of the following: a quantized LeakyReLU activation layer or a quantized ReLU activation layer.

55. The method according to any one of claims 1 to 54, wherein the indication of whether and / or how to apply the first module for predictive fusion and the second module for variance decoder utilizing super-prior information to the video data is for being indicated at one of the following: Sequence level, Picture group level, Picture level, stripe level, or Film group level.

56. The method according to any one of claims 1 to 54, wherein the indication of whether and / or how to apply the first module for predictive fusion and the second module for variance decoder utilizing super-prior information to the video data is for being indicated in one of the following: Sequence header, Picture header, Sequence Parameter Set (SPS), Video Parameter Set (VPS), Dependency Parameter Set (DPS), Decoding Capability Information (DCI), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Strip header, or Film group header.

57. The method according to any one of claims 1 to 54, wherein the indication of whether and / or how to apply the first module for predictive fusion and the second module for variance decoder utilizing super-prior information to the video data is one of: Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction Unit (PU), Transformation Unit (TU), Codec Unit (CU), Codec Tree Block (CTB), Codec Tree Unit (CTU).

58. The method according to any one of claims 1 to 57, further comprising: Determining whether and / or how to apply the first module for predictive fusion and the second module for a variance decoder using super-prior information to the video data based on coded information of the video data, wherein the coded information includes at least one of the following: Block size, Color format, Single-tree partitioning and / or dual-tree partitioning, Color component, Strip type, or Image type.

59. The method according to any one of claims 1 to 58, wherein the video data is applied together with other codec tools requiring chroma fusion.

60. The method according to any one of claims 1 to 59, wherein a syntax element (SE) is binarized as one of the following: a flag, a fixed length code, an EG(x) code, a unary code, a truncated unary code or a truncated binary code.

61. The method of claim 60, wherein the SE is signed or unsigned.

62. The method according to any one of claims 1 to 61, wherein the SE is encoded or decoded using at least one context model, or The SE is bypassed for coding.

63. The method of any one of claims 1 to 61, wherein the SE is signaled in a conditional manner.

64. The method of claim 63, wherein the SE is signaled only when the corresponding function is applicable, or The SE is transmitted via a signal only when a dimension including a width and / or a height of the video data satisfies a condition.

65. The method of any one of claims 1 to 62, wherein the SE is indicated at one of: Sequence level, Picture group level, Picture level, stripe level, or Film group level.

66. The method of any one of claims 1 to 64, wherein the SE is indicated at one of: Prediction Block (PB), Transform Block (TB), Codec Block (CB), Prediction Unit (PU), Transformation Unit (TU), Codec Unit (CU), Codec Tree Block (CTB), or Codec Tree Unit (CTU).

67. A method according to any one of claims 1 to 66, wherein the converting comprises encoding the visual data into the bitstream.

68. A method according to any one of claims 1 to 66, wherein the converting comprises decoding the visual data from the bitstream.

69. An apparatus for visual data processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 68.

70. A non-transitory computer-readable storage medium storing instructions for causing a processor to execute the method according to any one of claims 1 to 68.

71. A non-transitory computer-readable recording medium storing a bitstream of visual data generated by a method performed by an apparatus for visual data processing, wherein the method comprises: Applying a neural network (NN) based model comprising a first module for predictive fusion and a second module for a variance decoder utilizing super-prior information to video data, wherein at least one of the following conditions is satisfied for the first module for predictive fusion and / or the second module for a variance decoder utilizing super-prior information: the number of channels in a convolutional layer is less than or equal to a first threshold number, the number of convolutional layers is less than or equal to a second threshold number, or the kernel size is less than or equal to a threshold size; as well as The bitstream is generated based on the first module for prediction fusion and the second module for variance decoder utilizing super-prior information.

72. A method for storing a bitstream of visual data, comprising: Applying a neural network (NN) based model comprising a first module for predictive fusion and a second module for a variance decoder utilizing super-prior information to video data, wherein at least one of the following conditions is satisfied for the first module for predictive fusion and / or the second module for a variance decoder utilizing super-prior information: the number of channels in a convolutional layer is less than or equal to a first threshold number, the number of convolutional layers is less than or equal to a second threshold number, or the kernel size is less than or equal to a threshold size; generating the bitstream based on the first module for prediction fusion and the second module for variance decoder utilizing super-prior information; as well as The bitstream is stored in a non-transitory computer-readable recording medium.

Images