Visual data processing method, device and medium

Abstract

Embodiments of the present disclosure provide a solution for visual data processing. A method for visual data processing is presented. The method comprises: for a transformation between visual data and a bitstream of visual data using a neural network (NN)-based model, obtaining at least one intermediate representation of visual data in the NN-based model; applying a first filter in the NN-based model to the at least one intermediate representation, at least one parameter of the first filter being configured based on a format from the converted output visual data, the format indicating a relationship between a size of a first component of the output visual data and a size of a second component of the output visual data; and performing the transition based on the application.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to visual data processing techniques, and more specifically, to visual data encoding and decoding based on neural networks. Background Technology

[0002] The past decade has witnessed the rapid development of deep learning across various fields, particularly in computer vision and image processing. Neural networks were initially invented through interdisciplinary research in neuroscience and mathematics. They have demonstrated powerful capabilities in the context of nonlinear transformations and classification. In the past five years, neural network-based image / video compression technologies have made significant progress. It has been reported that the latest neural network-based image compression algorithms have achieved rate-distortion (RD) performance comparable to that of Multifunctional Video Coding (VVC). With the continuous improvement of neural image compression performance, neural network-based video compression has become an actively developing research area. However, the encoding and decoding efficiency of neural network-based image / video codecs is generally expected to be further improved. Summary of the Invention

[0003] Embodiments of this disclosure provide a solution for visual data processing.

[0004] In a first aspect, a method for visual data processing is proposed. The method includes: performing a transformation between visual data and a bitstream of visual data using a neural network (NN)-based model to obtain at least one intermediate representation of the visual data in the NN-based model; applying a first filter from the NN-based model to the at least one intermediate representation, wherein at least one parameter of the first filter is configured based on the format of the output visual data from the transformation, the format indicating a relationship between the dimensions of a first component and a second component of the output visual data; and performing the transformation based on the application.

[0005] Based on the method according to the first aspect of this disclosure, the parameters(s) of a first filter(s) applied to at least one intermediate representation of visual data are configured based on the format of the output visual data. Compared to conventional solutions where the parameters of the first filter are fixed, the proposed solution can advantageously support different output formats to cater to different applications. In this way, encoding and decoding flexibility can be improved, thereby enhancing encoding and decoding efficiency.

[0006] In a second aspect, an apparatus for visual data processing is proposed. The apparatus includes a processor and a non-transitory memory having instructions thereon. When executed by the processor, the instructions cause the processor to perform the method according to the first aspect of this disclosure.

[0007] In a third aspect, a non-transitory computer-readable storage medium is proposed. This non-transitory computer-readable storage medium stores instructions that cause a processor to perform the method according to the first aspect of this disclosure.

[0008] In a fourth aspect, another non-transitory computer-readable recording medium is proposed. This non-transitory computer-readable recording medium stores a bitstream of visual data generated by a method performed by an apparatus for visual data processing. The method includes: acquiring at least one intermediate representation of the visual data in a neural network (NN)-based model; applying a first filter from the NN-based model to the at least one intermediate representation, wherein at least one parameter of the first filter is configured based on a format from the transformed output visual data, the format indicating a relationship between the dimensions of a first component and a second component of the output visual data; and generating a bitstream based on the application using the NN-based model.

[0009] In a fifth aspect, a method for storing a bitstream of visual data is proposed. The method includes: acquiring at least one intermediate representation of the visual data in a neural network (NN)-based model; applying a first filter from the NN-based model to the at least one intermediate representation, wherein at least one parameter of the first filter is configured based on a format from the transformed output visual data, the format indicating a relationship between the dimensions of a first component and a second component of the output visual data; generating a bitstream using the NN-based model based on an application; and storing the bitstream in a non-transitory computer-readable recording medium.

[0010] This summary aims to present, in a simplified form, the selected concepts further described below in the detailed embodiments. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0011] The above and other objects, features and advantages of exemplary embodiments of the present disclosure will become clearer from the following detailed description with reference to the accompanying drawings, in which the same reference numerals generally refer to the same parts.

[0012] Figure 1A A block diagram of an example visual data encoding and decoding system according to some embodiments of the present disclosure is shown; Figure 1B This is a schematic diagram illustrating an example transform encoding / decoding scheme; Figure 2 An example potential representation of the image is shown; Figure 3 This is a schematic diagram illustrating an example autoencoder that implements a hyperprior model; Figure 4 This is a schematic diagram illustrating an example combined model configured to jointly optimize the context model together with the super-prior and the autoencoder; Figure 5 An example encoding process is shown; Figure 6 An example decoding process is shown; Figure 7 Example decoding processes according to some embodiments of this disclosure are shown; Figure 8 An example of a learning-based image codec architecture is shown; Figure 9 An example synthetic transform for learning-based image encoding and decoding is shown; Figure 10 An example LeakyReLU activation function is shown; Figure 11 An example ReLU activation function is shown; Figure 12 This is a flowchart of an example method for visual data processing according to embodiments of the present disclosure; Figure 13 This is a flowchart of an example method for visual data processing according to embodiments of the present disclosure; Figure 14 This is a flowchart of an example method for visual data processing according to embodiments of the present disclosure; Figure 15 This is a flowchart of an example method for visual data processing according to embodiments of the present disclosure; Figure 16 An example neural network according to an embodiment of the present disclosure is shown; Figure 17 An example neural network according to an embodiment of the present disclosure is shown; Figure 18 A block diagram illustrating a downsampling process according to an embodiment of the present disclosure is shown; Figure 19 A flowchart of a method for visual data processing according to embodiments of the present disclosure is shown; and Figure 20 A block diagram of a computing device in which various embodiments of the present disclosure may be implemented is shown.

[0013] In all accompanying drawings, the same or similar reference numerals usually refer to the same or similar elements. Detailed Implementation

[0014] The principles of this disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art understand and implement this disclosure, and do not imply any limitation on the scope of this disclosure. In addition to the methods described below, the disclosure described herein can be implemented in various other ways.

[0015] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0016] The terms "an embodiment," "embodiment," "example embodiment," etc., used in this disclosure refer to embodiments that may include specific features, structures, or characteristics, but not every embodiment is required to include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in conjunction with an example embodiment, whether explicitly described or not, it is believed that such a feature, structure, or characteristic affecting its relation to other embodiments is within the knowledge of those skilled in the art.

[0017] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “containing,” and / or “comprising” as used herein indicate the presence of the said features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0019] Example Environment Figure 1AThis is a block diagram illustrating an example visual data encoding / decoding system 100 from which the techniques of this disclosure can be utilized. As shown, the visual data encoding / decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a visual data encoding device, and the destination device 120 may also be referred to as a visual data decoding device. In operation, the source device 110 may be configured to generate encoded visual data, and the destination device 120 may be configured to decode the encoded visual data generated by the source device 110. The source device 110 may include a visual data source 112, a visual data encoder 114, and an input / output (I / O) interface 116.

[0020] Visual data source 112 may include sources such as visual data acquisition devices. Examples of visual data acquisition devices include, but are not limited to, interfaces for receiving visual data from visual data providers, computer graphics systems for generating visual data, and / or combinations thereof.

[0021] Visual data may include one or more pictures or images from a video. A visual data encoder 114 encodes the visual data from a visual data source 112 to generate a bitstream. The bitstream may include a sequence of bits forming an encoded / decoded representation of the visual data. The bitstream may include encoded / decoded pictures and associated visual data. The encoded / decoded pictures are encoded / decoded representations of pictures. The associated visual data may include sequence parameter sets, picture parameter sets, and other syntax structures. An I / O interface 116 may include a modulator / demodulator and / or a transmitter. The encoded visual data may be transmitted directly to a destination device 120 via network 130A through I / O interface 116. The encoded visual data may also be stored on a storage medium / server 130B for access by the destination device 120.

[0022] The destination device 120 may include an I / O interface 126, a visual data decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may acquire encoded visual data from the source device 110 or the storage medium / server 130B. The visual data decoder 124 may decode the encoded visual data. The display device 122 may display the decoded visual data to a user. The display device 122 may be integrated with the destination device 120, or it may be external to the destination device 120, which is configured to interface with an external display device.

[0023] The visual data encoder 114 and the visual data decoder 124 can operate according to visual data encoding and decoding standards, such as video encoding and decoding standards or still image encoding and decoding standards and other existing and / or future standards.

[0024] Some exemplary embodiments of this disclosure will be described in detail below. It should be understood that section headings are used in this document for ease of understanding and not to limit the embodiments disclosed in a section to that section only. Furthermore, although specific embodiments are described with reference to multi-function video codecs or other specific visual data codecs, the disclosed techniques are also applicable to other codec techniques. Furthermore, although some embodiments describe encoding and decoding steps in detail, it will be understood that the corresponding unencoding decoding steps will be implemented by the decoder. Additionally, the term visual data processing encompasses visual data encoding or compression, visual data decoding or decompression, and visual data transcoding, in which visual data is represented from one compressed format to another or at different compression bit rates.

[0025] 1. Preliminary Discussion This patent document relates to a neural network-based image and video compression method that uses an adaptive filtering layer to modify the components of an image. This includes determining whether the values ​​of samples of a first component are based on the values ​​of samples of a second component.

[0026] 2. Further discussion Deep learning is advancing across various fields, such as computer vision and image processing. Inspired by the successful applications of deep learning in computer vision, neural image / video compression technology is being researched for use in image / video compression. Neural networks are designed based on interdisciplinary research in neuroscience and mathematics. Neural networks demonstrate powerful capabilities in the context of nonlinear transformations and classification. Example neural network-based image compression algorithms achieve RD performance comparable to Multifunctional Video Coding (VVC), a video codec standard developed by the Joint Video Experts Group (JVET) in collaboration with experts from the Moving Picture Experts Group (MPEG) and the Video Codec Experts Group (VCEG). Neural network-based video compression is a rapidly developing research area, leading to continuous improvements in the performance of neural image compression. However, due to the inherent difficulty of the problems addressed by neural networks, neural network-based video coding and decoding remains a largely unexplored discipline.

[0027] 2.1 Image / Video Compression Image / video compression generally refers to the computational technique of compressing video images into binary code for easier storage and transmission. The binary code may or may not support lossless reconstruction of the original image / video. Encoding and decoding without data loss is called lossless compression, while encoding and decoding that allows for targeted data loss is called lossy compression. Most encoding and decoding systems use lossy compression because lossless reconstruction is not always necessary. The performance of image / video compression algorithms is typically evaluated based on the resulting compression ratio and reconstruction quality. The compression ratio is directly related to the number of binary codes produced by compression; fewer binary codes result in better compression. Reconstruction quality is measured by comparing the reconstructed image / video with the original image / video; higher similarity indicates better reconstruction quality.

[0028] Image / video compression techniques can be categorized into video encoding / decoding methods and neural network-based video compression methods. Video encoding / decoding schemes employ transform-based solutions, where statistical dependencies in latent variables (such as Discrete Cosine Transform (DCT) and wavelet coefficients) are carefully hand-designed using entropy encoding / decoding to model these dependencies in the quantization domain. Neural network-based video compression can be grouped into neural network-based encoding / decoding tools and end-to-end neural network-based video compression. The former is embedded as an encoding / decoding tool within existing video codecs and serves only as part of the framework, while the latter is a separate framework developed based on neural networks, independent of the video codec.

[0029] A range of video codec standards have been developed to meet the growing demand for visual content transmission. The International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) has two expert groups: the Joint Group of Picture Experts (JPEG) and the Moving Picture Experts Group (MPEG). The International Telecommunication Union (ITU) Telecommunication Standardization Sector (ITU-T) also has a Video Codec Expert Group (VCEG) for standardizing image / video codec technologies. Influential video codec standards published by these organizations include JPEG, JPEG 2000, H.262, H.264 / Advanced Video Codec (AVC), and H.265 / High-Efficiency Video Codec (HEVC). The Joint Video Experts Group (JVET), comprised of MPEG and VCEG, developed the Multi-Functional Video Codec (VVC) standard. Compared to HEVC, VVC reduces the bit rate by an average of 50% while maintaining the same visual quality.

[0030] Neural network-based image / video compression / encoding / decoding is also under development. Example neural network encoding / decoding architectures are relatively shallow, and the performance of such networks is unsatisfactory. Neural network-based methods benefit from the support of abundant data and powerful computing resources, and are therefore better utilized in a variety of applications. Neural network-based image / video compression has shown promising improvements and has been proven feasible. However, the technology is far from mature and many challenges remain to be addressed.

[0031] 2.2 Neural Networks Neural networks, also known as artificial neural networks (ANNs), are computational models used in machine learning techniques. Neural networks typically consist of multiple processing layers, each composed of several simple but non-linear basic computational units. One advantage of these deep networks is their ability to process data with multiple levels of abstraction and transform it into different kinds of representations. The representations created by neural networks are not manually designed. Instead, deep networks, including processing layers, learn from massive amounts of data using general machine learning processes. Deep learning eliminates the need for hand-crafted representations. Therefore, deep learning is considered particularly suitable for processing natively unstructured data, such as acoustic and visual signals. Processing such data has been a long-standing challenge in the field of artificial intelligence.

[0032] 2.3 Neural Networks for Image Compression Neural networks used for image compression can be divided into two categories: pixel probabilistic models and autoencoder models. Pixel probabilistic models employ predictive encoding / decoding strategies. Autoencoder models use transform-based solutions. Sometimes, these two approaches are combined.

[0033] 2.3.1 Pixel Probability Modeling According to Shannon's information theory, the optimal method for lossless encoding and decoding can achieve the lowest possible decoding rate, which is expressed as: ,in It is a symbol The probability. Arithmetic encoding and decoding is considered one of the best lossless encoding and decoding methods. Given a probability distribution. Arithmetic encoding and decoding make the encoding / decoding rate as close as possible to the theoretical limit without considering rounding errors. Therefore, the remaining problem is determining the probabilities, which is particularly challenging for natural images / videos due to the curse of dimensionality. The curse of dimensionality refers to the problem that increasing dimensionality causes a dataset to become sparse, thus requiring a rapidly increasing amount of data to effectively analyze and organize it.

[0034] Following the predictive encoding / decoding strategy, for One approach to modeling this is to predict pixel probabilities one by one in raster scan order based on previous observations. An image can be represented as follows: (1) in These are the height and width of the image, respectively. Previous observations are also referred to as the context of the current pixel. When the image is large, estimating the conditional probability can be difficult. Therefore, a simplified approach is to restrict the context of the current pixel as follows: (2) in It is a predefined constant that controls the scope of the context.

[0035] It should be noted that this condition can also consider the sample values ​​of other color components. For example, when encoding and decoding the red (R), green (G), and blue (B) (RGB) color components, the R sample depends on previously encoded pixels (including R samples, G samples, and / or B samples), and the current G sample can be encoded based on previously encoded pixels and the current R sample. Furthermore, when encoding and decoding the current B sample, previously encoded pixels, as well as the current R and G samples, can also be considered.

[0036] Neural networks can be designed for computer vision tasks and are effective in regression and classification problems. Therefore, neural networks can be used in a given context. Estimate under the circumstances The probability of.

[0037] Most methods directly model the probability distribution in the pixel domain. Some designs also model the probability distribution as a conditional probability distribution based on explicit or latent representations. Such a model can be represented as: (3) in It is an additional condition, and Indicator modeling is divided into unconditional models and conditional models. The additional conditions can be image label information or high-level representations.

[0038] 2.3.2 Automatic Encoder Now, let's describe autoencoders. Autoencoders are trained for dimensionality reduction and consist of an encoding component and a decoding component. The encoding component transforms a high-dimensional input signal into a low-dimensional representation. The low-dimensional representation can have a reduced spatial size but a greater number of channels. The decoding component recovers the high-dimensional input from the low-dimensional representation. Autoencoders enable the automatic learning of representations and eliminate the need for handcrafted features, which is considered one of the most significant advantages of neural networks.

[0039] Figure 1B This is a schematic diagram illustrating an example transform encoding / decoding scheme. Original image. Analysis network Transformation to achieve latent representation The latent representation y is quantized (q) and compressed into bits. The number of bits... Used to measure codec rate. Latent representation of quantization. Then by the synthetic network Inverse transform to obtain the reconstructed image Distortion (D) is the distortion in the perceptual space caused by using functions. Transformation The calculation produces z and , which is compared to obtain D.

[0040] Autoencoder networks can be applied to lossy image compression. The learned latent representations can be encoded from a well-trained neural network. However, applying autoencoders to image compression is not straightforward because the original autoencoder is not optimized for compression, making it inefficient to use directly as a trained autoencoder. Furthermore, other major challenges exist. First, low-dimensional representations should be quantized before encoding. However, quantization is non-differentiable, which is necessary during backpropagation when training the neural network. Second, the objectives differ in compression scenarios because both distortion and rate need to be considered. Rate estimation is challenging. Third, practical image encoding / decoding schemes should support variable rates, scalability, encoding / decoding speeds, and interoperability. In response to these challenges, various solutions are under development.

[0041] An example autoencoder for image compression using the example transform encoding / decoding scheme can be considered as a transform encoding / decoding strategy. Original image Using analysis networks Transformed, where It is the latent representation to be quantized and encoded / decoded. The synthetic network will quantize the latent representation. Inverse transform back to obtain the reconstructed image Framework utilization distortion loss function Trained, among which Distortion between It is a quantitative representation The calculated or estimated rate, and It is a Lagrange multiplier. It can be computed in the pixel domain or the receptive domain. Most example systems follow this prototype, and the differences between such systems may lie only in the network structure or the loss function.

[0042] 2.3.3 Advanced Prior Model Figure 2 An example potential representation of the image is shown. Figure 2 Includes image 201 from the Kodak dataset, visualization of the latent value 202 representing y from image 201, and the standard deviation of the latent value 202. 203 and the latent value y204 after introducing the super-prior network. The super-prior network consists of an encoder and a decoder that utilize super-prior information. In, for example... Figure 1B In the transform encoding / decoding method for image compression shown, the encoder subnetwork uses parameter analysis transform. Transform the image vector x into a latent representation Then quantify it to form .because It is a discrete value, so It can be losslessly compressed using entropy encoding and decoding techniques such as arithmetic encoding and decoding, and transmitted as a bit sequence.

[0043] from Figure 2 Potential value 202 and standard deviation 203 clearly shows that Significant spatial dependencies exist among the elements. It is noteworthy that their variances (standard deviations) also exhibit significant spatial dependencies. 203) appears to be coupled in the spatial domain. An additional set of random variables can be introduced. To capture spatial dependencies and further reduce redundancy. In this case, an image compression network such as... Figure 3 As shown.

[0044] Figure 3 This is a schematic diagram illustrating an example network architecture for an autoencoder that implements a super-prior model. The top side shows the image autoencoder network, and the bottom side corresponds to the super-prior subnetwork. The analytical transform and the synthetic transform are represented as follows: Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively. The hyperprior model consists of two sub-networks: an encoder utilizing hyperprior information (denoted as...). ) and decoders that utilize prior information (represented as The prior model generates quantized hyperlatent values ​​(). ), which includes quantifying potential values Information related to the probability distribution of the sample points. Included in the bitstream, and with They are transmitted together to the receiver (decoder).

[0045] exist Figure 3 The model on the top side is the encoder discussed above. and decoder The lower side is used to obtain... Additional encoders utilizing prior information and decoders that utilize prior information Network. In this architecture, the encoder passes the input image x through... This produces a response with a standard deviation that varies in the spatial domain. .response fed to In summary The distribution of standard deviations in the data. Then it is quantified ( The data is compressed and transmitted as side information. The encoder then uses the quantized vectors... To estimate the spatial distribution of standard deviation and use To compress and transmit quantized image representations The decoder first recovers the signal from the compressed signal. Then the decoder uses To obtain It provides the decoder with the correct probability estimate, so that it can also successfully recover the original value. Then the decoder will Feed to The reconstructed image is obtained from the image.

[0046] When an encoder and a decoder utilizing prior information are added to an image compression network, the quantization latent value is... Spatial redundancy is reduced. When using an encoder / decoder that utilizes prior information, Figure 2 The latent value y = 204 corresponds to the quantified latent value. (The difference is in the standard deviation.) Compared to 203, spatial redundancy is significantly reduced because the correlation of the samples of the quantized potential values ​​is lower.

[0047] 2.3.4 Context Model Although the prior model improves the quantification of latent values Modeling the probability distribution of quantified potential values ​​is possible, but additional improvements can be obtained by using an autoregressive model that predicts quantified potential values ​​from the causal context of the quantified potential values ​​(which may be called a context model).

[0048] The term autoregressive indicates that the output of a process is later used as the input to that process. For example, a contextual model subnetwork generates a sample of latent values, which is later used as input to obtain the next sample.

[0049] Figure 4 This is a schematic diagram illustrating an example combined model configured to jointly optimize a context model with a super-prior and an autoencoder. The meanings of the different symbols are shown below.

[0050] Table - Symbol Explanation

[0051] The combined model jointly optimizes the autoregressive components of the probability distribution of latent values ​​from the causal context of the latent values ​​(context model), along with the super-prior and low-level autoencoder. Real-valued latent representations are quantized (Q) to create quantized latent values ​​(…). ) and quantified overpotential value ( The image is compressed into a bitstream using an arithmetic encoder (AE) and decompressed by an arithmetic decoder (AD). The dashed areas correspond to components performed by the receiver (e.g., the decoder) to recover the image from the compressed bitstream.

[0052] The example system utilizes a joint architecture, where both a super-prior model subnetwork (an encoder and a decoder utilizing super-prior information) and a context model subnetwork are leveraged. The super-prior and context models are combined to learn quantized latent values. The probabilistic model on top of this is then used for entropy encoding and decoding. For example... Figure 4 As shown, the outputs of the context subnetwork and the decoder subnetwork utilizing prior information are combined by a subnetwork called the entropy parameter, which generates the mean for the Gaussian probability model. And variance (scale) (or variance) The parameters are then used. The Gaussian probability model is then used to encode the quantized latent values ​​as a bitstream with the aid of the arithmetic encoder (AE) module. In the decoder, the Gaussian probability model is used to obtain the quantized latent values ​​from the bitstream via the arithmetic decoder (AD) module. .

[0053] In one example, the latent value samples are modeled as a Gaussian distribution or a Gaussian mixture model (not limited to this). Based on... Figure 4 In the example, the context model and the hyperprior are jointly used to estimate the probability distribution of the potential samples. Since the Gaussian distribution can be defined by its mean and variance (also known as sigma or scale), the joint model is used to estimate the mean and variance (denoted as...). ).

[0054] 2.3.5 Gain Variational Automatic Encoder (G-VAE) In one example, neural network-based image / video compression methods require training multiple models to adapt to different rates. A gain variational autoencoder (G-VAE) is a variational autoencoder with a pair of gain units, designed to achieve continuous variable rate adaptation using a single model. It consists of a pair of gain units, typically inserted into the encoder's output and the decoder's input. The encoder's output is defined as the latent representation. ,in This represents the number, height, and width of the channels in the latent representation. Each channel of the latent representation is represented as... ,in A pair of gain units includes a gain matrix. and the inverse gain matrix, where This is the number of gain vectors. A gain vector can be represented as... ,in Indicates the index of the gain vector in the gain matrix.

[0055] The motivation for the gain matrix is ​​similar to the quantization table in JPEG, controlling the quantization loss based on the characteristics of different channels. To apply the gain matrix to the latent representation, each channel is multiplied by the corresponding value in the gain vector.

[0056]

[0057] in It is channel-wise multiplication, that is ,and It is the gain vector The first in There are several gain values. The inverse gain matrix used on the decoder side can be represented as... It includes One inverse gain vector, i.e. The inverse gain process is represented as...

[0058] in It is the decoded quantized latent representation, and It is the quantized latent representation of the inverse gain, which will be fed into the synthesis network.

[0059] To achieve continuous variable rate adjustment, interpolation is used between vectors. Given two pairs of gain vectors... The interpolation gain vector can be obtained through the following equation:

[0060] in These are interpolation coefficients, which control the bit rate of the generated gain vector pairs. Because... Since it is a real number, it is possible to achieve any bit rate between two given pairs of gain vectors.

[0061] 2.3.6 Encoding process using a joint autoregressive hyperprior model Figure 4 The design described corresponds to the example combined compression method. The encoding and decoding processes are described separately in this and the next section.

[0062] Figure 5 An example encoding process is illustrated. The input image is first processed by the encoder sub-network. The encoder transforms the input image into a transform representation called the latent value, which is... express. It is then fed into the quantizer block, denoted by Q, to obtain the quantized potential value ( ). Then, using an arithmetic coding module (denoted as AE), it is converted into a bitstream (bits1). The arithmetic coding blocks are sequentially... Each sample point is converted into a bitstream (bits1) one by one.

[0063] The module utilizes an encoder with prior information, context, a decoder with prior information, and an entropy parameter subnetwork to estimate the quantization latent value. The probability distribution of the sample points. Latent value The input is fed into an encoder that utilizes prior information, and the encoder that utilizes prior information outputs a latent value (denoted as...). Then the overpotential value is quantized (). The second bitstream (bits2) is generated using the arithmetic coding (AE) module. The decomposition entropy module generates a probability distribution, which is used to encode the quantized hyperlatent values ​​into a bitstream. The quantized hyperlatent values ​​include those related to the quantized latent values ​​(…). Information related to the probability distribution of ().

[0064] Entropy parameter subnetworks generate probability distribution estimates, which are used to encode quantized latent values. Information generated from entropy parameters typically includes the mean. And variance (scale) (or variance) The parameters, together, are used to obtain the Gaussian probability distribution. The Gaussian distribution of the random variable x is defined as follows: , where parameters It is the mean or expected value of the distribution (also the median and mode), while the parameter This is its standard deviation (or variance or scale). To define a Gaussian distribution, the mean and variance need to be determined. The entropy parameter module is used to estimate the mean and variance values.

[0065] The subnetwork uses a decoder with prior information to generate part of the information used by the entropy parameter subnetwork, while another part is generated by an autoregressive module called the context module. The context module uses samples already encoded by the arithmetic encoding (AE) module to generate information related to the probability distribution of the quantized latent values. (Quantization of latent values...) It is usually a matrix composed of many sample points. According to the matrix The dimension, the sample points can use such as [i,j,k] or The index of [i,j] is used to indicate the sample points. [i,j] are encoded sequentially by the AE, typically using a raster scan order. In a raster scan order, the matrix rows are processed from top to bottom, with samples within each row processed from left to right. In scenarios where the AE encodes samples into the bitstream using a raster scan order, the context module uses samples encoded before the raster scan order to generate a sequence with the samples. Information related to [i,j]. The information generated by the context module and the decoder utilizing prior information is combined by the entropy parameter module to generate information used to quantize the latent values. The probability distribution encoded into the bit stream (bits1).

[0066] Finally, the first and second bitstreams, as the result of the encoding process, are transmitted to the decoder. It should be noted that other names can be used for the modules described above.

[0067] In the above description, Figure 5 All elements in the algorithm are collectively referred to as encoders. The analytical transformation that converts the input image into a latent representation is also called an encoder (or autoencoder).

[0068] 2.3.7 Decoding process using a joint autoregressive superprior model Figure 6 An example decoding process is shown. Figure 6 The decoding process is described separately.

[0069] During decoding, the decoder first receives a first bitstream (bits1) and a second bitstream (bits2) generated by the corresponding encoder. Bits2 is first decoded by the arithmetic decoding (AD) module using a probability distribution generated by a decomposition entropy subnetwork. The decomposition entropy module typically uses a predetermined template to generate the probability distribution, for example, using predetermined mean and variance values ​​in the case of a Gaussian distribution. The output of the arithmetic decoding process for bits2 is... This refers to the quantized over-latency value. The AD process reverts to the AE process applied in the encoder. Both the AE and AD processes are lossless, meaning the quantized over-latency value generated by the encoder... It can be reconstructed at the decoder without any changes.

[0070] In obtaining Subsequently, it is processed by a decoder utilizing prior information, and the output of the decoder is fed into the entropy parameter module. The three sub-networks employed in the decoder—the context, the decoder utilizing prior information, and the entropy parameter—are the same as those in the encoder. Therefore, the exact same probability distribution can be obtained in the decoder (as in the encoder), which is crucial for lossless reconstruction of the quantized latent values. This is crucial. As a result, the decoder can obtain the quantized latent values ​​obtained in the encoder. Same version.

[0071] After obtaining the probability distribution (e.g., mean and variance parameters) through the entropy parameter subnetwork, the arithmetic decoding module decodes the quantized latent value samples one by one from the bitstream bits1. From a practical perspective, the autoregressive model (contextual model) is inherently serial, and therefore cannot be accelerated using techniques such as parallelization. Finally, the fully reconstructed quantized latent values... Input into the synthesis transform (in) Figure 6 The module (represented as decoder) is used to obtain the reconstructed image.

[0072] In the above description, Figure 6 All elements in the image are collectively referred to as the decoder. The synthetic transformation that converts the quantized latent values ​​into the reconstructed image is also called the decoder (or automatic decoder).

[0073] 2.4 Neural Networks for Video Compression Similar to video encoding and decoding techniques, neural image compression serves as the foundation for intra-frame compression in neural network-based video compression. Therefore, the development of neural network-based video compression technology has lagged behind that of neural network-based image compression, as neural network-based video compression is more complex and requires greater effort to address its challenges. Compared to image compression, video compression necessitates effective methods to remove inter-frame redundancy. Inter-frame prediction is a key step in these example systems. Motion estimation and compensation are widely employed in video codecs, but are typically not implemented by trained neural networks.

[0074] Neural network-based video compression can be categorized into two types based on the target scenario: random access and low latency. In the random access case, the system allows decoding to begin at any point in the sequence, typically dividing the entire sequence into multiple separate segments, and allowing each segment to be decoded independently. In the low latency case, the system aims to reduce decoding time, allowing earlier frames to be used as reference frames for decoding subsequent frames.

[0075] 2.5 Prerequisites Almost all natural images and / or videos are in digital format. Grayscale digital images can be generated by... It means that, among them It is a collection of pixel values. It is the image height, and It is the image width. For example, This is an example setting, and in this case... Therefore, a pixel can be represented by an 8-bit integer. An uncompressed grayscale digital image has 8 bits per pixel (bpp), while compressed images certainly have fewer bits.

[0076] Color images are typically represented in multiple channels to record color information. For example, in the RGB color space, an image can be represented by... This means that three separate channels store red, green, and blue information. Similar to an 8-bit grayscale image, an uncompressed 8-bit RGB image has 24 bpp. Digital images / videos can be represented in different color spaces. Most neural network-based video compression schemes have been developed in the RGB color space, while video codecs typically use the YUV color space to represent video sequences. In the YUV color space, an image is decomposed into three channels: luminance (Y), blue chrominance (Cb), and red chrominance (Cr). Y is the luminance component, and Cb and Cr are the chrominance components. Compression of YUV is beneficial because Cb and Cr are often downsampled for pre-compression, as the human visual system is less sensitive to the chrominance component.

[0077] A color video sequence consists of multiple color images (also called frames) to record a scene at different timestamps. For example, in the RGB color space, a color video can be composed of... It means that, among them It is the number of frames in the video sequence, and .if If the video has 50 frames per second (fps), then the data rate of the uncompressed video is _____. Bits per second (bps). This results in approximately 2.32 gigabits per second (Gbps), which consumes a significant amount of storage and should be compressed before being transmitted over the internet.

[0078] Typically, lossless methods can achieve compression ratios of around 1.5 to 3 for natural images, which is clearly below streaming requirements. Therefore, lossy compression is employed to achieve better compression ratios, but at the cost of distortion. Distortion can be measured by calculating the mean squared difference between the original and reconstructed images, for example, based on MSE. For grayscale images, MSE can be calculated using the following equation.

[0079] (4) Accordingly, the quality of the reconstructed image compared to the original image can be measured by the peak signal-to-noise ratio (PSNR): (5) in The maximum value in the range, such as 255 for an 8-bit grayscale image. There are other quality assessment metrics, such as structural similarity (SSIM) and multi-scale SSIM (MS-SSIM).

[0080] To compare different lossless compression schemes, one can compare the compression ratios at a given rate of gain, and vice versa. However, to compare different lossy compression methods, the comparison must consider both rate of gain and reconstruction quality. This can be achieved, for example, by calculating the relative rates at several different quality levels and then averaging the rates. The average relative rate is called the Byontgarde differential rate (BD rate). Other aspects to consider when evaluating image and / or video codec schemes include encoding / decoding complexity, scalability, robustness, and more.

[0081] 2.6 Separate processing of the luminance and chrominance components of an image Figure 7 An example decoding process according to this disclosure is shown.

[0082] According to one implementation, the luminance and chrominance components of an image can be decoded using separate sub-networks. Figure 7 In this process, the luminance component of the image is processed by sub-networks such as "synthesis", "predictive fusion", "mask convolution", "decoder utilizing prior information", and "super-scaling decoder". The chrominance component is processed by sub-networks such as "synthesized UV", "predictive fusion UV", "mask convolution UV", "decoder utilizing prior information UV", and "super-scaling decoder UV".

[0083] The advantage of this separate processing is that it reduces the computational complexity of image processing. Typically, in neural network-based image and video decoding, computational complexity is proportional to the square of the number of feature maps. For example, if the total number of feature maps is 192, the computational complexity will be proportional to 192 × 192. On the other hand, if the feature maps are divided into 128 for luminance and 64 for chrominance (in the case of separate processing), the computational complexity is proportional to 128 × 128 + 64 × 64, which corresponds to a 45% reduction in complexity. Generally, separate processing of the luminance and chrominance components of an image does not lead to an excessive performance degradation because the correlation between the luminance and chrominance components is usually very small.

[0084] Figure 7 The processing (decoding process) in the code can be explained as follows: 1. First, the decomposition entropy model is used to decode the quantized latent values ​​for luminance and chrominance, i.e. Figure 7 In .

[0085] 2. The probability parameters (e.g., variance) generated by the second network are used to generate quantized residual latent values ​​by performing an arithmetic decoding process.

[0086] 3. For example Figure 7 As shown in orange, the quantized residual latent value is inversely amplified using an inverse gain unit (iGain). The output of the inverse gain unit is expressed for the luminance and chrominance components respectively. .

[0087] 4. For the luminance component, the following steps are performed in a loop until... All elements are obtained: a. The first subnetwork was used for... The obtained samples are used to estimate the quantized potential value ( The mean parameter of ).

[0088] b. Quantified residual potential value The mean was used to obtain The next element.

[0089] 5. In After all the samples are obtained, a synthetic transformation can be applied to obtain the reconstructed image.

[0090] 6. For the chromaticity component, steps 4 and 5 are the same, but with a separate set of networks.

[0091] 7. The decoded luminance component is used to obtain additional information for the chrominance component. Specifically, the Inter-Channel Related Information Filter (ICCI) subnetwork is used for chrominance component recovery. Luminance is fed as additional information into the ICCI subnetwork to assist in chrominance component decoding.

[0092] 8. After the luminance and chrominance components are reconstructed, adaptive color transformation (ACT) is performed.

[0093] The module named ICCI is a neural network-based post-processing module. Examples are not limited to the UCCI subnetwork. Any other neural network-based post-processing module can also be used.

[0094] Example implementations of the disclosed content are in Figure 7The decoding process is described in the diagram. The framework comprises two branches, one for the luma and one for the chroma components. Within each branch, the first sub-network includes context, prediction, and optionally a decoder module utilizing advanced prior information. The second network includes a super-scaling decoder module. The quantized advanced latent value is... The arithmetic decoding process generates quantized residual latent values, which are further fed into the iGain unit to obtain quantized residual latent values ​​of the gain. .

[0095] After the residual latent value is obtained, a recursive prediction operation is performed to obtain the latent value. The following steps describe how to obtain samples of potential values. Furthermore, the chromaticity components are processed in the same way but using different networks.

[0096] 1. The autoregressive context module is used when using samples. This is used to generate the first input to the prediction module, where the (m, n) pairs are the indices of the sample points of the obtained potential values.

[0097] 2. Optionally, the second input to the prediction module is obtained by using a decoder that utilizes prior information and a quantized over-latency value. Obtained.

[0098] 3. Using the first and second inputs, the prediction module generates the mean. .

[0099] 4. Mean and quantified residual potential value Added together to obtain potential value .

[0100] 5. Steps 1 through 4 are repeated for the next sample point.

[0101] Whether and / or how at least one method disclosed in the document can be applied, for example, in a bitstream transmitted from the encoder to the decoder via signal transmission.

[0102] Whether and / or how to apply at least one of the methods disclosed in the document can be determined by the decoder based on encoding and decoding information (such as dimensions, color format, etc.).

[0103] In addition, modules named MS1, MS2, or MS3+O (in) Figure 7 (The input) can be included in the processing stream. This module can perform operations on its input to obtain the output by multiplying the input by a scalar or adding an additional component to the input. The scalar or additional component used by this module can be indicated in the bitstream.

[0104] Figure 7 The module named RD or AD in the code can be an entropy decoding module. It can be a range decoder or an arithmetic decoder, etc.

[0105] The examples described in this article are not limited to Figure 7 The example demonstrates a specific combination of units. Some modules may be missing, and some modules may be shifted according to the processing order. Additionally, supplementary modules may be included. For example: 1. The ICCI module can be removed. In this case, the outputs of the synthesis module and the synthesis UV module can be combined by another module, which can be based on a neural network.

[0106] 2. One or more modules named MS1, MS2, or MS3+O can be removed. The removal of one or more of these scaling and adding modules does not affect the core of the public content.

[0107] exist Figure 7 The asterisk is also used to indicate other operations performed during the processing of the luminance and chrominance components. These operations are denoted as MS1, MS2, MS3+0. These operations can be, but are not limited to, adaptive quantization, latent sample scaling, and latent sample offset operations. For example, in adaptive quantization, this can correspond to scaling the samples with a multiplier before the prediction process, where the multiplier is predefined or its value is indicated in the bitstream. Latent value scaling can correspond to scaling the samples with a multiplier after the prediction process, where the multiplier value is predefined or indicated in the bitstream. Offset operations can correspond to adding an additional element to the sample, where the value of the additional element can be indicated, estimated, or predetermined in the bitstream.

[0108] Another operation can be slicing, where the samples are first sliced ​​(grouped) into overlapping or non-overlapping regions, each of which is processed independently. For example, samples corresponding to the luminance component can be divided into slices with a slice height of 20 samples, while the chrominance component can be divided into slices with a slice height of 10 samples for processing.

[0109] Another application is wavefront parallel processing. In wavefront parallel processing, multiple samples can be processed in parallel, and the number of samples that can be processed in parallel can be indicated by control parameters. These control parameters can be indicated, estimated, or predetermined in the bitstream. In the case of separate luma and chroma processing, the number of samples that can be processed in parallel can be different, so different indicators can be transmitted via signals in the bitstream to control the operation of luma and chroma processing respectively.

[0110] 2.7 Color Separation and Conditional Encoding / Decoding Figure 8An example of a learning-based image codec architecture is shown.

[0111] In one example, such as Figure 8 As shown, the primary and secondary color components of the image are encoded and decoded separately using networks with similar architectures but different numbers of channels. All boxes with the same name are subnetworks with similar architectures, differing only in input / output tensor sizes and the number of channels. The number of channels for the primary component is... The number of channels for the secondary component is The vertical arrows (pointing downwards) indicate the data flow related to the encoding and decoding of secondary color components. The vertical arrows show the data exchange between the primary and secondary component pipelines.

[0112] The input signal to be encoded is represented as The latent space tensor in the bottleneck of a variational autoencoder is The subscript "Y" indicates the primary component, while the subscript "UV" is used for the stitched secondary components, which include chromaticity components.

[0113] First, the input image in RGB color format is converted into primary (Y) and secondary (UV) components. Primary component... Independent of secondary components The image is encoded and decoded, and the size of the encoded / decoded image is equal to the size of the input / decoded image. Using data from the primary component... As used for encoding Auxiliary information and use auxiliary information from the main component Used as a potential tensor for decoding Reconstruction is then performed, with the secondary components conditionally encoded and decoded. The encoder-decoder structures for the primary and secondary components are almost identical, differing only in the number of channels, channel size, and the several entropy models used to convert the latent tensor into a bitstream. Therefore, the primary and secondary latent tensors will generate two different bitstreams based on two different entropy models. In encoding... Before, The module adjusts the sample point position through downsampling (in...) Figure 8 The superscript "s↓" essentially means that the encoded image size of the secondary component differs from that of the primary component. The scaling factor s is variable, but the default scaling factor is 0. In conditional encoding and decoding, the size of the auxiliary input tensor is adjusted so that the encoder receives primary and secondary component tensors with the same image size. After reconstruction, the secondary component utilizes a neural network-based upsampling filter module (…). Figure 8 The "NN color filter s↑" on the image is rescaled to the original image size, and the module output is factored. The secondary component that is upsampled.

[0114] Figure 8 The example illustrates an image encoding / decoding system where the input image is first transformed into a primary (Y) component and a secondary (UV) component. Output... This is the reconstructed output corresponding to the primary and secondary components. At the end of the processing, It is converted back to RGB color format. Typically, this is done before processing using encoding and decoding modules (neural networks). Downsampled (resized). For example, The size can be reduced by a factor of 50% in each of the vertical and horizontal dimensions. Therefore, the processing of minor components involves approximately 50% × 50% = 25% fewer samples, making it computationally less complex.

[0115] 2.8 Pruning Operations in Neural Network-Based Encoding and Decoding Figure 9 An example synthetic transform for learning-based image encoding and decoding is shown.

[0116] The above example of a synthetic transform consists of a sequence of four convolutions, each with an upsampling stride of 2. The synthetic transform subnetwork is in... Figure 9 The tensors in the different parts of the composite transform before the clipping layer are depicted in the diagram. Figure 9 The image above.

[0117] Clipping layers will tensor dimensions Change to ,in ;here This is the depth of the convolution performed in the codec architecture. For the principal components, the synthesized transform receive size is... The input tensor, where The output of the synthesis transform of the principal components is ,in .

[0118] For the secondary components, the synthesized transform receiver size is The input tensor; The output of the synthesis transform of the principal components is ,in For secondary components, the input size is... ,in It is the scaling factor. For example, the scaling factor could be 2, where the minor components are downsampled by a factor of 2.

[0119] Based on the above explanation, the operation of the clipping layer depends on the output dimensions H and W and the depth of the clipping layer. Figure 9 The leftmost clipping layer has a depth of 0. The output of this clipping layer must be equal to H and W (output dimensions). If the input dimensions of this clipping layer are greater than H or W in the horizontal or vertical dimension, respectively, then clipping needs to be performed in that dimension. The second clipping layer, counting from left to right, has a depth of 1. The output of the second clipping layer must be equal to... This means that if the input to the second clipping layer is greater than h1 or w1 in any dimension, clipping is applied to that dimension. In summary, the operation of the clipping layers is controlled by the output dimensions H and W. In one example, if both H and W are equal to 16, the clipping layer does not perform any clipping. On the other hand, if both H and W are equal to 17, all four clipping layers will perform clipping.

[0120] 2.9 Displacement by Shift Bitwise shift operators can use functions Let be the bitwise AND operator, where n is an integer. If n is greater than 0, it corresponds to the right shift operator (>>) and the left shift operator (<<). The right shift operator shifts the input bits to the right, and the left shift operator shifts the bits to the left. In other words, The operation corresponds to: ,or ,or .

[0121] Bitwise shift operations output integer values. In some implementations, the floor() function can be added to the definition.

[0122] floor(x) is equal to a maximum integer that is less than or equal to x.

[0123] The " / / " operator is the integer division operator. It includes division and truncation of the result towards zero. For example, 7 / 4 and... 7 / 4 is truncated to 1, and 7 / 4 and 7 / 4 was truncated to 1.

[0124] or

[0125] Equation 3: Bit shift operators are alternative implementations for right or left shift.

[0126] x >> yThe two's complement integer representation of x is arithmetically shifted right by y binary digits. This function is only defined for non-negative integer values of y. The bit shifted into the most significant bit (MSB) due to the right shift has the value of the MSB of x before the shift operation.

[0127] x << yThe two's complement integer representation of x is arithmetically shifted left by y binary digits. This function is only defined for non-negative integer values of y. The bit shifted into the least significant bit (LSB) due to the left shift has the value equal to 0.

[0128] 2.10 Convolution Operation The convolution operation starts with a kernel, which is a small matrix of weights. The kernel "slides" over the input data, performs element-wise multiplication with the part of the input it is currently on, and then sums the results into a single output pixel. In some cases, the convolution operation can include a "bias", which is added to the output of the element-wise multiplication operation.

[0129] The convolution operation can be described by the following mathematical formula. The output out1 can be obtained as:

[0130] where w1 is the multiplication factor and K1 is called the bias (additive term), is the k-th input, N is the kernel size in one direction, and P is the kernel size in the other direction. The convolution layer can include a convolution operation, and more than one output can be generated in the convolution operation. Other equivalent depictions of the convolution operation can be found below:

[0131] In the above equations, "c" indicates the channel number. It is equivalent to the output number, out[1,x,y] is one output, and out[2,x,y] is the second output. k is the input number, is an input, and is the second input. w1 or w describes the weights of the convolution operation.

[0132] 2.11 LeakyReLU Activation Function Figure 10 An example LeakyReLU activation function is shown. The LeakyReLU activation function is depicted in Figure 10 . According to this function, if the input is positive, the output is equal to the input. If the input (y) is negative, the output is equal to a y. a is typically (but not limited to) a value less than 1 and greater than 0. Since the multiplier a is less than 1, it can be implemented as a multiplication or division operation with non-integers. The multiplier a can be called the negative slope of the LeakyReLU function.

[0133] 2.12 ReLU activation function Figure 11 An example ReLU activation function is shown. The ReLU activation function is in Figure 11 The function is described in the diagram. According to this function, if the input is positive, the output equals the input. If the input (y) is non-positive, the output equals 0.

[0134] 3. The technical problem solved by the disclosed technical solution When image components (e.g., luminance and chrominance components) are processed using different synthesis subnetworks, the correlation between the different components is not fully utilized. In other words, information that might be important for the reconstruction of one component may also be relevant to the reconstruction of a second component. This joint information is not fully utilized when two different synthesis transforms are used to reconstruct two different components.

[0135] 4. List of solutions and implementation examples 4.1 Core Example The objective of this disclosure is to improve the quality of an image component by using information from another component. This objective is achieved by: • Use the common processing layer used in neural network implementations.

[0136] • And by including the weights and offset (bias) parameters of the processing layer in the bitstream.

[0137] Decoder operation: Based on some examples, using a neural network to convert a bitstream into a reconstructed image includes the following operations: Obtain weight values ​​from the bitstream.

[0138] Get the offset value.

[0139] Obtain the result value based on any or all of the following: o applies the offset value to the samples of the first component.

[0140] o Apply a threshold function (e.g., ReLU operation) to the samples of the first component.

[0141] o Apply the weight values ​​to the samples of the first component.

[0142] Based on the result value and the sample points of the second component, obtain the modified sample points of the second component.

[0143] The reconstructed image is obtained using samples from the first component and samples from the modified second component.

[0144] Encoder operation: Based on some examples, using a neural network to convert an image into a bitstream includes the following operations: Get / determine the offset value.

[0145] Obtain the result value based on any or all of the following: o applies the offset value to the samples of the first component.

[0146] o Apply a threshold function (e.g., ReLU operation) to the samples of the first component.

[0147] o Calculate the weight values.

[0148] Based on the result value and the sample points of the second component, obtain the modified sample points of the second component.

[0149] The reconstructed image is obtained using samples from the first component and samples from the modified second component, where weight values ​​are calculated (selected) to maximize the quality of the reconstructed image.

[0150] Include the weight values ​​in the bitstream.

[0151] The first component, or the second component, or any of the above components can be components of the image.

[0152] It can be a chromaticity component or a luminance component.

[0153] Prior to applying some embodiments of this disclosure, the mean could be subtracted from any component.

[0154] After applying some embodiments of this disclosure, the mean can be added to the upsampled component.

[0155] In one example, the first component is Y in the YCbCr color format, and the second component is either Cb or Cr.

[0156] In one example, the first component is the G component in the RGB color format, and the second component is the B / R component.

[0157] In one example, two offsets and / or two weights can be transmitted via signals in the bitstream.

[0158] • Alternatively, only one offset and / or one weight can be transmitted through the signal in the bitstream, and the second / third component can share the same value.

[0159] • Alternatively, predictive coding / decoding can be applied to encode / decode one of the two weights.

[0160] • Alternatively, predictive coding / decoding can be applied to code / decode one of the two offsets.

[0161] 4.2 Details of the Example • The five example implementations disclosed herein can be based on the following equation: (6) or (7) or (8) or (9) or (10) In the above equation, the first component is recY (e.g., the luminance component of the image).

[0162] The second component is recU (e.g., the chromaticity component of an image).

[0163] The threshold function is the RELU() function.

[0164] The weight is W[n]. In the above equation, M different weight values ​​are used.

[0165] The offset is b[n]. In the example equation, M distinct offset values ​​are used.

[0166] The index [1,x,y] indicates the sample point at coordinates [1, x, y], which is the coordinate of the sample point of the first component or the second component.

[0167] According to equation (6), the multiplication weight value W[n] is first applied to the samples of the first component. Then the additive offset value is applied. The weights and offsets are applied to the sample points. Then a threshold function (ReLU in the example) is applied. In the example, at most M such weights and offsets are applied to the first sample point, and the results are summed (…). The samples are added together. Finally, the result of the summation is added to the samples of the second component. Equation (8) is similar to equation (6).

[0168] o According to equation (7), the addition offset value First, it is applied to the sample points. Then, a threshold function (ReLU in the example) is applied. Then, the weight values ​​are multiplied ( ) is applied. In the example, at most M such weights and offsets are applied to the first point, and the result is summed ( The samples are added together. Finally, the result of the summation is added to the samples of the second component. Equation (9) is similar to equation (6).

[0169] Before inputting into the process, the mean can be subtracted from recU or recY. The mean can be the average of the samples in recU or recY.

[0170] The mean can be added to the modified recU. The mean can be the average of the samples in recU or recY.

[0171] • Figure 12 This is a flowchart of the video processing method for the example. Figure 13 This is a flowchart of the video processing method for the example. Figure 12 and Figure 13 The flowchart in the document illustrates an example implementation of this disclosure.

[0172] First, perform an offset subtraction (or addition) on component 1. Then, in Figure 12 In the first part, a threshold function is executed. Finally, weights are applied, and the output is added to the second component. At the end of the flowchart, the modified second component is obtained. The reconstructed image at the end of the decoder or encoder is obtained based on the first component and the modified second component.

[0173] o in Figure 13 In the middle, operation and Figure 12 Very similar, except that the order of the weight multiplication and threshold operation has been swapped.

[0174] o Before applying some embodiments of this disclosure, the mean can be subtracted from the input. The mean can be the mean (average value) of the samples of the first component of the second component.

[0175] The mean can be added to the method's output. The mean can be the average of the samples from the first component of the second component.

[0176] • The threshold function can be (but is not limited to) a ReLU operation, a Leaky ReLU operation, a Sigmoid operation, a hyperbolic tangent operation, or a MAX(x,y) operation or a MIN(x,y) operation. The MAX(x,y) operation outputs the maximum of two values ​​x or y, and the MIN(x,y) operation outputs the minimum of two values ​​x or y.

[0177] The Sigmoid function can be described as: .

[0178] The hyperbolic tangent can be described as: .

[0179] In the MAX(x,y) or MIN(x,y) operation, one of the input values ​​can be zero. In other words, the threshold function can be MAX(x, 0) or MIN(x, 0).

[0180] • Weight values ​​can be implemented as part of a convolution function.

[0181] • The offset value can be implemented as a part of the convolution function. More specifically, the offset value can be implemented as the bias value of the convolution function.

[0182] • The first component can be the luminance component or the luminance value component of the image.

[0183] • The second component can be the U chromaticity component, or the V chromaticity component, or a chromaticity component or a chromaticity value component.

[0184] • Figure 14 This is a flowchart of an example method for video processing. Figure 14 The flowchart in the document depicts another example implementation of this disclosure. In this example, the first component... From the first convolutional layer (e.g., as Figure 14 In the context of Conv(1x1, 2, 16, bias = 1), activation functions (e.g., Figure 14 ReLU in the second convolutional layer (e.g., ReLU in the second convolutional layer) and the second convolutional layer (e.g., ReLU in the second convolutional layer) Figure 14 The Conv(1x1, 16, 1, bias=0) function in the convolutional layer is used for processing. Convolutional layers can apply offsets (i.e., biases) and multiplicative values ​​(weights). Therefore, multiplicative weights and additive offsets can be applied through convolutional layers. Figure 14 The examples illustrate the fact that this disclosure can be implemented using the most common neural network processing layers, namely convolutional layers and activation layers such as the ReLU function.

[0185] • Figure 15 This is a flowchart of an example method for video processing. Figure 15The flowchart in the document depicts another exemplary implementation of this disclosure. This example is related to... Figure 14 Similarly, the difference is that both the first and second components are fed into the first convolutional layer (e.g., Conv(1x1, 2, 16, bias=1)), and the addition operation at the end is removed.

[0186] o in Figure 15 In this context, the following equation can be realized: ) • Figure 16 An example neural network is shown. Figure 16 The flowchart in the diagram illustrates how this disclosure is implemented within a larger network.

[0187] • According to this disclosure, the multiplication weight values ​​can be included in the bitstream at the encoder or obtained from the bitstream at the decoder.

[0188] • According to this disclosure, the addition offset (or bias value) can be obtained from the bitstream.

[0189] • The mean can be subtracted from recU or recY before being input into the process. The mean can be the average of the samples in recU or recY.

[0190] • The mean can be added to the modified recU. The mean can be the average of the samples in recU or recY.

[0191] • The offset value can be obtained from the maximum and / or minimum value.

[0192] The maximum value can be the maximum value of the sample points of the first component.

[0193] The minimum value can be the minimum value of the sample points of the first component.

[0194] The maximum or minimum value can be obtained from the bit stream.

[0195] The offset value can be obtained based on the value N used to divide the maximum and minimum values.

[0196] N can be predefined or obtained from a bitstream.

[0197] The offset values ​​1…N can be obtained as follows: , where n and N are integer values.

[0198] Figure 17 An example neural network is shown. Figure 17Another embodiment of this disclosure is described.

[0199] EFE luminance-assisted nonlinear filtering process The input to this process is The output of this process is .

[0200] Multiplication weight parameters Used.

[0201] Addition bias parameters Used.

[0202] For x in 0..W, y in 0..H, and k in 0..1, the following is executed: .

[0203] 4.3. Explanation and Benefits of the Example The example uses parameters obtained from the bitstream to improve the quality of the reconstructed image. The example is designed to achieve the following benefits: 1. Some parameters used in the equation are obtained from the bitstream. This provides the possibility of content adaptation. In neural network-based image compression networks, the network can be pre-trained using a very large dataset. After training is complete, the network parameters (e.g., weights and / or biases) cannot be adjusted. However, when using the network, it is used with entirely new images that are not part of the training dataset. Therefore, there is a difference between the training dataset and real-world images. To address this issue, a small set of parameters optimized for new images is passed to the decoder to improve adaptability to new content.

[0204] The second benefit of including parameters in the bitstream is that a much shorter network can be used to serve the same purpose when transmitting parameters. In other words, if parameters were transmitted as side information, a much longer neural network (including more convolutional and activation layers) might be needed to achieve the same result.

[0205] 2. The examples can be implemented using the most basic neural network layers. The equations used to explain the examples are designed so that they can be implemented using the most basic processing layers (i.e., convolution and ReLU operations) from neural network literature. This intentional choice is based on the expectation of implementing image encoders / decoders in a wide range of devices, including mobile phones. Importantly, an image encoded in one device can be decoded in almost any device. Although neural processing chipsets or GPUs in such devices are becoming increasingly complex, it remains impossible to implement arbitrary functions on such processing units. As a simple example, the function... Although it appears very simple, it cannot be efficiently implemented in a neural processing unit and can only be implemented in a general-purpose processing unit (such as a CPU). If the function cannot be implemented in a neural processing unit, processing speed and battery consumption will increase significantly.

[0206] The example eliminates the aforementioned problems by using the most basic processing layers from neural network literature. Convolutions and ReLU (as well as some other activation functions such as leaky ReLU, sigmoid, etc.) are almost guaranteed to be implemented in a neural processing unit or GPU. Therefore, it is expected that mobile phones with neural processing units or GPUs can efficiently perform the defined operations.

[0207] 3. The example utilizes cross-component information to improve the components of an image. According to the example, the quality of the components is enhanced, so the reconstructed image is closer to the original image, which is the goal of a good codec. The example achieves this by utilizing information included in one component to improve the quality of the second component.

[0208] 4.4. Further Solutions According to some embodiments of this disclosure, the input can have a format of 4:4:4, 4:2:0, or 4:2:2. The format of the image being processed indicates the size ratio between the image components. For example, 4:4:4 may indicate that the size of the luminance component (or primary component) is the same as the size of the chrominance component (or secondary component). 4:2:2 may indicate that the size of the secondary component can be half the size of the luminance component in one dimension (width or height). 4:2:0 may indicate that the size of the secondary component can be half the size of the luminance component in both dimensions (width and height).

[0209] According to some embodiments of this disclosure, the downsampling process can be performed first on the principal components. This is in Figure 18 The example is in the middle. Figure 18 A block diagram illustrating the downsampling process is shown.

[0210] “ "Indicates downsampling operations in both the horizontal and vertical directions."

[0211] • It can indicate the downsampling ratio in the vertical or horizontal direction.

[0212] • The downsampling ratio can be determined by the image format. Alternatively, it can be determined by the ratio of the size of the primary component to the size of the secondary component.

[0213] • The downsampling operation can be a nearest neighbor downsampling operation. Alternatively, the downsampling operation can be an average pooling operation.

[0214] • Two or one downsampling ratios can be used. When using a single downsampling ratio, the downsampling ratios in the horizontal and vertical dimensions can be considered equal.

[0215] • If the image format is 4:4:4, the downsampling ratio can be considered as 1 (no downsampling).

[0216] • If the image format is 4:2:2, the downsampling ratio can be considered as 1 in one dimension (horizontal or vertical) and as 2 in another dimension.

[0217] • If the image format is 4:2:0, the downsampling ratio can be considered as 2 in both dimensions.

[0218] According to some embodiments of this disclosure, after downsampling of the major components, they can be further processed by a process according to some embodiments of this disclosure.

[0219] Example implementations of some embodiments of this disclosure are presented below.

[0220] Nonlinear chromaticity enhancement filter This section details nonlinear filters used for minor component enhancement.

[0221] This procedure is invoked if EFE_nonlinear_filter_enabled_flag equals 1.

[0222] The input to this process is the output of the ICCI process. and as the second output of the 14.2 adaptive upsampler process as well as .

[0223] The output of this process is .

[0224] The following sequential steps are performed: The parsing procedure for the parsing table in 9.3.1.4 is called to obtain... numTiles, maxLuma, and minLuma.

[0225] The nonlinear filter parameter slicing process described in Section 14.3.1 is invoked, with the parsed syntax elements as input and the Tile2 tensor as output.

[0226] Addition bias parameters The following is obtained for i=0…numTiles: ] = , n=0...7. NLEnable[0] is set to equal nonLinear_enabled_U_flag, and NLEnable[1] is set to equal nonLinear_enabled_V_flag.

[0227] It was obtained using the nearest neighbor downsampling process, where As input, and As the downsampling ratio.

[0228] For x = 0... -1, y is at 0... In -1, and k is in 0..1, the following is executed: .

[0229] For x in the range of 0... In -1, y is at 0... In -1, the following is executed:

[0230]

[0231] Furthermore, the nearest neighbor downsampling process can be implemented as follows.

[0232] Nearest neighbor subsampling Using a single downsampling ratio Or two downsampling ratios Indicated. When only a single downsampling ratio is indicated, Assumed to be the same and equal to The receiving dimension of this layer is... tensor (Input), and output size is tensor (Output) Through this process, the spatial resolution of each tensor channel is reduced.

[0233] For this downsampling, no interpolation is needed; simply copy:

[0234] • U and V can indicate a minor component among minor components.

[0235] • It can indicate the main components. It can indicate the main component after downsampling.

[0236] • It can indicate the multiplier parameters. These can be parameters of the convolutional layer.

[0237] • RELU indicates a corrected linear unit operation.

[0238] • [0] or A flag indicating a value of 0 or 1.

[0239] • Indicates the secondary components of the image.

[0240] • This is the output.

[0241] • based on The value, input It can be output as is, or it can be modified. If If the value of is 0, then the input is not modified (and is equal to the output).

[0242] • It is the bias value, which is calculated based on two values, minLuma and maxLuma. minLuma and maxLuma can be included in the bitstream.

[0243] Further details of embodiments of this disclosure relating to neural network-based visual data encoding and decoding will now be described. As used herein, the term "visual data" may refer to video, images, pictures in video, or any other visual data suitable for encoding and decoding.

[0244] As discussed above, in existing designs for visual data encoding and decoding based on neural networks (NNs), the output visual data from the decoding process is always in a 4:4:4 format; that is, it is fixed. Therefore, this traditional solution cannot support different output formats, thus lacking flexibility.

[0245] To address the above-mentioned problems and other issues not mentioned, a visual data processing solution is disclosed as described below. The embodiments of this disclosure should be considered as examples illustrating general concepts and should not be interpreted in a narrow sense. Furthermore, these embodiments can be applied individually or in any combination.

[0246] Figure 19A flowchart of a method 1900 for visual data processing according to some embodiments of the present disclosure is shown. Method 1900 can be implemented during the conversion between visual data and a bitstream of visual data using a neural network (NN)-based model. As used herein, the NN-based model can be a model based on neural network techniques. For example, the NN-based model can specify a sequence of neural network modules (also called an architecture) and model parameters. A neural network module can include a set of neural network layers. Each neural network layer specifies tensor operations for receiving and outputting tensors, and each layer has trainable parameters. It should be understood that the possible implementations of the NN-based model described herein are illustrative only and should not be construed as limiting the present disclosure in any way.

[0247] like Figure 19 As shown, method 1900 begins at 1902, wherein at least one intermediate representation of visual data in a neural network-based model is obtained. In some example embodiments, the at least one intermediate representation may be in the pixel domain. In one example embodiment, at least one intermediate representation of visual data can be obtained by processing the output of a synthetic transform in a neural network-based model. By way of example, and not limitation, one or more filters may be applied to the output of the synthetic transform to obtain at least one intermediate representation of visual data. This will be described in detail below. In another example embodiment, at least one intermediate representation of visual data can be obtained by processing a latent representation of visual data using a synthetic transform in a neural network-based model. For example, at least one intermediate representation may be determined as the output of the synthetic transform. In some alternative example embodiments, at least one intermediate representation may be in the transform domain. For example, at least one intermediate representation may be a latent representation of visual data. It should be understood that the possible implementations of at least one intermediate representation described herein are merely illustrative and should not be construed as limiting this disclosure in any way.

[0248] At 1904, a first filter in the NN-based model is applied to at least one intermediate representation. At least one parameter of the first filter is configured based on the format of the output visual data from the transformation. This will be described in detail below. By way of example, and not limitation, the first filter can be a nonlinear filter, a nonlinear chroma enhancement filter, etc. The scope of this disclosure is not limited in this respect.

[0249] As used herein, the format of the output visual data from the transformation can refer to the format of the visual data produced by the transformation (e.g., the output of the entire decoding process). This format may also be referred to as a "first format," "output format," "output image format," and / or similar names. It should also be noted that the term "format" may also be referred to as "color format," etc. It should be noted that the first format is permitted to differ from the second format used to encode and decode the visual data. The second format indicates the format of the output of the synthetic transformation and may also be referred to as an "encoding / decoding format," "encoded / decoded format," "encoded / decoded image format," "encoding / decoding mode," and / or similar names. The input to the analytical transformation on the encoder side may also be this second format. Alternatively, the first format may be the same as the second format.

[0250] In some embodiments, the first format may be indicated by at least one indication in the bitstream. In this case, the first format can be obtained by parsing the bitstream. Additionally or alternatively, the second format may be indicated by at least one indication in the bitstream. Furthermore, the first format may be determined based on the second format or any other suitable encoding / decoding information for visual data. Additionally or alternatively, the first and second formats may be indicated by the same indication(s) in the bitstream.

[0251] The first format indicates the relationship between the dimensions of a first component and a second component of the output visual data. For example, the first format may indicate the ratio between the vertical dimensions (e.g., height) of the first component and the vertical dimensions (e.g., height) of the second component, and / or the ratio between the horizontal dimensions (e.g., width) of the first component and the horizontal dimensions (e.g., width) of the second component. Similarly, the second format used for encoding and decoding visual data may indicate the relationship between the dimensions of the first component and the second component of the encoded and decoded visual data. For example, the second format may indicate the ratio between the vertical dimensions (e.g., height) of the first component and the vertical dimensions (e.g., height) of the second component, and / or the ratio between the horizontal dimensions (e.g., width) of the first component and the horizontal dimensions (e.g., width) of the second component.

[0252] In some embodiments, the first format may be 4:4:4, 4:2:0, 4:2:2, etc. Additionally, the second format may be 4:4:4, 4:2:0, or 4:2:2, etc. By way of example, for a 4:4:4 format, the vertical dimension of the second component may be the same as the vertical dimension of the first component, and the horizontal dimension of the second component may be the same as the horizontal dimension of the first component. For a 4:2:2 format, the vertical dimension of the second component may be half the vertical dimension of the first component, and the horizontal dimension of the second component may be the same as the horizontal dimension of the first component. Alternatively, the vertical dimension of the second component may be the same as the vertical dimension of the first component, and the horizontal dimension of the second component may be half the horizontal dimension of the first component. For a 4:2:0 format, the vertical dimension of the second component may be half the vertical dimension of the first component, and the horizontal dimension of the second component may be half the horizontal dimension of the first component. In some embodiments, the first component may include one of the following: a principal component, a luminance component, or a Y component, and the second component may include one of the following: a principal component, a chrominance component, a U component, or a V component. It should be understood that the above examples are described for illustrative purposes only. The scope of this disclosure is not limited in this respect.

[0253] It can be seen that the ratio between the vertical dimension of the first component and the vertical dimension of the second component can be 1 or 2, and the ratio between the horizontal dimension of the first component and the horizontal dimension of the second component can be 1 or 2. It should be understood that the specific values ​​listed herein are intended to be illustrative and do not limit the scope of this disclosure.

[0254] At 1906, the conversion is performed based on the application. In some embodiments, the conversion may include encoding visual data into a bitstream. Additionally or alternatively, the conversion may include decoding visual data from the bitstream. It should be understood that the above description is for illustrative purposes only. The scope of this disclosure is not limited in this respect.

[0255] Based on the above, the parameters(s) of the first filter(s) applied to at least one intermediate representation of visual data are configured based on the format of the output visual data. Compared to conventional solutions where the parameters of the first filter are fixed, the proposed solution can advantageously support different output formats to cater to different applications. In this way, encoding and decoding flexibility can be improved, thereby enhancing encoding and decoding efficiency.

[0256] In some embodiments, in the first filter, a first intermediate result can be obtained using a downsampling process having a first intermediate representation from at least one intermediate representation as input and at least one downsampling ratio. By way of example and not limitation, the downsampling process may include a nearest neighbor downsampling process, which has been described in detail in the previous section. Additionally or alternatively, the downsampling process may include an average pooling operation. It should be understood that the possible implementations of the downsampling processes described herein are merely illustrative and should not be construed as limiting this disclosure in any way.

[0257] In some embodiments, the first intermediate representation may be associated with the first component. Figure 18 In the example shown, the first intermediate representation is represented as And it is associated with the Y component. At least one downsampling ratio for the downsampling process can be configured based on a first format. That is, the aforementioned at least one parameter of the first filter can include at least one downsampling ratio for the downsampling process.

[0258] In some embodiments, at least one downsampling ratio may include a first downsampling ratio in the vertical direction and / or a second downsampling ratio in the horizontal direction. For example, the relationship indicated by the first format may include a first ratio between the vertical dimension of a first component of the output visual data and the vertical dimension of a second component of the output visual data. Furthermore, the first downsampling ratio may be determined based on the first ratio. By way of example, the first downsampling ratio may be set to be equal to the first ratio, or the first downsampling ratio may be a function of the first ratio.

[0259] Additionally or alternatively, the relationship may include a second ratio between the horizontal dimension of the first component of the output visual data and the horizontal dimension of the second component of the output visual data. Furthermore, the second downsampling ratio may be determined based on this second ratio. By way of example, the second downsampling ratio may be set to be equal to the second ratio, or the second downsampling ratio may be a function of the second ratio.

[0260] In some embodiments, if the first format is 4:4:4, then the first downsampling ratio can be equal to 1, and the second downsampling ratio can be equal to 1. If the first format is 4:2:2, then one of the first and second downsampling ratios can be equal to 1, and the other downsampling ratio can be equal to 2. If the first format is 4:2:0, then the first downsampling ratio can be equal to 2, and the second downsampling ratio can be equal to 2. It should be understood that the above examples are described for illustrative purposes only. The scope of this disclosure is not limited in this respect.

[0261] In some embodiments, a second intermediate result can be generated by applying a first sub-network of a first filter to a second intermediate representation and a first intermediate result in at least one intermediate representation. For example, the first sub-network may include a concatenation operation, convolutional layers, rectified linear units (ReLU), etc. By way of example, and not limitation, the second intermediate representation may be associated with a second component. In one example embodiment, the second intermediate representation may be the output of an inter-channel related information (ICCI) filter in a neural network-based filter. Figure 18 In the example shown, the second intermediate representation can be represented as And it is associated with the U and V components. The second intermediate result can be represented as... .

[0262] Furthermore, the output of the first filter can be generated by applying a second sub-network of the first filter to a third intermediate representation and a second intermediate result in at least one intermediate representation. For example, the second sub-network may include splicing operations and / or convolutional layers. By way of example, and not limitation, the third intermediate representation may be associated with the second component. In one example embodiment, the third intermediate representation may be the output of an adaptive filter in a neural network-based filter. Figure 18 In the example shown, the third intermediate representation can be represented as And it is associated with the U component and the V component.

[0263] Based on the foregoing, the solutions according to some embodiments of this disclosure can advantageously support different output formats to cater to diverse applications. In this way, encoding / decoding flexibility can be improved, thereby enhancing encoding / decoding efficiency.

[0264] According to another embodiment of this disclosure, a non-transitory computer-readable recording medium is provided. This non-transitory computer-readable recording medium stores a bitstream of visual data generated by a method performed by a visual data processing apparatus. The method includes: acquiring at least one intermediate representation of the visual data in a neural network (NN)-based model; applying a first filter from the NN-based model to the at least one intermediate representation, at least one parameter of the first filter being configured based on a format from the transformed output visual data, the format indicating a relationship between the dimensions of a first component and a second component of the output visual data; and generating a bitstream based on the NN-based model for the application.

[0265] According to further embodiments of this disclosure, a method for storing a bitstream of visual data is provided. The method includes: acquiring at least one intermediate representation of the visual data in a neural network (NN)-based model; applying a first filter from the NN-based model to the at least one intermediate representation, the first filter having at least one parameter configured based on a format from the transformed output visual data, the format indicating a relationship between the dimensions of a first component of the output visual data and the dimensions of a second component of the output visual data; generating a bitstream using the NN-based model based on an application; and storing the bitstream in a non-transitory computer-readable recording medium.

[0266] Implementations of this disclosure may be described in accordance with the following entries, and its features may be combined in any reasonable manner.

[0267] Item 1. A method for visual data processing, comprising: for a transformation between visual data and a bitstream of the visual data using a neural network (NN)-based model, obtaining at least one intermediate representation of the visual data in the NN-based model; applying a first filter in the NN-based model to the at least one intermediate representation, wherein at least one parameter of the first filter is configured based on a format of output visual data from the transformation, the format indicating a relationship between the size of a first component of the output visual data and the size of a second component of the output visual data; and performing the transformation based on the application.

[0268] Item 2. The method according to Item 1, wherein in the first filter, a first intermediate result is acquired using a downsampling process having a first intermediate representation of the at least one intermediate representation as input, and the at least one parameter includes at least one downsampling ratio for the downsampling process.

[0269] Item 3. The method according to Item 2, wherein the first intermediate representation is associated with the first component.

[0270] Item 4. The method according to any one of Items 2 to 3, wherein the at least one downsampling ratio includes at least one of the following: a first downsampling ratio in the vertical direction, or a second downsampling ratio in the horizontal direction.

[0271] Item 5. The method according to Item 4, wherein the relationship includes a first ratio between the vertical dimension of the first component of the output visual data and the vertical dimension of the second component of the output visual data, and the first downsampling ratio is determined based on the first ratio.

[0272] Item 6. The method according to any one of Items 4 to 5, wherein the relationship includes a second ratio between the horizontal dimension of the first component of the output visual data and the horizontal dimension of the second component of the output visual data, and the second downsampling ratio is determined based on the second ratio.

[0273] Item 7. The method according to any one of items 4 to 6, wherein if the format is 4:4:4, the first downsampling ratio is equal to 1 and the second downsampling ratio is equal to 1; or if the format is 4:2:2, one of the first downsampling ratio and the second downsampling ratio is equal to 1 and the other of the first downsampling ratio and the second downsampling ratio is equal to 2; or if the format is 4:2:0, the first downsampling ratio is equal to 2 and the second downsampling ratio is equal to 2.

[0274] Item 8. The method according to any one of items 2 to 7, wherein the second intermediate result is generated by applying a first sub-network of the first filter to the second intermediate representation and the first intermediate result in the at least one intermediate representation.

[0275] Item 9. The method according to Item 8, wherein the first sub-network comprises at least one of the following: a splicing operation, a convolutional layer, or a rectified linear unit (ReLU).

[0276] Item 10. The method according to any one of items 8 to 9, wherein the second intermediate representation is associated with the second component, and the second intermediate representation is the output of the inter-channel related information (ICCI) filter in the NN-based filter.

[0277] Item 11. The method according to any one of Items 8 to 10, wherein the output of the first filter is generated by applying a second sub-network of the first filter to a third intermediate representation and the second intermediate result in the at least one intermediate representation.

[0278] Item 12. The method according to Item 11, wherein the second sub-network comprises at least one of the following: a splicing operation, or a convolutional layer.

[0279] Item 13. The method according to any one of items 11 to 12, wherein the third intermediate representation is associated with the second component, and the third intermediate representation is the output of the adaptive filter in the NN-based filter.

[0280] Item 14. The method according to any one of Items 2 to 13, wherein the downsampling process includes a nearest neighbor downsampling process.

[0281] Item 15. The method according to any one of Items 1 to 14, wherein the first component comprises one of the following: a principal component, a luminance component or a Y component, and the second component comprises one of the following: a principal component, a chromaticity component, a U component or a V component.

[0282] Item 16. The method according to any one of items 1 to 15, wherein the format is indicated by at least one indication in the bit stream.

[0283] Item 17. The method according to any one of items 1 to 16, wherein the format is permitted to be one of the following: 4:4:4 format, 4:2:0 format or 4:2:2 format.

[0284] Item 18. The method according to any one of items 1 to 17, wherein the first filter comprises a nonlinear filter.

[0285] Item 19. The method according to any one of items 1 to 18, wherein the at least one intermediate representation of the visual data is obtained by processing the output of the synthetic transformation in the NN-based model.

[0286] Item 20. The method according to any one of items 1 to 18, wherein the at least one intermediate representation of the visual data is obtained by processing the latent representation of the visual data using a synthetic transformation in the NN-based model.

[0287] Item 21. The method according to any one of items 2 to 20, wherein the downsampling process includes an average pooling operation.

[0288] Item 22. The method according to any one of items 1 to 21, wherein the visual data includes video, a picture of the video, or an image.

[0289] Item 23. The method according to any one of items 1 to 22, wherein the conversion includes encoding the visual data into the bitstream.

[0290] Item 24. The method according to any one of items 1 to 22, wherein the conversion includes decoding the visual data from the bitstream.

[0291] Item 25. An apparatus for visual data processing, comprising a processor and a nontransitory memory having instructions, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of items 1 to 24.

[0292] Item 26. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method according to any one of items 1 to 24.

[0293] Item 27. A non-transitory computer-readable recording medium storing a bitstream generated by a method performed by an apparatus for visual data processing, wherein the method comprises: acquiring at least one intermediate representation of the visual data in a neural network (NN)-based model; applying a first filter in the NN-based model to the at least one intermediate representation, wherein at least one parameter of the first filter is configured based on a format of output visual data from the transformation, the format indicating a relationship between the size of a first component of the output visual data and the size of a second component of the output visual data; and generating the bitstream using the NN-based model based on the application.

[0294] Item 28. A method for storing a bitstream of visual data, comprising: obtaining at least one intermediate representation of the visual data in a neural network (NN)-based model; applying a first filter in the NN-based model to the at least one intermediate representation, wherein at least one parameter of the first filter is configured based on a format of output visual data from the transformation, the format indicating a relationship between the size of a first component of the output visual data and the size of a second component of the output visual data; generating the bitstream using the NN-based model based on the application; and storing the bitstream in a non-transitory computer-readable recording medium.

[0295] Example device Figure 20 A block diagram of a computing device 2000 in which various embodiments of the present disclosure may be implemented is shown. The computing device 2000 may be implemented as a source device 110 (or visual data encoder 114) or a target device 120 (or visual data decoder 124), or may be included in the source device 110 (or visual data encoder 114) or the target device 120 (or visual data decoder 124).

[0296] It should be understood that, Figure 20 The computing device 2000 shown is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of this disclosure.

[0297] like Figure 20As shown, computing device 2000 includes general-purpose computing device 2000. Computing device 2000 may include at least one or more processors or processing units 2010, memory 2020, storage unit 2030, one or more communication units 2040, one or more input devices 2050, and one or more output devices 2060.

[0298] In some embodiments, the computing device 2000 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia tablet computers, Internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, and includes accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 2000 can support any type of interface to the user (such as "wearable" circuitry devices, etc.).

[0299] Processing unit 2010 can be a physical processor or a virtual processor, and can perform various processes based on programs stored in memory 2020. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of computing device 2000. Processing unit 2010 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0300] Computing device 2000 typically includes various computer storage media. Such media can be any media accessible by computing device 2000, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 2020 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage cell 2030 can be any removable or non-removable media and may include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or visual data and can be accessed within computing device 2000.

[0301] The computing device 2000 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Although in Figure 20 Not shown, but a disk drive for reading from and / or writing to a removable non-volatile disk, and an optical disc drive for reading from and / or writing to a removable non-volatile optical disc may be provided. In this case, each drive may be connected to a bus (not shown) via one or more visual data media interfaces.

[0302] The communication unit 2040 communicates with another computing device via a communication medium. Furthermore, the functionality of the components in the computing device 2000 can be implemented by a single computing cluster or by multiple computing machines communicating via communication connections. Therefore, the computing device 2000 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0303] Input device 2050 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 2060 can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 2040, computing device 2000 can also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 2000 can also communicate with one or more devices that enable a user to interact with computing device 2000, or, if needed, with any device that enables computing device 2000 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via an input / output (I / O) interface (not shown).

[0304] In some embodiments, some or all components of computing device 2000 may be deployed in a cloud computing architecture, rather than integrated into a single device. In a cloud computing architecture, components may be provided remotely and work together to achieve the functionality described herein. In some embodiments, cloud computing provides computing, software, visual data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services via a wide area network (WAN), such as the Internet, using suitable protocols. For example, a cloud computing provider provides applications via a WAN that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding visual data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed across remote visual data center locations. Cloud computing infrastructure may provide services through a shared visual data center, although to the user they appear as a single access point. Therefore, a cloud computing architecture can be used to provide the components and functionality described herein from a service provider at a remote location. Alternatively, the components and functionality described herein may be provided by conventional servers or installed directly or otherwise on client devices.

[0305] The computing device 2000 can be used to implement visual data encoding / decoding in embodiments of this disclosure. The memory 2020 may include one or more visual data encoding / decoding modules 2025 having one or more program instructions. These modules are accessible and executable by the processing unit 2010 to perform the functions of the various embodiments described herein.

[0306] In an example embodiment of performing visual data encoding, input device 2050 may receive visual data as input 2070 to be encoded. The visual data may be processed, for example, by visual data encoding / decoding module 2025 to generate an encoded bitstream. The encoded bitstream may be provided as output 2080 via output device 2060.

[0307] In an example embodiment of performing visual data decoding, input device 2050 may receive an encoded bitstream as input 2070. The encoded bitstream may be processed, for example, by a visual data encoding / decoding module 2025 to generate decoded visual data. The decoded visual data may be provided as output 2080 via output device 2060.

[0308] While this disclosure has been specifically shown and described with reference to preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application as defined by the appended claims. These variations are intended to be covered by the scope of this application. Therefore, the foregoing description of embodiments of this application is not intended to be limiting.

Claims

1. A method for visual data processing, comprising: For the transformation between visual data and the bitstream of the visual data using a neural network (NN)-based model, at least one intermediate representation of the visual data in the NN-based model is obtained; A first filter in the NN-based model is applied to the at least one intermediate representation, and at least one parameter of the first filter is configured based on the format of the output visual data from the transformation, the format indicating the relationship between the size of a first component of the output visual data and the size of a second component of the output visual data; as well as The conversion is performed based on the application.

2. The method of claim 1, wherein in the first filter, a first intermediate result is acquired using a downsampling process having a first intermediate representation of the at least one intermediate representation as input, and the at least one parameter includes at least one downsampling ratio for the downsampling process.

3. The method of claim 2, wherein the first intermediate representation is associated with the first component.

4. The method according to any one of claims 2 to 3, wherein the at least one downsampling ratio comprises at least one of the following: The first downsampling ratio in the vertical direction, or The second downsampling ratio in the horizontal direction.

5. The method of claim 4, wherein the relationship includes a first ratio between the vertical dimension of the first component of the output visual data and the vertical dimension of the second component of the output visual data, and the first downsampling ratio is determined based on the first ratio.

6. The method according to any one of claims 4 to 5, wherein the relationship includes a second ratio between the horizontal dimension of the first component of the output visual data and the horizontal dimension of the second component of the output visual data, and the second downsampling ratio is determined based on the second ratio.

7. The method according to any one of claims 4 to 6, wherein if the format is a 4:4:4 format, then the first downsampling ratio is equal to 1, and the second downsampling ratio is equal to 1, or If the format is 4:2:2, then one of the first downsampling ratio and the second downsampling ratio is equal to 1, and the other downsampling ratio is equal to 2, or If the format is 4:2:0, then the first downsampling ratio is equal to 2, and the second downsampling ratio is equal to 2.

8. The method according to any one of claims 2 to 7, wherein the second intermediate result is generated by applying the first sub-network of the first filter to the second intermediate representation and the first intermediate result in the at least one intermediate representation.

9. The method of claim 8, wherein the first sub-network comprises at least one of the following: splicing operation, Convolutional layer, or Corrected linear unit (ReLU).

10. The method according to any one of claims 8 to 9, wherein the second intermediate representation is associated with the second component, and the second intermediate representation is the output of the inter-channel related information (ICCI) filter in the NN-based filter.

11. The method according to any one of claims 8 to 10, wherein the output of the first filter is generated by applying a second sub-network of the first filter to a third intermediate representation and the second intermediate result in the at least one intermediate representation.

12. The method of claim 11, wherein the second sub-network comprises at least one of the following: splicing operation, or Convolutional layer.

13. The method according to any one of claims 11 to 12, wherein the third intermediate representation is associated with the second component, and the third intermediate representation is the output of the adaptive filter in the NN-based filter.

14. The method according to any one of claims 2 to 13, wherein the downsampling process includes a nearest neighbor downsampling process.

15. The method according to any one of claims 1 to 14, wherein the first component comprises one of: a principal component, a luminance component, or a Y component, and The second component includes one of the following: principal component, chromaticity component, U component, or V component.

16. The method according to any one of claims 1 to 15, wherein the format is indicated by at least one indication in the bit stream.

17. The method according to any one of claims 1 to 16, wherein the format is permitted to be one of the following: 4:4:4 format, 4:2:0 format, or 4:2:2 format.

18. The method according to any one of claims 1 to 17, wherein the first filter comprises a nonlinear filter.

19. The method according to any one of claims 1 to 18, wherein the at least one intermediate representation of the visual data is obtained by processing the output of the synthetic transformation in the NN-based model.

20. The method according to any one of claims 1 to 18, wherein the at least one intermediate representation of the visual data is obtained by processing the latent representation of the visual data using a synthetic transformation in the NN-based model.

21. The method according to any one of claims 2 to 20, wherein the downsampling process includes an average pooling operation.

22. The method according to any one of claims 1 to 21, wherein the visual data includes video, a picture of the video, or an image.

23. The method according to any one of claims 1 to 22, wherein the conversion comprises encoding the visual data into the bitstream.

24. The method according to any one of claims 1 to 22, wherein the conversion comprises decoding the visual data from the bitstream.

25. An apparatus for visual data processing, comprising a processor and a nontransitory memory, the nontransitory memory having instructions, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 24.

26. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method according to any one of claims 1 to 24.

27. A non-transitory computer-readable recording medium storing a bitstream generated by a method performed by means of visual data processing, wherein the method includes: Obtain at least one intermediate representation of the visual data in a neural network (NN)-based model; A first filter in the NN-based model is applied to the at least one intermediate representation, and at least one parameter of the first filter is configured based on the format of the output visual data from the transformation, the format indicating the relationship between the size of a first component of the output visual data and the size of a second component of the output visual data; as well as The bitstream is generated using the NN-based model in the application described above.

28. A method for storing a bitstream of visual data, comprising: Obtain at least one intermediate representation of the visual data in a neural network (NN)-based model; A first filter in the NN-based model is applied to the at least one intermediate representation, and at least one parameter of the first filter is configured based on the format of the output visual data from the transformation, the format indicating the relationship between the size of a first component of the output visual data and the size of a second component of the output visual data; The bitstream is generated using the NN-based model based on the application. as well as The bitstream is stored in a non-transitory computer-readable recording medium.

Images