Method or apparatus for generating film grain parameters, method or apparatus for generating blocks of pixels having a film grain pattern - Patents.com

Abstract

At least one method and apparatus for efficiently handling film grain while encoding or decoding video is presented. For example, the method includes receiving film grain information including at least one parameter specifying an attribute of film grain associated with an image block, applying a transform to a block of random values, filtering the transformed block, the filtering being defined by at least one parameter in the received film grain information, and applying a respective inverse transform to the filtered transformed block to generate a block of pixels having the film grain pattern of the image block. The transform may be a core transform of one of the video coding standards, e.g., DCT-II, DCT-VIII, DST-VII.

Description

[Technical field]

[0001] At least one of the embodiments relates generally to a method or apparatus for film grain estimation and film grain synthesis in video coding, video distribution and video rendering, and more specifically to a method or apparatus for generating a pixel block having a film grain pattern for an image block. [Background technology]

[0002] Film grain is often a desired feature in video production, by creating a natural look and helping to express creative intent. However, film grain is not sufficiently compressible by modern video compression standards such as Versatile Video Coding (VVC), known as ITU-T H.266 and ISO / IEC 23090-3. In fact, during various filtering and lossy compression stages, film grain is suppressed without being able to be reconstructed. However, information about film grain can be communicated as metadata, for example through the SEI message specified by Versatile Supplemental Enhancement Information (VSEI, also known as ITU-T Recommendation H.274 and ISO / IEC 23002-7). Thus, film grain is often modeled and removed before compression, and resynthesized at the decoder side with the aid of appropriate metadata. Moreover, film grain can also be used as a tool to mask coding artifacts resulting from compression. Various approaches have been investigated regarding film grain modeling. In the context of VVC, frequency filtering solutions can be used to parameterize and resynthesize film grain.

[0003] Existing methods for film grain modeling exhibit several limitations in the design complexity of film grain modeling, and therefore there is a need to improve this situation. Summary of the Invention

[0004] SUMMARY OF THE DISCLOSURE The general aspects described herein overcome and address the shortcomings and drawbacks of the prior art.

[0005] According to a first aspect, there is provided a method comprising receiving film grain information including at least one parameter specifying an attribute of film grain associated with an image block, applying a transform to a block of random values, filtering the transformed block of random values, the filtering being defined by at least one parameter in the received film grain information, and applying a respective inverse transform to the filtered transformed block to generate a block of pixels having the film grain pattern of the image block. Conveniently, the transform is a standardized transform, for example any of DCT-II, DCT-VIII, DST-VII, corresponding to the VVC core transform.

[0006] According to another aspect, a second method is provided that includes receiving a film grain block representing a film grain estimate in an image block, applying a transform to the film grain block, and generating at least one parameter from the transformed film grain block that specifies an attribute of film grain associated with the image block. The transform is conveniently a standardized transform, e.g., DCT-II, DCT-VIII, or DST-VII, corresponding to the VVC core transform.

[0007] According to another aspect, an apparatus is provided, the apparatus including one or more processors configured to implement the method for generating a block of pixels having a film grain pattern according to any of its variants. According to another aspect, an apparatus for generating a block of pixels having a film grain pattern for an image block includes means for receiving film grain information including at least one parameter specifying an attribute of a film grain associated with an image block, means for applying a transformation to a block of random values, means for filtering the transformed block of random values, the filtering being defined by at least one parameter in the received film grain information, and means for applying a respective inverse transform to the filtered transformed block to generate a block of pixels having the film grain pattern for the image block. Conveniently, the means for applying a transform implement a standardized transform, for example one of DCT-II, DCT-VIII, DST-VII, corresponding to the VVC core transform.

[0008] According to another aspect, another apparatus is provided, the apparatus including one or more processors configured to implement the method for generating at least one parameter specifying an attribute of film grain associated with an image block according to any of its variants. According to another aspect, the apparatus for generating film grain parameters includes means for receiving a film grain block representing a film grain estimation in an image block, applying a transform to the film grain block, and generating at least one parameter from the transformed film grain block specifying an attribute of film grain associated with the image block. The means for applying the transform may conveniently implement a standardized transform, e.g., DCT-II, DCT-VIII, DST-VII, corresponding to the VVC core transform.

[0009] According to another general aspect of at least one embodiment, there is provided a device including an apparatus according to any of the embodiments relating to decoding and at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to limit the received signal to a band of frequencies including the video block; or (iii) a display configured to display an output representing the video block.

[0010] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment there is provided a signal including video data generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0013] According to another general form of at least one embodiment, a computer program product is provided that includes instructions that, when executed by a computer, cause the computer to perform any of the described encoding / decoding embodiments or variations.

[0014] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. [Brief description of the drawings]

[0015] In the drawings, some example embodiments are illustrated. [Figure 1]FIG. 1 is a simplified block diagram of the use of film grain in a video coding / decoding framework. [Diagram 2] 1 is a simplified block diagram of a method for generating blocks of a film grain pattern in accordance with a general aspect of at least one embodiment. [Diagram 3] FIG. 1 is a modified block diagram for using film grain in a video coding / decoding framework, in accordance with a general aspect of at least one embodiment. [Figure 4a] FIG. 2 is a modified block diagram of a method for generating blocks of a film grain pattern in accordance with a general aspect of at least one embodiment. [Figure 4b] 1 is a modified block diagram of a method for generating film grain parameters in accordance with a general aspect of at least one embodiment. [Diagram 5] FIG. 13 is a block diagram of film grain modification by adjusting film grain parameters at the decoder side. [Figure 6] FIG. 2 is a block diagram of an embodiment of a video encoder in which various aspects of the embodiments may be implemented. [Figure 7] FIG. 2 is a block diagram of an embodiment of a video decoder in which various aspects of the embodiments may be implemented. [Figure 8] FIG. 1 is a block diagram of an example device in which various aspects of the embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] It should be understood that the drawings and description are simplified to illustrate elements relevant to a clear understanding of the present principles, and for purposes of clarity, exclude many other elements that may be found in a typical encoding and / or decoding device. Although terms such as first and second may be used herein to describe various elements, it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another.

[0017] Regarding image encoding / decoding, various embodiments are described, which may be applied to encode / decode portions of an image, such as slices or tiles, groups of tiles, or entire sequences of images.

[0018] Various methods have been described above, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0019] At least some embodiments relate to a method for generating a block of pixels having film grain, the transformation used during the generation being one of DCT-II, DCT-VIII, DST-VII. The method for generating a block of pixels having film grain is implemented, for example, in a video decoding scheme. Furthermore, at least some embodiments relate to a method for estimating and generating a film grain parameter, the transformation used during the generation being one of DCT-II, DCT-VIII, DST-VII. The method for generating a film grain parameter is implemented, for example, in a video encoding scheme.

[0020] FIG. 1 shows a simplified block diagram of the use of film grain in a video coding / decoding framework. Film grain is a pleasing noise that enhances the natural appearance of video content. It is generated during the physical process of exposing and developing photographic film. However, digital sensors do not undergo such a process and are therefore free of film grain. This produces a noise-free digital image whose flawlessness, crisp and prominent edges, as well as monotonous areas, can worsen the subjective experience of the viewer. Thus, re-noising the image can improve the visual experience and is often used by content producers before distributing their content. This has been embraced, among others, by the movie industry, where many producers have begun to utilize techniques to add film grain to video content to add texture and warmth to the image or to create a sense of nostalgia (e.g., to depict an earlier era as required by the narrative). Moreover, film grain can be used to mask compression artifacts even when not present in the source video.

[0021] Compression is a necessary step to support the growing demand for new content delivery to end users, which requires increasing image resolution and quality after playback and delivers huge amounts of data. This places a heavy burden on current networks. It should be noted that before delivery, the video is therefore usually subjected to various pre-processing steps, which inevitably lead to video compression. However, during the various steps of filtering and lossy compression, the film grain is suppressed without being able to be reconstructed. One way to alleviate this problem is to use a lower quantization parameter (Qp) to better preserve fine details such as film grain. However, this can increase the bitrate significantly. Another solution is to model the film grain before compression, which can then be resynthesized later at the decoder side.

[0022] Therefore, film grain should be preserved during coding since it is considered as a desirable noise. This is not an easy task since film grain is known to have a high level at high frequencies (e.g., DCT domain) and is usually suppressed by the quantization process. In order to preserve the appearance of film grain and at the same time improve the coding efficiency, a parameterized model is used to resynthesize the film grain. Furthermore, film grain is filtered out during pre-processing steps and / or suppressed in compression. Therefore, it is more efficient to use a parameterized film grain model to quickly pre-define or estimate its parameters, remove it by various pre-processing steps and / or during compression, and synthesize it back into the video content after decompression. In this way, the film grain parameters are transmitted to the user side (decoder) via appropriate metadata, e.g., SEI message.

[0023] The final benefits of film grain modeling are: 1) The final bitrate can be lowered since film grain does not need to be strictly preserved after compression. 2) Since film grain is temporally uncorrelated, the final bitrate can be improved if it is filtered before compression, which allows for improved prediction. 3) Film grain can be modeled to be the same as in the original content, resulting in higher visual quality in the reconstructed data (film grain is suppressed even at low Qp values ​​during compression). 4) Film grain can improve visual quality and can mask compression artifacts even if they are not present in the original content.

[0024] In general, film grain modeling and synthesis for video coding consists of two parts, one located on the encoder side and the other on the decoder side. These two parts are noise removal and parameterization at the encoder and noise synthesis at the decoder side according to the received metadata. One promising model for film grain parameterization and synthesis is presented in "Film Grain Technology-Specifications for H.264|MPEG-4 AVC Bitstreams.", also known as SMPTE-RDD5, by Joan Llach. Note that this describes a bit-accurate film grain model for adding film grain to decoded frames (and thus defines the film grain synthesis method). Nevertheless, conclusions regarding the encoder / parameter estimation side can be implicitly derived. A simplified block diagram of the whole process is shown in Fig. 1. First, a pre-processing step 100 is applied to the input video. The filtered video is processed through film grain estimation 101 using a specific internal transform implementation. This step produces film grain (FG) parameters. In step 102, the video is encoded and the FG parameters are inserted into the FG SEI message. In step 103, the decoder decodes the bitstream as well as the FG SEI message. This produces the decoded video, which can be further enhanced by the FG synthesis process in step 104. The transforms used during the FG estimation and synthesis process are generally specific and different from the core transforms used in the encoding and decoding processes. Note that, if necessary, steps 100 and 101 can be skipped and replaced by a fixed set of manually tuned parameters.

[0025] This disclosure uses a standardized core transform to adhere to the presented model while improving its computational efficiency. Several transforms are part of modern video coding standards, corresponding to variations of the discrete cosine transform (DCT) and discrete sine transform (DST), so any of them can be used in the proposed solution. It is beneficial to use such transforms, as they are efficient by design, standardized for such purposes. Also, since they are part of widely used video compression standards, many efficient software and hardware implementations have been presented and are often implemented in standard consumer devices. It is also noted that SMPTE-RDD5 is only one of the possible implementations of frequency filtering methods for film grain. However, none of them focuses on a specific DCT implementation.

[0026] Thus, the model is based on filtering in the frequency / transform domain. With this model, the film grain pattern is modeled in the frequency domain by quickly setting or estimating a cutoff frequency that defines a low-pass filter. Some variations support band-pass filtering, but such an approach is not supported by the current SEI design. It is important to note that the VVC SEI specification only provides a syntax to transmit the parameters of the model, but does not provide a way to estimate these parameters or to synthesize the film grain. The work on SMPTE-RDD5 allows a closer look at the synthesis part. This is defined for the H.264 standard, but no modifications are needed for VVC or HEVC, since both support the same metadata. Only minor modifications are needed to support bit depths higher than 8 bits.

[0027] Therefore, to simulate a desired film grain pattern, two parameters are set or estimated and communicated to the synthesis unit, for example, via an appropriate SEI message. These parameters represent the horizontal high cutoff frequency (denoted as Horizontal_Cutoff) as well as the vertical high cutoff frequency (denoted as Vertical_Cutoff), thus defining the film grain pattern. Thus, according to a frequency filtering model, different pairs of cutoff frequencies are used to synthesize each film grain pattern. If no parameters are transmitted via the SEI message, default parameters can be agreed upon for the decoded frame once the synthesis unit is enabled.

[0028] 2 shows a simplified block diagram of a method for generating a block of a film grain pattern according to a general aspect of at least one embodiment. It starts in step 200 by defining an N×M block of pseudo-random numbers following a Gaussian distribution. To obtain the block of pseudo-random numbers, any Gaussian random number generator already proven in the literature can be utilized. The block of pseudo-random numbers can be obtained quickly or can be predefined, for example during an initialization step, and stored for further use. Then, the film grain pattern is simulated as follows:

[0029] A block b of N × M pseudorandom values ​​generated with a normalized Gaussian distribution N(0,1) is processed with low-pass filtering performed in the frequency domain by:

[0030] [Table 1]

[0031] Hereby, b' represents a film grain image (or block). Note that N and M can take any value, but in practice, N=M and a size of 64x64 are shown to be standardly used. The previous example is illustrated with a specific transform implementation based on DCT, but other transforms, e.g. Fast Fourier Transform, can also be used. Each block b' then represents an NxM film grain image that is used to add grain to the decoded frames. The different film grain patterns (for different cutoff pairs) can be pre-computed by creating a database of available film grain patterns, or they can be calculated quickly when each decoded frame is ready for processing. For example, SMPTE-RDD5 specifies a 64x64 integer inverse transform that is used to create a database of different film grain patterns. It proposes to use a LUT of Gaussian pseudorandom numbers after the transform, which is pre-computed during an initialization step and stored for further use. Finally, after obtaining the NxM film grain image, additional operations such as scaling, deblocking, etc. can be applied, as stated in SMPTE-RDD5. The same is true for all color components.

[0032] On the encoder side, the cut-off frequency can be estimated quickly from the real data (including blocks 100 and 101). Although this is not a required step, if one wants to obtain the appearance of the original film grain, it is desirable to estimate it accurately rather than using default parameters defined a priori. In such a case, a denoising is first performed to estimate the grain parameters rigorously in flat areas of the frame based on the difference between the original frame and the noise-free (filtered) frame. The denoising / filtering can utilize any algorithm that allows reducing the noise in the processed frame. Instead of performing filtering, the reconstructed image can be used. However, in such a case, additional artifacts resulting from compression may interfere with the estimation process. In any case, this results in the film grain pattern. For example, the film grain pattern in such a case is an N×M residual block obtained by subtracting the original frame from the filtered frame, obtained in flat image areas since edges and complex textures may lead to erroneous estimation. The film grain pattern is then input to a transformation process, e.g., DCT, to receive a set of transformation coefficients. By analyzing the resulting set of transformed coefficients, cut-off frequencies can be estimated that sufficiently describe the film grain pattern. These cut-off frequencies are embedded in the bitstream via the SEI message and are used at the decoder side to simulate film grain, as previously described, for example in SMPTE-RDD5. Note that to estimate the cut-off frequencies, each block representing a noise image (the difference between the filtered and original frames) is subject to transformation. The transformation process using a custom integer approximation of the DCT introduces additional computations at the encoder side during the pre-processing step and represents an additional computational burden on the encoder. It is therefore crucial to introduce computational reductions whenever possible. This problem becomes more acute when considering the implementation and hardware design of the decoder.

[0033] This is solved and addressed by the general aspects described herein, which are directed to a method and device for generating blocks of pixels having a film grain pattern, where the transform used in the generation is one of DCT-II, DCT-VIII, DST-VII, which is also used as the VVC core transform. Corresponding methods for estimating film grain and generating film grain parameters during encoding are also disclosed, where the transform used in the estimation is one of DCT-II, DCT-VIII, DST-VII, i.e., the VVC core transform.

[0034] It is convenient if we note that the algorithm is suitable for any of several kinds of DCTs used as VVC core transforms, as well as DSTs. This allows to obtain as well as estimate film grain patterns in an efficient way for various cutoff frequencies. Regarding efficient implementation, there are already many implementations and hardware designs proposed for the core transforms of VVC. By using a standardized transform for film grain, the exact same implementation will be used by many capable devices (or vendors), and can perform accurate film grain estimation and synthesis without the fear of doing something differently. In this way, interoperability between various devices is ensured, which may be especially important if film grain becomes a mandatory element in video coding standards.

[0035] In addition to using the core transforms already implemented in the encoder / decoder, we also eliminate the need to implement yet another transform (DCT). We also optimized these three core transforms. The design is efficient: they are already implemented (in software and / or hardware as part of the encoder / decoder), and implementing another one (one that is not as well designed as these three) would only add complexity, and would cost extra, especially if it had to be designed in hardware.

[0036] FIG. 3 illustrates a modified block diagram for using film grain in a video coding / decoding framework in accordance with a general aspect of at least one embodiment.

[0037] The principles of the present invention disclose the use of standardized transforms on both sides (film grain estimation and synthesis). According to one embodiment, the standardized transforms can be used at the encoder side to generate information about film grain (if present in the source video) as well as when on-the-fly parameter estimation is required. According to another embodiment, a fixed set of manually adjusted parameters is used, in which case no transform is used for film grain at the encoder side. The encoding step 102 of FIG. 3 partially represents an encoder, or a module of an encoding method, implemented, for example, in the exemplary encoder of FIG. 6. Then, film grain information, including at least one parameter specifying the attributes of the film grain appearing in the block, is embedded as metadata in the bitstream. At the decoder side, the bitstream and the film grain information are decoded to simulate the film grain according to the received metadata. The decoding step 103 of FIG. 3 partially represents a decoder, or a module of a decoding method, implemented, for example, in the exemplary decoder of FIG. 7. The implementation in these film grain situations is straightforward, since the encoder and decoder already use standardized transforms. Therefore, bit-precise transform coefficients are obtained on any supported codec. The transformed coefficients are then analyzed to obtain film grain parameters in step 301, or filtered to obtain film grain image / pattern in step 304. The film grain estimation step 301 and film grain synthesis step 304 are modified versions of steps 101 and 103, respectively, which replace the internal transform by the core transform defined in the video codec.

[0038] Fig. 4a shows a modified block diagram of a method for generating a block of a film grain pattern according to a general aspect of at least one embodiment. In a preliminary step not shown in Fig. 4a, film grain information is received, the film grain information including at least one parameter specifying the attributes of the film grain occurring in the block. In a decoder, the film grain information includes receiving and decoding a supplemental enhancement information message including the at least one parameter. A transform corresponding to one of DCT-II, DCT-VII or DST-VII is then applied to the block of random values ​​to obtain a set of transform coefficients in the frequency domain, also referred to as transform block of random values. According to a non-limiting variant, in step 200, a block of random values ​​selected from a list of Gaussian random numbers is generated. According to a non-limiting variant, the size of the block of pixels is NxM, N being an integer in the range [2 to 64] and M being an integer in the range [2 to 64]. According to a non-limiting variant, N or M is an integer greater than 64 and the block size is scaled to fit the core transform size. As mentioned above, any Gaussian random number generator already established in the literature can be used to obtain the block of pseudo-random numbers. The block of pseudo-random numbers can be obtained quickly or can be predefined, for example during an initialization step, and stored for further use. Then, in step 401, a core transform is applied to the block of random values, the core transform being one of DCT-II, DCT-VIII, DST-VII. This transform results in a set of transform coefficients in the frequency domain. As for step 200, the set of transform coefficients can be obtained quickly or can be predefined and stored for further use. Then, in 202, the coefficients of the transformed block are filtered with a low-pass filter. As mentioned above, the filtering is defined by at least one parameter in the received film grain information, which represents a cut-off frequency for each of the vertical and horizontal edges.In step 403, an inverse core transform is applied to the set of filtered coefficients for the block to generate a block of pixels having the film grain pattern. The inverse core transform is an inverse transform corresponding to the transform used in step 401, i.e., one of DCT-II, DCT-VIII, DST-VII, i.e., the standardized core transform.

[0039] In terms of supported transform implementations, VVC implements several kinds of transforms of variable size, known as the Multiple Transform Selection (MTS) tool, which was first introduced in VVC. It supports: DCT-II, block sizes from 2x2 to 64x64, square and non-square. DCT-VIII, block size from 4x4 to 32x32, square and non-square. DST-VII, block sizes from 4x4 to 32x32, square and non-square.

[0040] Note that the above transforms are integer approximations of the original floating-point transforms, and are designed to support efficient implementation in hardware and software.

[0041] Therefore, the process of establishing a set of transformed coefficients for film grain modeling purposes can occur in a variety of ways. Any combination of the above-mentioned transforms and block sizes (including non-square) is envisioned. Below, we describe some envisioned embodiments, but also show that there can be other embodiments based on the use of one (or a combination of multiple) of the available transforms and available transform sizes.

[0042] FIG. 4b shows a modified block diagram of a method for estimating film grain parameters and generating at least one parameter specifying film grain attributes associated with an image block to be used in a decoder, according to a general aspect of at least one embodiment. In a preliminary step 404, a film grain block is received. In the encoder, the reception of the film grain block is the result of a pre-processing step. As mentioned above, to obtain the film grain block, the pre-processing step includes filtering of the original frame and deriving a mask. The mask is used to indicate flat regions of the frame. To derive the mask, any method known in the art that allows complex textures and edges to be detected can be used. The mask excludes these non-flat regions when selecting the block from which the film grain parameters are derived. The film grain block is then represented as an NxM residual signal taken from the flat regions of the frame indicated by the mask, and calculated as the difference between the original / input block and the filtered block. Such a block represents the film grain estimate that is subject to the core transform 401. Next, at 405, the transformed block is analyzed, after which at least one film grain parameter is calculated and communicated to the decoder side.

[0043] Various embodiments of general film grain pattern generation and estimation methods are described below. For film grain, one of the standardized DCT or DST transforms presented can be used. In VVC, the transforms are DCT-II going up to 64x64, and DCT-VIII and DST-VII going up to 32x32. Various transform sizes can be used, but it has been established that a square NxM transform with N=M gives the best performance, but is not required (non-square transforms are supported in VVC as well and can be used instead of the square transform for film grain modeling). N (and M) can be any of the supported sizes mentioned above, but N,M=64 and N,M=32 provide a good tradeoff between complexity and performance.

[0044] According to a particular embodiment, the same transform type and the same transform size are used for the luma and chroma components.

[0045] In an alternative embodiment, the block size for estimating / generating the luma and chroma FG patterns is set to 64x64 and the DCT-II defined in the VVC specification is used.

[0046] In a further variant, the block size for estimating / generating the luma and chroma FG patterns is set to 32x32 and the DCT-II defined in the VVC specification is used.

[0047] In a further variant, the luma and chroma components use the DCT-II defined in the VVC specification, with a transform size less than 32x32.

[0048] In a further variant, DCT-VIII or DST-VII as specified in the VVC specification can be used instead of DCT-II to estimate / generate FG patterns for luma and chroma components using any supported block size, for example using DCT-VIII or DST-VII of size 32x32.

[0049] In further variant embodiments, sizes smaller than 32x32 are used to estimate / generate FG patterns for luma and chroma components, in which case the embodiment may use DCT-II, DCT-VIII, or DCT-VII.

[0050] In all the above variants, the transform type and transform size are known a priori and no further signaling is required: the same transform type and transform size are used for both luma and chroma components.

[0051] According to another particular embodiment, the same transform type but different transform sizes are used for the luma and chroma components.

[0052] In a further variant embodiment, for example, when the chroma format is 4:2:0 (but not strictly limited to this), the block size for generating the FG pattern for luma is set to 64x64, and the block size for generating the FG pattern for chroma is set to 32x32, and the DCT-II defined in the VVC specification is used. Therefore, different transform sizes are used for the luma and chroma components. Similarly, for example, we use the DST-VII, and further use a block size of 32x32 for luma and a block size of 16x16 for chroma. Here, as in the previous embodiment, the transform type is the same for luma and chroma, but the size is different. However, the block size is preset and there is no need for signaling. Therefore, any combination of the DCT-II and various transform sizes for the luma and chroma components can be used. The same can be said for other transform types, for example, the DCT-VIII or DST-VII. In this case, for example, we use a block size of 32x32 for luma and a block size of 16x16 for chroma, or any other size combination for luma and chroma.

[0053] According to a particular embodiment, the same transform size is used for the luma and chroma components, although the transform types are different.

[0054] In one variant, the block size for generating the FG pattern is 32x32. In this case, DCT-II, DCT-VIII or DST-VII, as defined in VVC, can be used. In this variant, the type of transformation is different for luma and chroma. In another variant, the size is smaller than 32x32. In this case, as before, the size and transformation are preset and no further signaling is required. Only this time, the type of transformation is different for luma and chroma components. For example, for FG luma blocks, DCT-II is used and for FG chroma blocks, DCT-VIII is used. The reason for using different transformations is that luma and chroma noise signals may present significantly different statistics.

[0055] According to a particular embodiment, different transform types and different transform sizes are used for the luma and chroma components.

[0056] In an alternative embodiment, both the transform size and the transform type are different for the luma and chroma components, however the size and type are pre-set and do not need to be signaled.

[0057] According to certain embodiments, the type and size of the transformation are signaled or inferred instead of an a priori defined type and size.

[0058] As an extension of the previous embodiment, the transform to be used does not need to be specified in advance. Instead, the transform to be used in the film grain estimation is selected from among various transforms (by manual control or by content analysis) and signaled for synthesis. In another variant, the transform size used to generate and estimate the film grain is signaled. In another variant, both the transform size and the transform type are signaled. The signaling is the same for luma and chroma components or different (separated) for luma and chroma components. According to one exemplary embodiment, the film grain parameters are exactly the same for all color components, thus signaling only once. In such a case, additional metadata indicates the proposed design, for example a flag indicating that one set of parameters applies to all color components.

[0059] In another variation, the transform size and / or transform type are signaled for the luma component only, and the transform type and size for the chroma are derived implicitly. For example, a set of parameters for the chroma components is determined based on a set of parameters signaled for the luma component. For example, the same transform is used for the chroma as for the luma, which makes it derivable based on the parameters signaled for the luma component only, and the cutoff frequency is the cutoff frequency after downsampling the cutoff frequency of the luma.

[0060] Example syntax is provided in the table below to implement the various embodiments described above.

[0061] [Table 2]

[0062] In a further embodiment, the transform type and size are inferred for luma and chroma. For example, the most frequently used transform to encode / decode the current frame is further used for film grain purposes. A syntax element "transform_type_luma_inferred_flag" is also inserted to indicate whether the luma transform type is inferred or not. If it is not inferred, the luma transform type is signaled.

[0063] It should be noted that there may be many different variations on the signaling, explicitly or implicitly derived parameters, which cannot all be enumerated in this disclosure, but those skilled in the art will be able to see many other variations on this theme.

[0064] Thus, in one variant, the type of transformation used to estimate and / or generate the FG pattern is indicated by a syntax element. The same concept can also be applied for transformation size or non-square block patterns. Signaling can be at sequence level, I-period, GOP, per frame, per CTU or per CU, depending on the specific requirements and the specific implementation. The syntax element may be inserted in the SEI message, but also in the adaptation parameter set (APS), sequence parameter set (SPS) and / or picture parameter set (PPS).

[0065] Those skilled in the art can conclude that larger transform sizes and other types are also envisioned in the future. As video coding standards are constantly evolving, future standards may support different (larger) transform sizes and more diverse transform types. In that case, the new / additional sizes and types can be applied to estimate / generate film grain in the same manner as described in this disclosure.

[0066] According to a particular embodiment, the horizontal and vertical transform type as well as the size are adapted.

[0067] VVC also allows the use of different horizontal and vertical transform types, as well as rectangular transform blocks. In one embodiment, FG block patterns are generated using different horizontal and vertical transforms. In another embodiment, FG block patterns are generated using different horizontal and vertical dimensions (transform sizes). In one embodiment, the horizontal and vertical transform types are signaled. Similarly, the recommended horizontal and vertical sizes for the FG pattern blocks are signaled.

[0068] Even if only DCT-II is considered for chroma components and luma blocks of size larger than 32 defined by the VVC specification, there is no prohibition to use other types and sizes to estimate and synthesize film grain. Film grain can be done in a pre-processing / post-processing step, the idea is to reuse available transform implementations (hardware or software), in which case the limitations of the video coding standard in terms of block sizes and available types for different color components do not apply to film grain.

[0069] Moreover, the proposed method is compatible and can be used with any chroma subsampling format.

[0070] According to a particular embodiment, the FG parameters are scaled to the block size. According to another embodiment, further scaling is performed on the film grain parameters, e.g., cutoff frequency. For example, if the parameters are intended to be used on 64x64 blocks, e.g., estimated on 64x64 blocks, but used to create a 32x32 film grain pattern (synthesized by using 32x32 film grain blocks), the parameters are downscaled to match the block requirements. For example, the parameters are estimated on 64x64 blocks at the encoder side, but due to complexity or implementation principles, they need to be used to create a 32x32 film grain pattern at the decoder side. Similarly, if the parameters are set / estimated to be used on 32x32 blocks, but the decoder side uses them on 64x64 blocks, then upscaling of the parameters is performed.

[0071] 5 shows a block diagram of film grain correction by adjusting film grain parameters at the decoder side. Before FG synthesis, a new step of adjusting FG parameters is inserted (step 305).

[0072] According to a particular embodiment, the FG parameters are adapted to the transform type. Also, in one embodiment, if necessary, the FG can be estimated by using one transform type at the encoder side and synthesized at the decoder side by using a different type of transform. For example, it can be estimated using DCT-II and the FG pattern can be synthesized using a DCT-VIII transform type. In a particular embodiment, some additional adjustments can be made to support such an approach. One example can be resampling of the post-transform coefficients when creating the FG pattern before the inverse transform due to the changed transform. Another example can be adding an offset (positive or negative) to the post-signaling cutoff frequency, for example, to compensate for different transform types.

[0073] According to certain embodiments, core transforms from other video coding standards are used. Additionally, in one embodiment, instead of VVC transforms, HEVC or H.264 / AVC standardized transforms are used. In this case, some other restrictions in terms of transform size or transform type may be imposed according to the standard specification.

[0074] According to a particular embodiment, a different kind of filtering than low pass is used.

[0075] Even though the present disclosure describes the film grain pattern using low-pass filtering, those skilled in the art can conclude that different filtering can be used in some embodiments. For example, instead of low-pass filtering represented by two cutoff frequencies as mentioned above, band-pass filtering can be used. In that case, four different frequencies (vertical high cutoff, vertical low cutoff, horizontal high cutoff, and horizontal low cutoff frequencies) are used to define the film grain pattern. It can be concluded that other filtering can be used to generate the film grain pattern in the frequency domain.

[0076] Further embodiments and information In this application, various aspects are described, including tools, features, embodiments, models, methods, and the like. Many of these aspects are described with specificity and in what may often sound definitive terms to at least indicate their individual characteristics. However, this is for purposes of clarity of description and not to limit the applicability or scope of the aspects. In fact, all of the various aspects can be combined and substituted to provide further aspects. Moreover, these aspects may also be combined and substituted with aspects described in previous applications as well.

[0077] The aspects described and contemplated in this application can be implemented in many different forms. Figures 6, 7, and 8 below provide some embodiments, but other embodiments are contemplated, and discussion of Figures 6, 7, and 8 does not limit the scope of implementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0078] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably.

[0079] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. The order and / or use of specific steps and / or actions may be modified or combined, unless a specific order of steps or actions is required for proper operation of the method. In addition, terms such as "first", "second", etc. may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding". The use of such terms does not imply a modified ordering of operations, unless specifically required. Thus, in this example, the first decoding need not be performed before the second decoding, but may occur, for example, before, during, or during an overlapping time with the second decoding.

[0080] Various methods and other aspects described herein can be used to modify modules of the video encoder 600 and the video decoder 700, such as pre-encoding modules or post-processing decoding modules (601, 785) as seen in Figures 6 and 7. Furthermore, aspects of the present disclosure are not limited to VVC or HEVC, but can be applied to other standards and recommendations, such as existing or future developments, and extensions of any such standards and recommendations, including VVC and HEVC. Unless otherwise specified or technically excluded, aspects described herein can be used individually or in combination.

[0081] In this application, various numerical values ​​are used, such as the number of transforms, the number of transform levels, transform indexes, etc. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0082] 6 illustrates an example encoder 600. Variations of this encoder 600 are contemplated, but for clarity, the following describes the encoder 600 without describing all possible variations.

[0083] Before being encoded, a video sequence may undergo pre-encoding processing 601, such as applying a color transformation to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resistant to compression (e.g., using a histogram equalization of one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0084] In the encoder 600, a picture is coded by the encoder elements, as described below. The picture to be coded is divided into units, e.g., CUs, and processed in 602. Each unit is coded, e.g., using either intra mode or inter mode. If the unit is coded in intra mode, the unit performs intra prediction in 660. In inter mode, it performs motion estimation in 675 and motion compensation in 670. In 605, the encoder decides whether to use intra mode or inter mode to code the unit, and indicates the intra / inter decision, e.g., by a prediction mode flag. In 610, a prediction residual is calculated, e.g., by subtracting a prediction block from a source image block.

[0085] This prediction residual is then transformed at 625 and quantized at 630. The quantized transform coefficients, as well as the motion vectors and other syntax elements, are entropy coded at 645 to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is directly coded without applying a transform or quantization process.

[0086] The encoder decodes the coded block to provide a reference for further prediction. At 640, the quantized transform coefficients are dequantized and inverse transformed at 650 to decode the prediction residual. At 655, the decoded prediction residual is combined with the prediction block to reconstruct an image block. At 665, an in-loop filter is applied to the reconstructed picture to perform, for example, deblocking / sample adaptive offset (SAO) filtering to reduce coding artifacts. At 680, the filtered image is stored in a reference picture buffer.

[0087] Figure 7 shows a block diagram of a video decoder 700. In the decoder 700, the bitstream is decoded by decoder elements, as described below. In general, the video decoder 700 performs a decoding pass that is inverse to the encoding pass described in Figure 6. Additionally, the encoder 600 generally performs video decoding as part of the video data encoding.

[0088] In particular, the input of the decoder includes a video bitstream, which may be generated by the video encoder 600. First, the bitstream is entropy decoded at 730 to obtain transform coefficients, motion vectors, and other coded information. The picture partition information indicates how the picture is partitioned. Thus, at 735, the decoder can partition the picture according to the decoded picture partition information. At 740, the transform coefficients are inverse quantized and inverse transformed at 750 to decode the prediction residual. At 755, the decoded prediction residual and the prediction block are combined to reconstruct the image block. At 770, the prediction block can be obtained from intra prediction 760 or motion compensated prediction (i.e., inter prediction) 775. At 765, an in-loop filter is applied to the reconstructed image. At 780, the filtered image is stored in a reference picture buffer.

[0089] The decoded picture may further undergo post-decoding processing 785, such as an inverse color transformation (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in pre-encoding processing 601. The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0090] FIG. 8 is a block diagram of an example of a system in which various aspects and embodiments are implemented. System 800 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected appliances, and servers. The elements of system 800, alone or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 800 are distributed across multiple ICs and / or separate components. In various embodiments, system 800 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 800 is configured to implement one or more of the aspects described herein.

[0091] The system 800 includes at least one processor 810 configured to execute instructions loaded therein, for example to implement various aspects described herein. The processor 810 may include embedded memory, input / output interfaces, and various other circuitry known in the art. The system 800 includes at least one memory 820 (e.g., volatile and / or non-volatile memory devices). The system 800 includes a storage device 840, including non-volatile and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. The storage device 840 may include, by way of non-limiting example, an internal storage device, an attached storage device (including removable and non-removable storage devices), and / or a network-accessible storage device.

[0092] The system 800 includes an encoder / decoder module 830 configured to process data to provide encoded or decoded video, for example, the encoder / decoder module 830 may include its own processor and memory. The encoder / decoder module 830 is a module or modules that may be included in a device that performs encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 830 may be implemented as a separate element of the system 800 or may be incorporated within the processor 810 as a combination of hardware and software, as is known to those skilled in the art.

[0093] Program code to be loaded into the processor 810 or the encoder / decoder 830 to perform various aspects described herein may be stored in the storage device 840 and subsequently loaded into the memory 820 for execution by the processor 810. According to various embodiments, one or more of the processor 810, the memory 820, the storage device 840, and the encoder / decoder module 830 may store one or more of a variety of items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and computational logic.

[0094] In some embodiments, memory internal to the processor 810 and / or the encoder / decoder module 830 is used to store instructions and provide operating memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 810 or the encoder / decoder module 830) is used for one or more of these functions. The external memory may be the memory 820 and / or the storage device 840, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM is used as the operating memory for video coding and decoding operations such as MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2, or VVC (Versatile Video Coding, a new standard being developed by JVET (the Joint Video Experts Team)).

[0095] Inputs to the elements of system 800 can be provided through various input devices, shown in block 805. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals transmitted, for example, via wireless communication by a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high definition multimedia interface (HDMI) input terminal. Although not shown in FIG. 6, another example includes composite video.

[0096] In various embodiments, the input devices of block 805 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a band of frequencies), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select a signal frequency band, which may be referred to as a channel in a particular embodiment (for example), (iv) demodulating the down-converted band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various such functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near baseband frequency) or to baseband. In one embodiment for a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and refiltering to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0097] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 800 to other electronic devices via USB and / or HDMI connections. It should be appreciated that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or implemented within the processor 810, as desired. Similarly, aspects relating to the USB or HDMI interface processing may be implemented in a separate interface IC or within the processor 810, as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 810 and an encoder / decoder 830 operating in combination with memory and storage elements that process the data streams as desired for presentation on an output device.

[0098] The various elements of system 800 can be provided within an integrated housing in which the various elements can be interconnected and data can be transmitted between them using a suitable connection device 815, for example, an internal bus as is known in the art, including an Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0099] The system 800 includes a communication interface 850 that enables communication with other devices over a communication channel 890. The communication interface 850 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 890. The communication interface 850 may include, but is not limited to, a modem or a network card, and the communication channel 890 may be implemented within a wired and / or wireless medium, for example.

[0100] In various embodiments, data is streamed or otherwise provided to the system 800 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in such embodiments is received over a communication channel 890 and communication interface 850 adapted for Wi-Fi communication. Typically, the communication channel 890 in such embodiments is connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, streaming data is provided to the system 800 using a set-top box that delivers data via an HDMI connection of the input block 805. In yet other embodiments, streaming data is provided to the system 800 using an RF connection of the input block 805. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, e.g., a cellular network or a Bluetooth network.

[0101] The system 800 can provide output signals to various output devices, including a display 865, a speaker 875, and other peripheral devices 885. The display 865 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 865 may be for a television, a tablet, a laptop, a mobile phone, or other device. Additionally, the display 865 may be integrated with other components (e.g., in a smart phone) or separate (e.g., an external monitor for a laptop). In various examples of embodiments, the other peripheral devices 885 include one or more of a stand-alone digital video disc (or digital versatile disc) (for both terms, a DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 885 that provide functionality based on the output of the system 800. For example, a disc player performs the function of playing the output of the system 800.

[0102] In various embodiments, control signals are communicated between the system 800 and the display 865, speaker 875, or other peripheral device 885 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow inter-device control with or without user intervention. The output devices can be communicatively coupled to the system 800 via dedicated connections through the respective interfaces 865, 875, and 885. Alternatively, the output devices can be connected to the system 800 using a communication channel 890 via the communication interface 850. The display 865 and speaker 875 may be integrated as a single unit with other components of the system 800 in an electronic device such as, for example, a television. In various embodiments, the display interface 865 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0103] Alternatively, the display 865 and speakers 875 may be separate from one or more of the other components, for example if the RF portion of the input 805 is part of a separate set-top box. In various embodiments where the display 865 and speakers 875 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0104] The embodiments can be executed by the processor 810 or by computer software implemented by hardware or a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 820 can be of any type suitable for the technology environment and can be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memories. The processor 810 can be of any type suitable for the technology environment and can include, as non-limiting examples, one or more of a microprocessor, a general purpose computer, a special purpose computer, and a processor based on a multi-core architecture.

[0105] Various implementations include decoding. As used herein, "decoding" can encompass all or part of the processes performed on a received encoded sequence to generate a final output suitable for, e.g., a display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as, e.g., entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by the decoders of the various implementations described herein, including, e.g., inverse transform.

[0106] As a further example, in one embodiment, "decoding" refers to entropy decoding only, in another embodiment, "decoding" refers to differential decoding only, and in another embodiment, "decoding" refers to a combination of entropy and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a portion of the operations or to the broader decoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0107] Various implementations involve encoding. Similar to the above discussion of "decoding," "encoding" as used herein can encompass all or part of the processes performed on an input video sequence, e.g., to generate an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as, for example, partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processing also or alternatively includes processes performed by encoders in various implementations described herein, such as, for example, transforming image blocks to the frequency domain.

[0108] As a further example, in one embodiment, "encoding" refers to entropy encoding only, in another embodiment, "encoding" refers to differential encoding only, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a portion of the operations or to the broader encoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0109] It should be noted that syntax elements as used herein are descriptive terms, and therefore do not exclude the use of other syntax element names.

[0110] Where a figure is presented as a flow chart, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow chart of the corresponding method / process.

[0111] Various embodiments refer to rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is usually considered, often giving a computational complexity constraint. Rate-distortion optimization is usually formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on an extensive test of all encoding options, including all considered modes or coding parameter values, with a full evaluation of their coding costs and associated distortion of the reconstructed signal after coding and decoding. To reduce the encoding complexity, faster approaches may also be used, especially with the calculation of approximate distortion based on a prediction or prediction residual signal rather than on the reconstructed signal. A mixture of these two approaches may also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the coding cost and the associated distortion.

[0112] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus may be implemented in appropriate hardware, software, and firmware. The method may be implemented in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0113] Reference to "one embodiment" or "an embodiment" or "implementation" or "an implementation", as well as other variations thereof, means that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation", as well as other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0114] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0115] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0116] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves in some way, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0117] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed alternative (A), or selection of only the second listed alternative (B), or selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This may be expanded as many times as the number of items listed, as would be apparent to one of ordinary skill in the art and related arts.

[0118] Also, as used herein, the term "signaling" means to indicate something, in particular, to a corresponding decoder. For example, in a particular embodiment, an encoder signals a particular one of a plurality of parameters for a transform. Thus, in an embodiment, the same parameters are used at both the encoder and decoder sides. Thus, for example, an encoder can transmit a particular parameter to a decoder so that the decoder can use the same particular parameter (explicit signaling). In contrast, if the decoder already has the particular parameter as well as other parameters, a non-transmitting signaling (implicit signaling) can be used to simply allow the decoder to know and select the particular parameter. By avoiding the transmission of any actual function, bit savings are realized in various embodiments. It will be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. The above relates to the verb form of the word "signal", which may also be used as a noun in this specification.

[0119] This disclosure has described various information, such as syntax, that may be transmitted or stored. This information may be packaged or arranged in various manners, including, for example, common in video standards, such as placing the information in an SPS, PPS, NAL unit, header (e.g., a NAL unit header or slice header), or SEI message. Other manners are also available, including, for example, common in system-level or application-level standards, such as placing the information in: SDP (Session Description Protocol), a format for describing multimedia communication sessions for the purposes of session announcement and session invitation, e.g. as described in the RFCs and used in conjunction with RTP (Real-time Transport Protocol) transport; DASH MPD (Media Presentation Description) Descriptors, e.g. used in DASH and transmitted over HTTP, which are associated with a representation or a collection of representations to provide additional characteristics to the content representation; RTP header extensions, e.g. as used during RTP streaming, and / or ISO Base Media File Formats, such as those used in OMAF and in some specifications, which use boxes, object-oriented building blocks defined by a unique type identifier and a length, also known as "atoms".

[0120] As will be apparent to one skilled in the art, implementations can result in a variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. For example, such a signal can be formatted as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries can be, for example, analog information or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored in a processor-readable medium.

[0121] Several embodiments are described. The features of these embodiments may be provided alone or in any combination across various claim categories and types. Furthermore, the embodiments may include one or more of the following features, devices, or aspects, alone or in any combination across various claim categories and types. Adapting the film grain process in the decoder and / or encoder. Selecting from among the core transforms which transforms to apply in the film grain simulation process in the decoder and / or in the film grain estimation process in the encoder. Signaling information about the film grain process to be applied at the decoder. Deriving information about an applied film grain process from the film grain information, this derivation being applied in a decoder and / or an encoder. · Insertion of signaling syntax elements that allow the decoder to identify the film grain process used, such as transform type, transform size, etc. Selecting at least one transform type and transform size process to apply in the decoder based on these syntax elements. A bitstream or signal comprising one or more of the syntax elements described, or variations thereof. A bitstream or signal including syntax carrying information generated according to any of the described embodiments. · Inserting film grain into signaling syntax elements that allow the decoder to process it in a manner that corresponds to that used by the encoder. Creating, transmitting, receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements, or variations thereof. · Producing and / or transmitting and / or receiving and / or decoding according to any of the described embodiments. A method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that performs the film grain process adapted to the core conversion according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that performs an adapted film grain process on the core transform according to any of the described embodiments and displays the final image (e.g., using a monitor, screen, or other type of display). a television, set-top box, mobile phone, tablet, or other electronic device that selects a channel for receiving a signal containing the encoded image (e.g., using a tuner) and performs a film grain process adapted to the core transformation according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that wirelessly receives a signal containing an encoded image according to any of the described embodiments (e.g., using an antenna) and performs a film grain process adapted to the core transform.

Claims

1. 1. A method comprising: receiving film grain information including at least one parameter specifying an attribute of film grain associated with an image block; applying a transform to the block of random values, the transform being one of the standardized core transforms DCT-II, DCT-VII, or DST-VII; filtering the transformed block of random values, the filtering being defined by at least one parameter in the film grain information; and applying a respective inverse transform to the blocks of filtered transformed random values ​​to generate blocks of pixels having the film grain pattern of the image block.

2. 1. A method comprising: receiving a film grain block representing a film grain estimate within an image block; applying a transform to the film grain block, the transform being one of the standardized core transforms DCT-II, DCT-VIII, or DST-VII; generating film grain information from the transformed film grain block, the film grain information including at least one parameter specifying an attribute of film grain associated with the image block.

3. 3. The method of claim 1, wherein the transform is the standardized core transform DCT-II, and the size of the transform is NxM, where N is an integer in the range [2 to 64] and M is an integer in the range [2 to 64].

4. 3. The method of claim 1 or 2, wherein the transform is one of the standardized core transforms DCT-VIII, DST-VII, and the size of the transform is NxM, where N is an integer in the range [4 to 32] and M is an integer in the range [4 to 32].

5. The method of claim 1 or 2, wherein the same transform type and different transform sizes are used for luma and chroma components of the image block.

6. The method of claim 1 or 2, wherein different transform types and the same transform size are used for luma and chroma components of the image block.

7. The method of claim 1 or 2, wherein different transform types and different transform sizes are used for luma and chroma components of the image block.

8. The film grain information includes an indication of the type of transform used on the luma component of the image block; an indication of a transform size to be used for a luma component of the image block; an indication of the type of transformation to be used on the chroma components of said image block; an indication of a transform size to be used for a chroma component of said image block; an indication of the same transform type to be used for luma and chroma components of said image block; an indication of the same transform size to be used for luma and chroma components of said image block; An indication as to whether a luma transform type is inferred; an indication as to whether a chroma transformation type is inferred; An indication as to whether the luma transform size is inferred; The method of claim 1 or 2, further comprising at least one indication as to whether a chroma transform size is inferred or not.

9. The method of claim 8 , wherein a transform type and transform size used for chroma components of the image block are derived from the indication of a transform type for a luma component of the image block and the indication of a transform size for the luma component of the image block.

10. The method of claim 8 , wherein a horizontal transform type and a vertical transform type are used for luma and chroma components of the image block, and the horizontal transform type and vertical transform type are different.

11. 3. The method of claim 1, further comprising scaling at least one parameter specifying an attribute of the film grain associated with the image block according to at least one of a standardized core transform size or a standardized core transform type used to generate the at least one parameter.

12. 1. An apparatus comprising: a memory; and one or more processors, the one or more processors: receiving film grain information including at least one parameter specifying an attribute of film grain associated with an image block; applying a transform to the block of random values, the transform being one of the standardized core transforms DCT-II, DCT-VII, or DST-VII; filtering the transformed block of random values ​​based on the at least one parameter in the film grain information; and applying a respective inverse transform to the block of filtered transformed random values ​​to generate a block of pixels having the film grain pattern of the image block.

13. 1. An apparatus comprising: a memory; and one or more processors, the one or more processors: receiving a film grain block representing a film grain estimate within an image block; applying a transform to the film grain block, the transform being one of the standardized core transforms DCT-II, DCT-VIII, or DST-VII; generating, from the transformed film grain block, film grain information comprising at least one parameter specifying an attribute of film grain associated with the image block.

14. 14. The apparatus according to claim 12 or 13, wherein the transform is the standardized core transform DCT-II, the size of the transform is NxM, N is an integer in the range [2 to 64] and M is an integer in the range [2 to 64].

15. 14. The apparatus according to claim 12 or 13, wherein the transform is one of the standardized core transforms DCT-VIII, DST-VII, and the size of the transform is NxM, where N is an integer in the range [4 to 32] and M is an integer in the range [4 to 32].

16. The apparatus according to claim 12 or 13, wherein the same transform type and different transform sizes are used for luma and chroma components of the image block.

17. The apparatus according to claim 12 or 13, wherein different transform types and the same transform size are used for luma and chroma components of the image block.

18. The apparatus according to claim 12 or 13, wherein different transform types and different transform sizes are used for luma and chroma components of the image block.

19. The film grain information includes an indication of the type of transform used on the luma component of the image block; an indication of a transform size to be used for a luma component of the image block; an indication of the type of transformation to be used on the chroma components of said image block; an indication of a transform size to be used for a chroma component of said image block; an indication of the same transform type to be used for luma and chroma components of said image block; an indication of the same transform size to be used for luma and chroma components of said image block; An indication as to whether a luma transform type is inferred; an indication as to whether a chroma transformation type is inferred; An indication as to whether the luma transform size is inferred; 14. Apparatus according to claim 12 or 13, further comprising at least one of an indication as to whether a chroma transform size is inferred or not.

20. The apparatus of claim 12 or 13, wherein the transform type and transform size used for the chroma components of the image block are derived from the indication of a transform type for a luma component of the image block and the indication of a transform size for the luma component of the image block.

21. The apparatus of claim 12 or 13, wherein a horizontal transform type and a vertical transform type are used for luma and chroma components of the image block, and the horizontal transform type and the vertical transform type are different.

22. The apparatus of claim 12 or 13, further comprising: scaling at least one parameter specifying an attribute of the film grain associated with the image block according to at least one of a standardized core transform size or a standardized core transform type used to generate the at least one parameter.

Images