Film particle synthesis using encoded information

Abstract

A method comprising: obtaining (601) at least one characteristic of a current sample of a picture prior to reconstructing the current sample; determining (602) a parameter of a film particle synthesis process to be applied to the current sample according to the at least one characteristic; storing (603) the determined parameters and information representative of the position of the current sample; reconstructing (604) the current sample; and applying (605) the film particle synthesis process to the current sample with the determined parameters.

Description

1. Technical Field

[0001] At least one of the present embodiments generally relates to a method and apparatus for film grain synthesis. 2. Background technology

[0002] In order to achieve high compression efficiency, video coding schemes typically use prediction and transformation to exploit spatial and temporal redundancy in video content. During encoding, pictures of video content are divided into blocks of samples (i.e., pixels), which are then divided into one or more sub-blocks, hereinafter referred to as original sub-blocks. Intra-frame or inter-frame prediction is then applied to each sub-block to exploit intra-frame or inter-frame image correlation. Regardless of the prediction method used (intra-frame or inter-frame), a predicted value sub-block is determined for each original sub-block. The sub-block representing the difference between the original sub-block and the predicted value sub-block (usually represented as a prediction error sub-block, a prediction residual sub-block, or simply a residual sub-block) is then transformed, quantized, and entropy encoded to generate an encoded video stream. In order to reconstruct the video, the compressed data is decoded by an inverse process corresponding to transformation, quantization, and entropy encoding.

[0003] In the entertainment industry, film grain is widely present in film and TV material and is considered part of the creative intent. Grain is inherent in analog motion picture film due to the process of exposure and development of silver halide crystals dispersed in the photosensitive emulsion, as randomly distributed grains appear where the silver crystals have formed. Digital cameras do not produce film grain, however in post-production film grain is often added to captured footage to create a "film" look. Therefore, when encoding film and TV content, it is important to preserve film grain to maintain the creative intent of the content creator.

[0004] The random nature of this film grain, which can be considered as random noise, makes it difficult to compress using traditional encoding tools. Common parameters of encoding tools, such as those selected for low bit rates, can remove film grain. High bit rates are required to preserve and reconstruct film grain with sufficient quality, which contradicts the goal of compression tools to save bits. To overcome this problem, film grain is usually modeled before the encoding stage and then added back at the decoding stage during the so-called compositing step.

[0005] In some implementations, once estimated at the encoding stage, film grain parameters are sent in the form of metadata along with the compressed video data. After decoding, film grain is synthesized and added back to the reconstructed video picture. The synthesis of film grain typically includes a step of extracting information from the reconstructed picture to adapt the added film grain to the picture content. The extraction of this information adds complexity to the decoding stage.

[0006] It is desirable to propose a solution allowing to overcome the above mentioned problems. In particular, it is desirable to propose a solution which reduces the complexity of the film grain synthesis process. 3. Summary of the invention

[0007] In a first aspect, one or more of the present embodiments provide a method comprising: obtaining at least one characteristic of a current sample of a picture before reconstructing the current sample; determining parameters of a film grain synthesis process to be applied to the current sample based on the at least one characteristic; storing the determined parameters and information representing the position of the current sample; reconstructing the current sample; and applying the film grain synthesis process to the current sample using the determined parameters.

[0008] In an embodiment, the at least one characteristic of the current sample is at least one characteristic of a current block comprising the current sample.

[0009] In an embodiment, the at least one characteristic of the current block comprises at least one of: information representing a shape of the current block, a position of the current block, a sum of bits used to encode all components of the current block, information representing a surface delimited by the current block, information indicating that a prediction residual of the current block comprises at least one non-zero transform coefficient.

[0010] In an embodiment, in response to the at least one characteristic of the current block comprising information representing a shape and a position of the current block, the parameters of the film grain synthesis process are the shape and position of the block comprising the film grain sample to be applied to the film grain sample of the current sample.

[0011] In an embodiment, applying a film grain synthesis process to the current sample comprises adding a film grain value to the value of the current sample, wherein the film grain value is obtained by weighting a value obtained using a film grain model by a first weighting factor that depends on the value of the current sample or an average of samples of the current block.

[0012] In an embodiment, in response to the at least one characteristic of the current block comprising a sum of bits used to encode all components of the current block, the value obtained using the film grain model is further weighted by a weighting factor that depends on the sum of bits used to encode all components of the current block.

[0013] In an embodiment, in response to the at least one characteristic of the current block comprising information representative of a surface bounded by the current block, the value obtained using the film grain model is further weighted by a weighting factor dependent on the surface bounded by the current block.

[0014] In an embodiment, in response to the at least one characteristic of the current block comprising information indicating that the prediction residual of the current block includes at least one non-zero transform coefficient, the value obtained using the film grain model is further weighted by a weighting factor, wherein the weighting factor depends on the information indicating that the prediction residual of the current block includes at least one non-zero transform coefficient.

[0015] In a second aspect, one or more of the present embodiments provides a device comprising an electronic circuit, wherein the electronic circuit is configured to: obtain at least one characteristic of a current sample of a picture before reconstructing the current sample; determine parameters of a film grain synthesis process to be applied to the current sample based on the at least one characteristic; store the determined parameters and information representing the position of the current sample; reconstruct the current sample; and apply the film grain synthesis process to the current sample using the determined parameters.

[0016] In an embodiment, the at least one characteristic of the current sample is at least one characteristic of a current block comprising the current sample.

[0017] In an embodiment, the at least one characteristic of the current block comprises at least one of: information representing a shape of the current block, a position of the current block, a sum of bits used to encode all components of the current block, information representing a surface delimited by the current block, information indicating that a prediction residual of the current block comprises at least one non-zero transform coefficient.

[0018] In an embodiment, in response to the at least one characteristic of the current block comprising information representing a shape and a position of the current block, the parameters of the film grain synthesis process are the shape and position of the block comprising the film grain sample to be applied to the film grain sample of the current sample.

[0019] In an embodiment, applying a film grain synthesis process to the current sample comprises adding a film grain value to the value of the current sample, wherein the film grain value is obtained by weighting a value obtained using a film grain model by a first weighting factor that depends on the value of the current sample or an average of samples of the current block.

[0020] In an embodiment, in response to the at least one characteristic of the current block comprising a sum of bits used to encode all components of the current block, the value obtained using the film grain model is further weighted by a weighting factor that depends on the sum of bits used to encode all components of the current block.

[0021] In an embodiment, in response to the at least one characteristic of the current block comprising information representative of a surface bounded by the current block, the value obtained using the film grain model is further weighted by a weighting factor dependent on the surface bounded by the current block.

[0022] In an embodiment, in response to the at least one characteristic of the current block comprising information indicating that the prediction residual of the current block includes at least one non-zero transform coefficient, the value obtained using the film grain model is further weighted by a weighting factor, wherein the weighting factor depends on the information indicating that the prediction residual of the current block includes at least one non-zero transform coefficient.

[0023] In a third aspect, one or more of the present embodiments provides a computer program comprising program code instructions for implementing the method according to the first aspect.

[0024] In a fourth aspect, one or more of the present embodiments provides a non-transitory information storage medium storing program code instructions for implementing the method according to the first aspect. 4. Description of the drawings

[0025] Figure 1 schematically illustrates a context in which embodiments are implemented;

[0026] Figure 2 schematically illustrates an example of segmentation undergone by a picture of pixels of an original video;

[0027] Figure 3 A method for encoding a video stream is schematically described;

[0028] Figure 4 A method for decoding an encoded video stream is schematically described;

[0029] Figure 5A Schematically illustrates an example of a hardware architecture of a processing module capable of implementing an encoding module or a decoding module in which various aspects and embodiments are implemented;

[0030] Figure 5B illustrates a block diagram of an example of a first system in which various aspects and embodiments are implemented;

[0031] Figure 5C illustrates a block diagram of an example of a second system in which various aspects and embodiments are implemented;

[0032] Figure 6 An embodiment that allows for reducing the complexity of the film grain synthesis process is illustrated;

[0033] Fig. 7A The film grain modeling framework is schematically represented;

[0034] Figure 7B Schematic illustration of film grain synthesis and the re-noising process;

[0035] Fig. 8Aschematically represents neighboring samples of the current block, which can be used to estimate the average value of the samples of the current block; and

[0036] Figure 8B The DC coefficient that can be used to estimate the DC coefficient of the current block is schematically represented. 5. Specific implementation methods

[0037] Examples of the following embodiments are described in the context of a video format similar to VVC (Versatile Video Coding (VVC) developed by a joint collaborative group of ITU-T and ISO / IEC experts, known as the Joint Video Experts Team (JVET)). However, the embodiments are not limited to video encoding / decoding methods corresponding to VVC. The embodiments are particularly suitable for various video formats, including, for example, HEVC (ISO / IEC 23008-2-MPEG-H Part 2, High Efficiency Video Coding / ITU-T H.265), AVC ((ISO / CEI 14496-10), EVC (Basic Video Coding / MPEG-5), AV1, AV2, and VP9.

[0038] Figure 1 The context in which the embodiments are implemented is schematically illustrated.

[0039] exist Figure 1 In the embodiment, system 11, which may be a camera, storage device, computer, server, or any device capable of transmitting a video stream, transmits a video stream to system 13 using communication channel 12. The video stream is encoded and transmitted by system 11, or received and / or stored by system 11 and then transmitted. Communication channel 12 is a wired (e.g., Internet or Ethernet) or wireless (e.g., WiFi, 3G, 4G, or 5G) network link.

[0040] The system 13, which may be, for example, a set-top box, receives and decodes the video stream to generate a sequence of reconstructed pictures. Post-processing, such as a film grain synthesis process, is applied to the reconstructed pictures.

[0041] The obtained sequence of post-processed reconstructed pictures is then transmitted to a display system 15 using a communication channel 14, which may be a wired or wireless network. The display system 15 then displays the pictures.

[0042] In an embodiment, the system 13 is included in the display system 15. In that case, the system 13 and the display system 15 are included in a TV, a computer, a tablet computer, a smart phone, a head mounted display, or the like.

[0043] Figure 2 , 3 and 4 introduce examples of video formats.

[0044] Figure 2 An example of segmentation is shown of a picture of pixels 21 of an original video sequence 20. Here, a pixel is considered to consist of three components: a luminance component and two chrominance components. However, other types of pixels including fewer or more components (such as only a luminance component or an additional depth component or transparency component) are also possible.

[0045] The picture is divided into multiple coding entities. First, Figure 2 As shown by reference numeral 23 in the figure, the picture is divided into a grid of blocks called coding tree units (CTUs). A CTU consists of a block of N×N luminance samples and two corresponding blocks of chrominance samples. N is usually a power of 2, which has a maximum value of "128" for example. Secondly, the picture is divided into one or more groups of CTUs. For example, it can be divided into one or more tile rows and tile columns, and a tile is a sequence of CTUs covering a rectangular area of ​​the picture. In some cases, a tile can be divided into one or more bricks, each of which consists of at least one row of CTUs within the tile. On top of the concepts of tiles and blocks, there is another coding entity, called a slice, which can contain at least one tile of a picture or at least one block of a tile.

[0046] exist Figure 2 In the example in FIG. 2 , as indicated by reference numeral 22 , a picture 21 is divided into three slices S1 , S2 and S3 in a raster scan slice mode, each slice including a plurality of tiles (not shown), each tile including only one block.

[0047] like Figure 2 As shown by reference numeral 24 in the figure, a CTU can be partitioned into a hierarchical tree of one or more sub-blocks called coding units (CUs). A CTU is the root (i.e., parent node) of the hierarchical tree and can be partitioned into multiple CUs (i.e., child nodes). Each CU becomes a leaf of the hierarchical tree if it is not further partitioned into smaller CUs, or becomes a parent node of a smaller CU (i.e., child node) if it is further partitioned.

[0048] exist Figure 2 In the example of , CTU 24 is first split into "4" square CUs using quadtree type splitting. The top left CU is a leaf of the hierarchical tree because it is not further split, i.e., it is not the parent of any other CU. Using quadtree type splitting again, the top right CU is further split into "4" smaller square CUs. The bottom right CU is vertically split into "2" rectangular CUs using binary tree type splitting. The bottom left CU is vertically split into "3" rectangular CUs using ternary tree type splitting.

[0049] During encoding of a picture, the partitioning is adaptive, and each CTU is partitioned in order to optimize the compression efficiency of the CTU scale.

[0050] The concepts of prediction unit (PU) and transform unit (TU) appear in HEVC. In fact, in HEVC, the coding entity for prediction (i.e. PU) and the coding entity for transform (i.e. TU) can be a subdivision of CU. For example, Figure 2 As shown in , a CU of size 2N×2N may be divided into PUs 2411 of size N×2N or PUs 2412 of size 2N×N. In addition, the CU may be divided into "4" TUs 2412 of size N×N or TUs of size 2N×N. The “16” TUs.

[0051] It can be noted that in VVC, except in some special cases, the boundaries of TU and PU are aligned on the boundaries of CU. Therefore, a CU usually includes one TU and one PU.

[0052] In this application, the term "block" or "picture block" may be used to refer to any of CTU, CU, PU and TU. In addition, the term "block" or "picture block" may be used to refer to macroblocks, partitions and sub-blocks as specified in H.264 / AVC or other video coding standards, and more generally to arrays of samples of various sizes.

[0053] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture", "sub-picture", "slice" and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstruction" is used at the encoder side, while "decoding" is used at the decoder side.

[0054] Figure 3 The method for encoding a video stream performed by an encoding module is schematically depicted. Figure 3 The method for encoding is performed by a processing module of the system 11. The processing module corresponds to the following description of Figure 5A The processing module 500 is described in detail. Variations of this method for encoding are considered, but for clarity, the following description is Figure 3 of methods used for encoding without describing all the expected variations.

[0055] Before being encoded, the current original picture of the original video sequence may be pre-processed. For example, in the pre-processing step 301, film grain analysis is applied to the original picture.

[0056] Fig. 7A Schematic representation of the film grain modeling framework.

[0057] For example, during step 301, Fig. 7A process.

[0058] In step 3011, the processing module 500 obtains the original image and removes film grain from the original image using a denoising process. Many methods for film grain denoising have been proposed in the literature. For example, during step 3011, the processing module 500 applies the denoising process described in the following document: J.C. Kit Yan and D. Hatzinakos, "Signal-dependent film grain noise removal and generation based on higher-order statistics", Proc. IEEE Signal Processing Workshop on Higher-Order Statistics, July 1997, Banff, Canada.

[0059] In step 3012, the processing module 500 analyzes the denoised picture to determine smooth areas. In practice, it is important to ensure that only smooth areas of the picture are used in the estimation of the film grain model, since edges and textures may affect the estimation of film grain intensity and pattern. To determine smooth areas of the input picture, the processing module applies, for example, a Canny edge detector to the denoised image at different scales, followed by a dilation operation.

[0060] In step 3013, the processing module 500 subtracts the denoised image from the original image to obtain a noisy image.

[0061] In step 3014, the processing module 500 uses the determined smooth regions to estimate the film grain intensity and pattern from the noisy picture. In these smooth regions, the film grain pattern is modeled using an autoregressive model (AR). Let G(x, y) be the zero-mean film grain sample at the current position (x, y) in the picture. For a lag parameter L=2, the grain sample G(x, y) is calculated as follows:

[0062] G(x,y)=a0.G(x-2,y-2)+a1.G(x-1,y-2)+a2.G(x,y-2)+…+z (Equation 1)

[0063] where a0, ..., a n is the AR coefficient, G(x+k, y+m) is the film grain sample value in the causal neighborhood at the current position (x, y), and z is the unit variance Gaussian noise obtained from a predefined set stored at the decoder and encoder sides. i The number of is determined by the lag parameter L and is equal to 2L(L+1) for luma and 2L(L+1)+1 for chroma components. In the chroma components, there is an additional coefficient a ito capture the correlation with brightness grain samples at the same spatial location. The lag L can take values ​​from "0" to "3". L = 0 corresponds to modeling Gaussian noise, while higher L values ​​can correspond to film grains with larger grain sizes. AR coefficients a0...a n For example, it is estimated by a method based on the Yule-Walker AR equation.

[0064] Note that other types of film grain models may be used instead of the AR model, such as the frequency filtering model defined in the document SMPTE: Film Grain Technology - Specifications for H.264 | MPEG-4 AVC Bitstreams / RDD 5-2006.

[0065] Film grain intensity can vary with signal strength. When adding film grain to the luma component, the following model is used:

[0066] Y′(x,y)=Y(x,y)+f(Y(x,y)).G(x,y) (Equation 2)

[0067] Where Y′(x,y) is the luma sample denoised with film grain at position (x,y), Y(x,y) is the reconstructed luma sample at position (x,y), and G(x,y) is the film grain sample at position (x,y). f() is a piecewise linear function that scales the film grain according to the luma component values ​​fitted by measuring the noise intensity over a smooth region. The piecewise linear function can be implemented as a precomputed lookup table (LUT) that is initialized prior to running the film grain synthesis. Fitting the scaling function to the data can be accomplished in a variety of ways. For example, the scaling function is determined by fitting the local standard deviation of a smooth region to its local mean intensity value using a least squares method. Some additional criteria may be used, such as for zero luma values, the scaling function is equal to zero. A similar approach is applied to determine the scaling function for the chroma components.

[0068] Note that in the equation (Equation 2), the scaling function f() gives a different value for each sample Y(x, y). In other implementations, a single scaling value may be calculated for a block of samples. In that case, the average of the sample values ​​of the block is calculated And the average value of all samples from the block Calculate or export a single scaling value The average value of the sample values ​​from the block is described in the document SMPTE: Film Grain Technology - Specifications for H.264 | MPEG-4 AVC Bitstreams / RDD 5-2006 Export a single scale value in Then use the following model:

[0069]

[0070] The picture output by the pre-processing step 301 (such as the denoised picture generated during step 3011) is referred to as the pre-processed picture in the following.

[0071] As about Figure 2 As described, the encoding of the pre-processed picture starts with the partitioning of the pre-processed picture during step 302. The pre-processed picture is thus partitioned into CTUs, CUs, PUs, TUs, etc.

[0072] For each block, the encoding module then determines the coding mode between intra prediction and inter prediction.

[0073] Intra prediction consists in predicting the pixels of the current block from a prediction block derived from pixels of a reconstructed block located in the causal neighborhood of the current block to be coded according to an intra prediction method during a step 303. The result of the intra prediction is a prediction direction indicating which pixels of the nearby blocks are to be used, and a residual block resulting from the calculation of the difference between the current block and the predicted block.

[0074] Inter prediction consists in predicting the pixels of the current block from a block of pixels of a picture preceding or following the current picture, called a reference block, which picture is called a reference picture. During the encoding of the current block according to the inter prediction method, the block of the reference picture closest to the current block is determined by a motion estimation step 304 according to a similarity criterion. During step 304, a motion vector indicating the position of the reference block in the reference picture is determined. The motion vector is used during a motion compensation step 305, during which a residual block is calculated in the form of the difference between the current block and the reference block. In the first video compression standards, the above-mentioned unidirectional inter prediction mode was the only inter mode available. With the development of video compression standards, the family of inter modes has grown significantly and now includes many different inter modes.

[0075] During the selection step 306 , the prediction mode that optimizes the compression performance is selected by the encoding module from among the tested prediction modes (intra-frame prediction mode, inter-frame prediction mode) according to a rate / distortion optimization criterion (ie, RDO criterion).

[0076] When a prediction mode is selected, the residual block is transformed during step 307. The transformed block is then quantized during step 309.

[0077] Note that the encoding module can skip the transform and directly apply quantization on the untransformed residual signal.

[0078] When the current block is encoded according to an intra prediction mode, the prediction direction and the transformed and quantized residual block are encoded by the entropy encoder during step 310. When the current block is encoded according to an inter prediction, the motion vector of the block is predicted, when appropriate, from a prediction vector selected from a set of motion vector prediction values ​​derived from reconstructed blocks located in the spatial and temporal vicinity of the block to be encoded. Next, during step 310, motion information is encoded by the entropy encoder in the form of a motion residual and an index for identifying the prediction vector. During step 310, the transformed and quantized residual block is encoded by the entropy encoder.

[0079] Note that the encoding module can bypass both transform and quantization, ie, apply entropy encoding to the residual without applying transform or quantization process. The result of entropy encoding is inserted into the encoded video stream 311.

[0080] In addition, some CUs (or TUs) can be coded without any residual (i.e., where all transform coefficients are equal to zero). This information is signaled by the coded block flag (CBF). For example, VVC uses three CBFs to indicate whether a CU is coded with residual:

[0081] tu_y_coded_flag equal to "1" indicates that the luma component of the CU contains one or more transform coefficient levels not equal to zero. tu_y_coded_flag equal to "0" indicates that all transform coefficients of the CU are equal to zero.

[0082] tu_cb_coded_flag equal to "1" indicates that the Cb component of the CU contains one or more transform coefficient levels not equal to zero. tu_cb_coded_flag equal to "0" indicates that all transform coefficients of the CU are equal to zero.

[0083] tu_cr_coded_flag equal to "1" indicates that the Cr component of the CU contains one or more transform coefficient levels not equal to zero. tu_cr_coded_flag equal to "0" indicates that all transform coefficients of the CU are equal to zero.

[0084] Metadata such as SEI (Supplemental Enhancement Information) messages may be attached to the coded video stream 311. A SEI message as defined, for example, in a standard such as AVC, HEVC or VVC (or in the standard Generic Supplemental Enhancement Information (VSEI) message for coded video bitstreams - H.274) is a data container or syntax structure associated with a video stream and includes metadata that provides information related to the video stream. For example, a SEI message for transmitting film grain information has been defined in the following document: "SEI message for film grain encoding" by C. Gomila, A. Kobilansky, ISO / IEC JTC1 / SC29 / WG11, ITU-TSG16Q.6 document JVT-H022, Geneva, Switzerland, May 2003. The SEI message allows the transmission of information that allows a decoder to apply a film grain synthesis process, the SEI message including, for example, the parameters a of the AR model described by equation (Equation 1) i and a set of points with a piecewise linear scaling function f() for each color component.

[0085] After the quantization step 309, the current block is reconstructed so that the pixels corresponding to the block can be used for future predictions. This reconstruction phase is also called a prediction loop. Therefore, during step 312, inverse quantization is applied to the transformed and quantized residual block, and an inverse transform is applied during step 313. According to the prediction mode for the block obtained during step 314, the prediction block of the block is reconstructed. If the current block is encoded according to the inter-frame prediction mode, during step 316, the encoding module uses the motion vector of the current block to apply motion compensation when appropriate in order to identify the reference block of the current block. If the current block is encoded according to the intra-frame prediction mode, during step 315, the prediction direction corresponding to the current block is used to reconstruct the prediction block of the current block. The prediction block and the reconstructed residual block (if any) are added to obtain the reconstructed current block.

[0086] After reconstruction, in-loop filtering aimed at reducing coding artifacts is applied to the reconstructed blocks during step 317. This filtering is called in-loop filtering because it occurs in the prediction loop in order to obtain the same reference pictures at the decoder as at the encoder and thus avoid drift between the encoding and decoding processes. In-loop filtering tools include deblocking filtering, SAO (Sample Adaptive Offset) and ALF (Adaptive Loop Filtering).

[0087] When the block is reconstructed, it is inserted into a reconstructed picture stored in a memory of reconstructed pictures 319, usually called a decoded picture buffer (DPB), during a step 318. The reconstructed picture stored in this way can then be used as a reference picture for other pictures to be encoded.

[0088] Figure 4 Schematically depicts a decoding method for decoding performed by a decoding module according to Figure 3 The method described herein is a method for encoding a video stream 311. For example, Figure 4 The method for decoding is performed by the processing module 500 of the system 13. Variations of this method for decoding are considered, but for clarity, the following describes Figure 4 of methods for decoding without describing all the expected variations.

[0089] The decoding is performed block by block. For the current block, during a step 410, it starts with entropy decoding of the CTU comprising the current block (to determine the partitioning of the CTU), and then entropy decoding of the information representing the current block. Entropy decoding allows obtaining at least the prediction mode of the block.

[0090] If the block has been coded according to inter prediction mode, entropy decoding allows obtaining the prediction vector index, the motion residual and the residual block (if any) when appropriate.During a step 408, the motion vector is reconstructed for the current block using the prediction vector index and the motion residual.

[0091] If the block has been coded according to an intra prediction mode, entropy decoding allows obtaining the prediction direction and the residual block (if any). Steps 412, 413, 414, 415, 416 and 417 implemented by the decoding module are identical in all respects to steps 312, 313, 314, 315, 316 and 317 implemented by the encoding module, respectively.

[0092] In step 418, the decoded blocks are saved in a decoded picture, and the decoded picture is stored in the DPB 419. When the decoding module decodes a given picture, the picture stored in the DPB 419 is the same as the picture stored in the DPB 319 by the encoding module during encoding of the given picture. The decoded picture can also be output by the decoding module, for example, to be displayed.

[0093] After the in-loop filtering (i.e. after the decoded picture is generated), a post-processing step 421 may be applied. In particular, film grain may be added during the post-processing step 421. In that case, the post-processing step 421 comprises a film grain synthesis and a denoising process.

[0094] Figure 7B Schematic illustration of the film grain synthesis and denoising process.

[0095] In step 4211, the processing module 500 obtains film grain pattern parameters. For example, the processing module 500 receives SEI, which includes the parameters a of the AR model described by equation (Equation 1): iand a set of points with a piecewise linear scaling function f() for each color component.

[0096] In step 4212, the processing module 500 uses the parameter a of the AR model described in equation (Equation 1) i To generate film grain samples. Typically, the film grain samples are generated in the form of blocks of film grain samples. The size of the blocks of film grain samples is typically predefined and is, for example, equal to 32×32 for luma blocks and 16×16 for chroma blocks.

[0097] In step 4213, the processing module 500 adds the block of film grain samples to the block of reconstruction samples of the same size using equation (Equation 2). In other implementations, equation (Equation 3) may be used instead of equation (Equation 2). In that case, using the reconstructed block The average of the sample values ​​is used to derive a single scaling value for the reconstructed block.

[0098] As can be seen, the shape and position of the blocks of film grain samples used in the film grain synthesis process do not take into account the segmentation of the picture (e.g. Figure 2 ). This can be a problem because the blocks resulting from such a segmentation are considered sufficiently homogeneous on a rate / distortion basis to be encoded together. Therefore, there is no reason to segment the picture differently. Furthermore, the film grain synthesis process involves extracting features of samples of the picture to obtain film grain samples that are appropriate for the content of the picture. The extraction of these features increases the computational cost on the decoder side. The extraction process does not take into account that some of these features are already available during decoding, or will already be easily derived from data obtained during the decoding process.

[0099] Figure 5A , 5B 5C describe examples of devices, apparatuses and / or systems that allow implementation of various embodiments.

[0100] Figure 5A Schematically illustrates an example of a hardware architecture of a processing module 500 capable of implementing an encoding module or a decoding module, wherein the encoding module or the decoding module can respectively implement the modified according to different aspects and embodiments. Figure 3 The method used to encode and Figure 4 For example, when the system 11 is responsible for encoding a video stream, the encoding module is included in the system 11. The decoding module is included in the system 13, for example.

[0101] The processing module 500 connected by the communication bus 5005 includes: a processor or CPU (central processing unit) 5000, which includes one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures as non-limiting examples; a random access memory (RAM) 5001; a read-only memory (ROM) 5002; a storage unit 5003, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drive and / or optical disk drive, or storage medium reader, such as SD (Secure Digital) card reader and / or hard disk drive (HDD) and / or network accessible storage device; at least one communication interface 5004 for exchanging data with other modules, devices or systems. The communication interface 5004 may include but is not limited to a transceiver configured to transmit and receive data through a communication channel. The communication interface 5004 may include, but is not limited to, a modem or a network card.

[0102] If the processing module 500 implements a decoding module, the communication interface 5004 enables, for example, the processing module 500 to receive a coded video stream and provide a sequence of decoded pictures. If the processing module 500 implements an encoding module, the communication interface 5004 enables, for example, the processing module 500 to receive a sequence of raw picture data for encoding and provide a coded video stream.

[0103] The processor 5000 can execute instructions loaded into the RAM 5001 from the ROM 5002, from an external memory (not shown), from a storage medium or from a communication network. When the processing module 500 is powered on, the processor 5000 can read the instructions from the RAM 5001 and execute them. These instructions form a computer program that enables, for example, the processor 5000 to implement the following operations. Figure 4 Decoding methods described and / or Figure 3 The encoding method described, and Figure 6 The illustrated method includes various aspects and embodiments described below in this document.

[0104] Figure 3 , 4 All or some of the algorithms and steps of the methods of 6 may be implemented in software by executing a set of instructions by a programmable machine, such as a DSP (digital signal processor) or a microcontroller, or in hardware by a machine or dedicated components, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0105] As can be seen, microprocessors, general purpose computers, special purpose computers, processors based on or not based on multi-core architectures, DSPs, microcontrollers, FPGAs, and ASICs are suitable or configured to at least partially implement Figure 3 , 4 and an electronic circuit of a method of 6.

[0106] Figure 5C A block diagram of an example of a system 13 in which various aspects and embodiments are implemented is illustrated. System 13 can be implemented as a device including the various components described below, and is configured to perform one or more of the aspects and embodiments described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and head-mounted displays. The elements of system 13 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, system 13 includes a processing module 500 that implements a decoding module. In various embodiments, system 13 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 13 is configured to implement one or more of the aspects described in this document.

[0107] Inputs to processing module 500 may be provided through various input modules, as shown in block 531. Such input modules include, but are not limited to, (i) a radio frequency (RF) module that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input module (or a group of COMP input modules), (iii) a universal serial bus (USB) input module, and / or (iv) a high-definition multimedia interface (HDMI) input module. Figure 5C Other examples not shown include composite video.

[0108] In various embodiments, the input module of block 531 has associated corresponding input processing elements as known in the art. For example, the RF module can be associated with elements suitable for the following operations: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting a narrower frequency band to select a signal frequency band that can be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF module of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion can include a tuner that performs various functions in these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or a baseband. In a set-top box embodiment, RF module and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band again by filtering, down-conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions.Adding element can include inserting element between existing element, such as for example inserting amplifier and analog-to-digital converter.In various embodiments, RF module comprises antenna.

[0109] In addition, the USB and / or HDMI modules may include corresponding interface processors for connecting the system 13 to other electronic devices across the USB and / or HDMI connections. It is to be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processing module 500 as desired. Similarly, various aspects of USB or HDMI interface processing may be implemented in a separate interface IC or in the processing module 500 as desired. The demodulated, error corrected, and demultiplexed stream is provided to the processing module 500.

[0110] The various elements of the system 13 may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected and data may be transferred between them using a suitable connection arrangement, such as an internal bus as known in the art, including an inter-IC (I2C) bus, wiring, and a printed circuit board. For example, in the system 13, the processing module 500 is interconnected to the other elements of the system 13 via a bus 5005.

[0111] The communication interface 5004 of the processing module 500 allows the system 13 to communicate on the communication channel 12. As already mentioned above, the communication channel 12 may be implemented in, for example, a wired and / or wireless medium.

[0112] In various embodiments, data is streamed or otherwise provided to the system 13 using a wireless network, such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received via a communication channel 12 and a communication interface 5004 suitable for Wi-Fi communications. The communication channel 12 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use an RF connection of an input block 531 to provide streaming data to the system 13. As shown above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0113] The system 13 can provide output signals to various output devices, including a display system 15, a speaker 535, and other peripherals 536. The display system 15 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display system 15 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), a head-mounted display, or other devices. The display system 15 can also be integrated with other components (for example, as in a smart phone), or separated (for example, an external monitor for a laptop computer). In various examples of embodiments, other peripherals 536 include one or more of an independent digital video disk (or digital versatile disk) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripherals 536 that provide functions based on the output of the system 13. For example, a disk player performs the function of playing the output of the system 13.

[0114] In various embodiments, control signals are transmitted between the system 13 and the display system 15, speakers 535, or other peripheral devices 536 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 13 via dedicated connections through respective interfaces 532, 533, and 534. Alternatively, the output devices may be communicated using the communication channel 12 via the communication interface 5004 or using a corresponding Figure 5C The display interface 532 is connected to the system 13 through a dedicated communication channel of the communication channel 12 in the display interface 532. The display system 15 and the speaker 535 can be integrated into a single unit with other components of the system 13 in an electronic device (such as, for example, a television). In various embodiments, the display interface 532 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0115] Alternatively, the display system 15 and the speaker 535 can be separate from one or more of the other components. In various embodiments in which the display system 15 and the speaker 535 are external components, the output signal can be provided via a dedicated output connection (including, for example, an HDMI port, a USB port, or a COMP output).

[0116] Figure 5B A block diagram of an example of a system 11 in which various aspects and embodiments are implemented is illustrated. System 11 is very similar to system 13. System 11 can be implemented as a device including various components described below, and is configured to perform one or more of the aspects and embodiments described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, cameras, and servers. The elements of system 11 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, system 11 includes a processing module 500 that implements a coding module. In various embodiments, system 11 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 11 is configured to implement one or more of the aspects described in this document.

[0117] The input to the processing module 500 may be provided by various input modules, as already described with respect to Figure 5C As shown in block 531 of the description.

[0118] The various elements of the system 11 can be provided in an integrated housing. Within the integrated housing, the various elements can be interconnected and data can be transmitted between them using a suitable connection arrangement, such as an internal bus as known in the art, including an inter-IC (I2C) bus, wiring, and a printed circuit board. For example, in the system 11, the processing module 500 is interconnected to the other elements of the system 11 via a bus 5005.

[0119] The communication interface 5004 of the processing module 500 allows the system 11 to communicate over the communication channel 12 .

[0120] In various embodiments, data is streamed or otherwise provided to the system 11 using a wireless network, such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received via a communication channel 12 and a communication interface 5004 adapted for Wi-Fi communications. The communication channel 12 of these embodiments is typically connected to an access point or router that provides access to an external network, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use the RF connection of the input block 531 to provide streaming data to the system 13.

[0121] As shown above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0122] The data provided to the system 11 may be provided in different formats. In various embodiments, these data are encoded and conform to known video compression formats, such as AV1, VP9, ​​VVC, HEVC, AVC, EVC, AV2, etc. In various embodiments, these data are raw data provided, for example, by a picture and / or audio acquisition module connected to the system 11 or included in the system 11. In that case, the processing module 500 is responsible for the encoding of these data.

[0123] System 11 may provide output signals to various output devices, such as system 13, which may be capable of storing and / or decoding the output signals.

[0124] Various implementations involve decoding. "Decoding" as used in this application may include, for example, all or part of the processing performed on a received encoded video stream to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transformation, and prediction. In various embodiments, such processes also include or alternatively include processes performed by the decoder of the various implementations described in this application, for example, for applying a film grain synthesis process in a post-processing step.

[0125] Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or broadly to a broader decoding process will be clear based on the context of the specific description and is considered to be well understood by those skilled in the art.

[0126] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding", "encoding" as used in this application may include, for example, all or part of a process performed on an input video sequence to produce an encoded video stream. In various embodiments, such a process includes one or more of the processes typically performed by an encoder, such as segmentation, prediction, transform, quantization, and entropy encoding. In various embodiments, such a process also includes or alternatively includes a process performed by an encoder of the various implementations described in this application, for example, for applying a film grain modeling process and / or for encoding metadata representing a film grain model, for example in the form of an SEI message.

[0127] Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or broadly to a wider encoding process will be clear based on the context of the specific description and is considered to be well understood by those skilled in the art.

[0128] Note that the syntax element names as used herein are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0129] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0130] Various embodiments refer to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is generally considered. Rate-distortion optimization is generally expressed as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, the method can be based on extensive testing of all coding options (including all considered modes or coding parameter values), and a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to avoid (save) coding complexity, especially the calculation of approximate distortion based on prediction or prediction residual signals rather than reconstructed residual signals. A mixture of these two methods can also be used, such as by using approximate distortion for only some of the possible coding options and using full distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many methods use any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both coding costs and associated distortions.

[0131] The implementations and aspects described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., a device or program). The device may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication equipment, such as, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other equipment that facilitates the communication of information between end users.

[0132] References to "one embodiment" or "an embodiment" or "an implementation" or "implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearance of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations thereof appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0133] Furthermore, the present application may involve "determining" various pieces of information. Determining information may include, for example, one or more of: estimating information, calculating information, predicting information, retrieving information from a memory, or obtaining information, for example, from another device, module, or from a user.

[0134] Furthermore, the present application may involve "accessing" various pieces of information. Accessing information may include, for example, one or more of: receiving information, retrieving information (e.g., retrieving information from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0135] Furthermore, the present application may involve "receiving" various pieces of information. As with "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of: accessing information or retrieving information (e.g., retrieving information from a memory). Furthermore, "receiving" is generally involved in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0136] It is to be understood that, for example, in the case of “A / B,” “A and / or B,” and “at least one of A and B,” and “one or more of A and B,” use of any of the following “ / ,” “and / or,” and “at least one of,” and “one or more of” is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of “A, B, and / or C,” and “at least one of A, B, and C,” and “one or more of A, B, and C,” such wording is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first listed option and the second listed option (A and B), or only the first listed option and the third listed option (A and C), or only the second listed option and the third listed option (B and C), or all three options (A, B, and C). As will be apparent to one of ordinary skill in this and related arts, this may be extended to as many items as listed.

[0137] In addition, among other things, as used herein, the word "signaling" also points to the corresponding decoder to indicate something. For example, in some embodiments, the encoder signals the use of some coding tools. In this way, in an embodiment, the same parameters can be used at both the encoder side and the decoder side. Therefore, for example, the encoder can transmit (explicitly signal) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without transmitting (implicitly signaling) to allow only the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit saving is achieved in various embodiments. It is to be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more grammatical elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signaling" is mentioned above, the word "signal" can also be used as a noun in this article.

[0138] As will be apparent to one of ordinary skill in the art, implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may include a signal indicating how to apply a CC encoding tool. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or as a baseband signal. Formatting may include, for example, encoding an encoded video stream and modulating a carrier with the encoded video stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0139] Figure 6 Embodiments are illustrated that allow for reducing the complexity of the film grain synthesis process.

[0140] Figure 6 The process of Figure 4 is performed during the illustrated decoding process. Figure 6 The process is executed by the processing module 500 of the system 13.

[0141] In step 601, the processing module 500 obtains at least one characteristic of a current sample of a picture before reconstructing the current sample. Since each sample of a picture belongs to a block (e.g., CU), the at least one characteristic of the current sample is usually at least one characteristic of the block (referred to as the current block) including the sample.

[0142] In a first embodiment, at least one characteristic of the current sample includes the position and shape (width and height) of the current block containing the current sample. This information is obtained during entropy decoding of the CTU including the current block (step 410). The position and shape of the current block are obtained without reconstruction of the block and can be obtained before inverse quantization (step 412), inverse transformation (step 413), intra-frame or inter-frame prediction (steps 414, 408, 416 and 415) and in-loop filtering (step 417) of the block.

[0143] In step 602, the processing module 500 determines parameters of a film grain synthesis process to be applied to the current sample based on at least one characteristic.

[0144] In a first embodiment, when the at least one characteristic includes the position and shape of the current block, the processing module 500 determines the position and shape of a block of at least one film grain sample to be applied to the current block including the current sample. In the first embodiment, the block of film grain samples has the same position and the same shape as the current block. In a variation of the first embodiment, particularly for large blocks, a block of multiple film grain samples corresponding to subdivisions of the current block is determined.

[0145] In step 603, the processing module 500 stores the determined at least one characteristic. In the first embodiment, the processing module 500 stores the position and shape of the current block.

[0146] In step 604, the processing module 500 reconstructs the sample. This step includes completing Figure 4 The process of rebuilding the current block.

[0147] In step 605, the processing module applies a film grain synthesis process to the current sample using the determined parameters and using the stored shape and position of the current block. In a first embodiment, step 605 includes determining a block of film grain samples having the same position and shape as the current block using equation (Equation 1) and adding the block of film grain samples to the reconstructed current block using equation (Equation 2). The film grain synthesis process can be applied to only the luma component of the current block, or to both the luma and chroma components of the current block.

[0148] In a first variation of the first embodiment, the film grain synthesis process includes applying equation (Equation 3). In that case, the average of the sample values ​​of the current block is obtained from Export a single scaling value for the current block in As described in the document SMPTE: FilmGrain Technology–Specifications for H.264|MPEG-4AVC Bitstreams / RDD 5-2006. Without fully reconstructing the current block, the exact average of the sample values ​​of the current block cannot be determined. However, the average Estimates It can be derived from samples adjacent to the current block, or from information obtained after partial decoding of the current block. In a first variant, at least one characteristic of the current sample includes an average value representing the sample values ​​of the current block. Value And the parameter of the film grain synthesis process is a single scaling value of the current block

[0149] Several solutions allow estimating the mean of the sample values ​​representing the current block Value

[0150] Fig. 8A represents the first solution, where the value is the average of the samples 802 and 804 in the vicinity of the current block 800. In that case, only the partitioning applied to obtain the block 800 needs to be entropy decoded (step 410).

[0151] In the second solution, if the current block is coded in intra mode, the direction of intra prediction can be used to improve the value For example, if the direction of intra prediction is horizontal (prediction samples on the left side of the current block), the value is the average of samples 804. If the direction of intra prediction is vertical (prediction samples on top of the current block), then the value is the average value of the samples 802. In that case, the intra prediction is obtained during the entropy decoding step 410.

[0152] In a third solution, if the current block is coded in inter-frame mode, the value is estimated from the average value of the samples of the reference block (or reference blocks in case of bidirectional prediction) In that case, the reference block(s) are identified using motion information associated with the current block. In step 408, motion information is obtained.

[0153] Figure 8B represents the fourth solution, where the value is the average of the DC values ​​of the neighboring blocks. The DC value is the value calculated when the transform is applied to a block that represents the average value of the blocks. Figure 8B Medium, value is the average of the DC value 811 of block 810 and the DC value 821 of block 820. In that case, the partitioning applied in order to obtain block 800 needs to be entropy decoded (step 410) and the transform applied to the neighboring blocks above and on the left of the current block to obtain their DC coefficients.

[0154] In the fifth solution, an approximation of the DC value 801 of block 800 is obtained by reconstructing the current block up to the inverse quantization step 412 and by adding the DC value of the inverse quantized residual to the average value calculated in the first, second or third solution or to the DC value calculated in the fourth solution.

[0155] In the sixth solution, the average of the samples of the block (i.e., the block intensity) and the film grain intensity (i.e., the single scale value ) can be modeled via any "3" degree polynomial function, the coefficients of which are estimated during film grain model estimation at the encoding stage and transmitted with the video stream and then scaled / adjusted according to the block strength level.

[0156] In a second embodiment, the bit per pixel value bpp is calculated for the current block and used to scale the block using the scaling function f() in equation (Equation 2) or the single scaling value in equation (Equation 3). Adjusts the scaling factor obtained for the current sample. The bits per pixel value bpp is the sum of bits used to encode all components of a block. The bits per pixel value bpp defines the cost of a block in terms of bitrate. As is known, textured areas have a higher cost than uniform areas. This property is used to adjust the scaling factor by increasing the intensity of film grain samples for samples in blocks with high bits per pixel value and decreasing the intensity of film grain samples for samples in blocks with low bits per pixel value, as it is also known that in order to match the intention of the technician, it is recommended to add grain with higher intensity in textured areas. This is done by increasing the scaling factor obtained for the current sample (i.e. f() or ) is combined with a first additional scaling factor Y1 depending on the bit per pixel value bpp to obtain an increase or decrease in the intensity of the film grain sample. In that case, the equation (Equation 2) becomes:

[0157] Y′(x,y)=Y(x,y)+Y1.f(Y(x,y)).G(x,y) (Equation 4)

[0158] And the equation (Equation 3) becomes:

[0159]

[0160] The calculation of the first additional scaling factor Y1 as a function of the bit per pixel value bpp can be implemented in the form of a LUT, a piecewise function or a linear function.Table TAB1 represents the calculation of the first additional scaling factor Y1 as a function of the range of bit per pixel value bpp.

[0161] Bpp range Y1 [0；0.2] 0.5 [0.2；1] 1 [1；2] 1.5 [2；] 2

[0162] Table TAB 1

[0163] In a third embodiment, the surface bounded by the current block is used to use the scaling function f() in equation (Equation 2) or the single scaling value in equation (Equation 3) Adjust the scaling factor obtained for the current sample. It is well known that uniform areas with low costs should be located in blocks with large surfaces. Textured areas with high costs should be located in blocks with small surfaces. In this embodiment, the intensity of the film grain samples is adjusted according to the current block size (representing the surface of the current block) to take into account the information of the texture given by the block size. It is proposed to increase the intensity of film grain samples for blocks with small block sizes and to reduce the intensity of film grain samples for blocks with large block sizes. The increase or decrease in the intensity of the film grain samples is achieved by adjusting the intensity of the film grain samples for the current sample (i.e., f() or ) is obtained by combining the scaling factor obtained with a second additional scaling factor Y2 depending on the current block size. In that case, the equation (Equation 2) becomes:

[0164] Y′(x,y)=Y(x,y)+Y2.f(Y(x,y)).G(x,y) (Equation 6)

[0165] And the equation (Equation 3) becomes:

[0166]

[0167] The calculation of the second additional scaling factor Y2 as a function of the block size can be implemented in the form of a LUT, a piecewise function or an affine function. For example, an example of an affine function allowing the calculation of Y2 can be:

[0168] Y2=a.X+b

[0169] Where X is the current block size, and a and b are parameters of an affine model obtained, for example, using a training sequence or by defining points. For example, for block size = 16, the two points may be Y2 = 2, and for block size = 1024, the two points may be Y2 = 0.5. In that case, a = 0.015, and b = 1.9762.

[0170] In a fourth embodiment, the CBF of the current block is used to use the scaling function f() in equation (Equation 2) or the single scaling value in equation (Equation 3) Adjust the scaling factor obtained for the current sample. The CBF indicates whether the (inter-frame or intra-frame) prediction residual of the current block contains at least one non-zero transform coefficient. A CBF indicating that all predicted transform coefficients are zero can be considered as information representing low activity in the block. It is proposed to increase the strength of the film grain samples of the current block with a CBF indicating that the current block is associated with a prediction residual containing at least one non-zero transform coefficient, and to not modify the strength of the film grain samples of the current block with a CBF indicating that the current block is associated with a prediction residual containing only zero transform coefficients. The increase in the film grain sample strength is achieved by adding a CBF to the prediction residual for the current sample (i.e., f() or ) is obtained by combining the scaling factor obtained with the third additional scaling factor Y3 depending on the CBF. In that case, the equation (Equation 2) becomes:

[0171] Y′(x,y)=Y(x,y)+Y3.f(Y(x,y)).G(x,y) (Equation 8)

[0172] And the equation (Equation 3) becomes:

[0173]

[0174] In that case, Y3=1 when the CBF of the current block indicates that the prediction residual of the current block contains only zero transform coefficients, and Y3=2 when the CBF of the current block indicates that the prediction residual of the current block contains at least one non-zero transform coefficient.

[0175] In the specific case of a block encoded in intra mode using DC mode or planar mode and the CBF indicates that the current block is associated with a prediction residual containing only zero transform coefficients, the probability that the current block is uniform is very high. Therefore, in a variation of the fourth embodiment, when the current block is encoded in DC mode or planar mode and the CBF indicates that the current block is associated with a prediction residual containing only zero transform coefficients, the intensity of the film grain sample is reduced, where Y3=0.5.

[0176] In a fifth embodiment, an estimate of the complexity of the texture of the current block is used to use a scaling function f() in equation (Equation 2) or a single scaling value in equation (Equation 3) Adjust the scaling factor obtained for the current sample. An estimate of the complexity of the texture of the block can be obtained by calculating the variance of samples adjacent to the current block (eg, the variance of samples used to calculate the intra-frame prediction value of the current block), such as Fig. 8A The higher the variance, the more complex the texture of the current block. The lower the variance of the sample, the closer the block is to a uniform block. It is proposed to increase the strength of the film grain sample when the variance of the samples adjacent to the current block is high, and to reduce the strength of the film grain sample when the variance of the samples adjacent to the current block is low. ) is combined with a fourth additional scaling factor Y4 depending on the variance of the samples neighboring the current block to obtain an increase in the intensity of the film grain samples. In that case, the equation (Equation 2) becomes:

[0177] Y′(x,y)=Y(x,y)+Y4.f(Y(x,y)).G(x,y) (Equation 10)

[0178] And the equation (Equation 3) becomes:

[0179]

[0180] In that case, when the variance V of the samples adjacent to the current block is high (e.g., when V>5), Y4=3, and when the variance V of the samples adjacent to the current block is low (e.g., when V<1), Y4=0.1. In addition, when the variance V of the samples adjacent to the current block is between "1" and "5", Y4=1.

[0181] Any combination of the first, second, third, fourth and fifth embodiments is possible.

[0182] For example:

[0183] The first embodiment in which the edge of the block of film grain samples is aligned on the edge of the current block can be combined with inserting an additional scaling factor in equation (Equation 2) or equation (Equation 3) as described in the second, third and fourth embodiments;

[0184] ·The second, third, fourth and fifth embodiments do not require that the edges of the blocks of film grain samples be aligned on the edges of the current block. In fact, just the bits per pixel value bpp, the block size or the CBF of the current block comprising the current samples is sufficient to implement the second, third and fourth embodiments respectively. In that case, the size of the blocks of film grain samples may be predefined and, for example, equal to 32×32 for luminance blocks and 16×16 for chrominance blocks. If the size of the current block is smaller than a predetermined size, an area of ​​the size of the current block may be randomly selected in the blocks of film grain samples. If the size of the current block is larger than a predetermined size, a combination of several blocks of film grain samples may be used, which combination is cropped if the size of the block is not a multiple of the predetermined size;

[0185] The second, third, fourth and fifth embodiments can be combined. For example, in that case, the equation (Equation 2) becomes:

[0186] Y′(x,y)=Y(x,y)+Y1.Y2.Y3.Y4.f(Y(x,y)).G(x,y) (Equation 12)

[0187] We have described a number of embodiments above. Features of these embodiments may be provided individually or in any combination. In addition, embodiments may include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types:

[0188] • A TV, set-top box, cell phone, tablet computer or other electronic device that implements at least one of the described embodiments.

[0189] • A TV, set-top box, cell phone, tablet computer or other electronic device that performs at least one of the described embodiments and displays (eg, using a monitor, screen or other type of display) the resulting picture.

[0190] A TV, set-top box, cellular phone, tablet computer, or other electronic device that tunes (eg, using a tuner) a channel to receive a signal comprising an encoded video stream and performs at least one of the described embodiments.

[0191] • A TV, set-top box, cell phone, tablet computer, or other electronic device that receives over the air (eg, using an antenna) a signal including an encoded video stream and performs at least one of the described embodiments.

Claims

1. A method comprising: Prior to reconstructing the current sample, obtaining (601) at least one characteristic of a current sample of the picture; determining (602) parameters of a film grain synthesis process to be applied to the current sample based on the at least one characteristic; storing (603) the determined parameters and information indicating the position of the current sample; reconstructing (604) the current sample; and, A film grain synthesis process is applied (605) to the current sample using the determined parameters. 2 . The method of claim 1 , wherein the at least one characteristic of the current sample is at least one characteristic of a current block including the current sample.

3. The method of claim 2, wherein the at least one characteristic of the current block comprises at least one of: information representing a shape of the current block, a position of the current block, a sum of bits used to encode all components of the current block, information representing a surface delimited by the current block, information indicating that a prediction residual of the current block includes at least one non-zero transform coefficient.

4. The method of claim 3, wherein in response to at least one characteristic of the current block including information representing a shape and a position of the current block, the parameters of the film grain synthesis process are the shape and position of the block including the film grain sample to be applied to the film grain sample of the current sample.

5. A method according to any preceding claim, wherein applying a film grain synthesis process to the current sample comprises adding a film grain value to the value of the current sample, the film grain value being obtained by weighting a value obtained using a film grain model by a first weighting factor, the first weighting factor being dependent on the value of the current sample or an average of samples of the current block.

6. A method according to claim 5 as appended to claim 3 or 4, wherein in response to at least one characteristic of the current block comprising the sum of bits used to encode all components of the current block, the value obtained using the film grain model is further weighted by a weighting factor that depends on the sum of bits used to encode all components of the current block.

7. A method according to claim 5 or claim 6 as dependent on claim 3 or 4, wherein in response to at least one characteristic of the current block including information representing a surface bounded by the current block, the value obtained using the film grain model is further weighted by a weighting factor, the weighting factor being dependent on the surface bounded by the current block.

8. A method according to claim 5, claim 6 or claim 7 as dependent on claim 3 or 4, wherein in response to at least one characteristic of the current block including information indicating that the prediction residual of the current block includes at least one non-zero transform coefficient, the value obtained using the film grain model is further weighted by a weighting factor, the weighting factor being dependent on the information indicating that the prediction residual of the current block includes at least one non-zero transform coefficient.

9. A device comprising an electronic circuit, the electronic circuit being configured to: Prior to reconstructing the current sample, obtaining (601) at least one characteristic of a current sample of the picture; determining (602) parameters of a film grain synthesis process to be applied to the current sample based on the at least one characteristic; storing (603) the determined parameters and information indicating the position of the current sample; reconstructing (604) the current sample; and, A film grain synthesis process is applied (605) to the current sample using the determined parameters.

10. The apparatus of claim 9, wherein the at least one characteristic of the current sample is at least one characteristic of a current block including the current sample.

11. The apparatus of claim 10, wherein the at least one characteristic of the current block comprises at least one of: information representing a shape of the current block, a position of the current block, a sum of bits used to encode all components of the current block, information representing a surface delimited by the current block, information indicating that a prediction residual of the current block includes at least one non-zero transform coefficient.

12. The apparatus of claim 11, wherein in response to at least one characteristic of the current block including information representing a shape and a position of the current block, the parameters of the film grain synthesis process are the shape and position of the block including the film grain sample to be applied to the film grain sample of the current sample.

13. An apparatus according to any preceding claim from claim 9 to claim 12, wherein applying a film grain synthesis process to a current sample comprises adding a film grain value to a value of the current sample, the film grain value being obtained by weighting a value obtained using a film grain model by a first weighting factor, the first weighting factor being dependent on the value of the current sample or an average of samples of the current block.

14. An apparatus according to claim 13 as dependent on claim 11 or 12, wherein in response to at least one characteristic of the current block comprising the sum of bits used to encode all components of the current block, the value obtained using the film grain model is further weighted by a weighting factor that depends on the sum of bits used to encode all components of the current block.

15. A device according to claim 13 or claim 6 as dependent on claim 3 or 4, wherein in response to at least one characteristic of the current block including information representing a surface bounded by the current block, the value obtained using the film grain model is further weighted by a weighting factor, the weighting factor being dependent on the surface bounded by the current block.

16. An apparatus according to claim 13 or claim 6 or claim 7 as dependent on claim 3 or 4, wherein in response to at least one characteristic of the current block including information indicating that the prediction residual of the current block includes at least one non-zero transform coefficient, the value obtained using the film grain model is further weighted by a weighting factor, the weighting factor being dependent on the information indicating that the prediction residual of the current block includes at least one non-zero transform coefficient.

17. A computer program comprising program code instructions for implementing the method according to any preceding claim from claims 1 to 8.

18. A non-transitory information storage medium storing program code instructions for implementing the method according to any preceding claim from claim 1 to claim 8.

Images