Metadata-assisted film grain removal

Abstract

Metadata-assisted film grain removal method and corresponding apparatus. Certain exemplary embodiments enable a video decoder to substantially completely remove film grain from a digital video signal that has undergone lossy video compression and then video decompression. Different embodiments may rely on only spatial domain grain removal processing, only time domain grain removal processing, or a combination of spatial and time domain grain removal processing. Both spatial and time domain grain removal processing may use metadata provided by a corresponding video encoder, the metadata including one or more parameters corresponding to digital film grain injected into the host video at the encoder. In order for different film grain injection formats to be accepted by the video decoder, signal pre-processing is used that is directed to providing an input to the film grain removal module of the video decoder that is compatible with the film grain removal method implemented therein.

Description

[Technical field]

[0001] 1. CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority from the following priority applications: U.S. Provisional Patent Application No. 63 / 332,332 (Docket No. D22003USP01), filed April 19, 2022, and European Patent Application No. 22168812.0 (Docket No. D22003EP), filed April 19, 2022, each of which is incorporated by reference in its entirety herein.

[0002] 2. Field of disclosure Various exemplary embodiments relate to image and video processing, and more particularly, but not exclusively, to digital film grain techniques. [Background technology]

[0003] 3. background On physical film, film grain is the random physical texture created by tiny metallic silver particles found on processed photographic celluloid. In digital photography, the visual and artistic effect of film grain can be simulated by adding a digital grain pattern to a digital image after the image is captured. Because digital film grain can be difficult to encode, for example, due to its (quasi-)random nature, video encoders may typically remove the film grain during encoding, and then, during playback, a corresponding video decoder may synthesize the film grain and add it back. Summary of the Invention [Means for solving the problem]

[0004] Various embodiments of metadata-assisted film grain removal methods and corresponding apparatus are disclosed herein. Certain exemplary embodiments enable a video decoder to substantially completely remove film grain from a digital video signal that has undergone lossy video compression and then video decompression. Different embodiments may rely on only spatial domain grain removal processes, only time domain grain removal processes, or a combination of spatial and time domain grain removal processes. Both spatial and time domain grain removal processes may use metadata provided by a corresponding video encoder, the metadata including one or more parameters corresponding to the digital film grain to be injected into the host video at the encoder. In order for different film grain injection formats to be accepted by the video decoder, signal pre-processing is used that is directed to providing an input to the film grain removal module of the video decoder that is compatible with the film grain removal method implemented therein.

[0005] According to an example embodiment, a video delivery system capable of film grain removal is provided, the system comprising: an input interface that receives an encoded bitstream including a compressed video bitstream and metadata, the compressed video bitstream having a sequence of images including digital film grain encoded therein, the metadata including one or more parameters corresponding to the digital film grain; and a video decoder configured to: decompress the compressed video bitstream to generate a respective decompressed representation of each image, each decompressed representation including a respective host image component and a respective film grain component; and a processor configured to perform the steps of: calculating a respective estimate of the respective film grain component for each decompressed representation based on the metadata; and removing the respective estimates from each decompressed representation to generate a corresponding estimate of the respective host image component.

[0006] According to another exemplary embodiment, a video delivery system capable of film grain removal is provided, the system comprising: an input interface for receiving an encoded bitstream including a compressed video bitstream and metadata, the compressed video bitstream having a sequence of images including digital film grain encoded therein, the metadata including one or more parameters corresponding to the digital film grain; and a processor configured to perform: decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each decompressed representation including a respective host image component and a respective film grain component; and temporally averaging the sequence of respective decompressed representations using a plurality of temporal sliding windows, each of the temporal sliding windows corresponding to a respective image frame pixel, at least some of the temporal sliding windows having different respective lengths selected based on the metadata.

[0007] According to yet another exemplary embodiment, there is provided a machine-implemented method for removing film grain from video data, the method including: receiving an encoded bitstream including a compressed video bitstream and metadata, where the compressed video bitstream has a sequence of images encoded therein including digital film grain, the metadata including one or more parameters corresponding to the digital film grain; decompressing the compressed video bitstream to generate a respective decompressed representation of each image, where each decompressed representation includes a respective host image component and a respective film grain component; calculating a respective estimate of the respective film grain component for each decompressed representation based on the metadata; and removing the respective estimates from each decompressed representation to generate a corresponding estimate of the respective host image component.

[0008] According to yet another exemplary embodiment, a non-transitory machine-readable medium is provided having program code encoded thereon, the program code, when executed by a machine, performing a method comprising: receiving an encoded bitstream comprising a compressed video bitstream and metadata, the compressed video bitstream having a sequence of images encoded therein comprising digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each decompressed representation comprising a respective host image component and a respective film grain component; calculating a respective estimate of the respective film grain component for each decompressed representation based on the metadata; and removing the respective estimate from each decompressed representation to generate a corresponding estimate of the respective host image component. [Brief description of the drawings]

[0009] Other aspects, features, and advantages of the various disclosed embodiments will become more fully apparent from the following detailed description and the accompanying drawings, by way of example.

[0010] [Figure 1] 1 illustrates an example process for a video delivery pipeline in which at least some embodiments may be implemented.

[0011] [Diagram 2] 2 is a block diagram illustrating a video encoder that can be used in the process of FIG. 1 according to one embodiment.

[0012] [Diagram 3] 2 is a block diagram illustrating a video decoder that can be used in the process of FIG. 1 according to one embodiment.

[0013] [Figure 4] 4 is a process flow diagram illustrating a spatial domain particle removal process that may be implemented in the video decoder of FIG. 3 according to one embodiment.

[0014] [Diagram 5] 4 graphically illustrates exemplary film grain removal results that may be achievable using a scaled Gaussian filter in the video decoder of FIG. 3 according to first embodiments.

[0015] [Figure 6] 4 graphically illustrates exemplary film grain removal results that may be achievable using a scaled Gaussian filter in the video decoder of FIG. 3 according to second embodiments.

[0016] [Figure 7]4 graphically illustrates exemplary film grain removal results that may be achievable using a scaled Gaussian filter in the video decoder of FIG. 3 according to third embodiments.

[0017] [Figure 8] 4 graphically illustrates exemplary film grain removal results that may be achievable using a scaled Gaussian filter in the video decoder of FIG. 3 according to fourth embodiments.

[0018] [Figure 9] 4 is a process flow diagram illustrating a time-domain grain removal process that may be implemented in the video decoder of FIG. 3 according to one embodiment.

[0019] [Figure 10] 4 illustrates an example of an intensity-dependent scaling factor that can be accommodated in the video decoder of FIG. 3 using signal pre-processing, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The present disclosure and aspects thereof may be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces, as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​various aspects of the present disclosure and is in no way intended to limit the scope of the present disclosure.

[0021] In the following description, numerous details are set forth, such as optical device configurations, timing, operation, etc., to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to one of ordinary skill in the art that these specific details are merely examples and are not intended to limit the scope of the present application.

[0022] Additionally, while this disclosure will primarily focus on examples in which the various circuits are used in digital projection systems, it will be understood that these are merely examples. It will be further understood that the disclosed systems and methods can be used in any device that needs to project light, such as cinema, consumer, and other commercial projection systems, heads-up displays, virtual reality displays, etc. The disclosed systems and methods may be implemented in additional display devices having OLED displays, LCD displays, quantum dot displays, etc.

[0023] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest gray (black) to brightest white (highlight). In this sense, DR relates to "scene-based" intensity. DR may also relate to the ability of a display device to adequately and / or approximately render an intensity range of a particular width. In this sense, DR relates to "display-based" intensity. Unless a particular meaning is explicitly specified as having special importance at any point in the description herein, it should be presumed that the term may be used in either sense, e.g., interchangeably.

[0024] As used herein, the term "high dynamic range" (HDR) refers to a DR width that spans about 14-15 orders of magnitude of the human visual system (HVS). In practice, the DR at which humans can simultaneously perceive a wide range of intensities may be somewhat truncated for HDR. As used herein, the terms "enhanced dynamic range" (EDR) or "visual dynamic range" (VDR), individually or interchangeably, may refer to the DR perceivable in a scene or image by the human visual system, including eye movements, taking into account any light adaptation changes across the scene or image.

[0025] In practice, an image includes one or more color components (e.g., luma Y and chroma Cb and Cr), with each color component represented with n bits of precision per pixel (e.g., n=8). Using linear luminance encoding, images with n≦8 may be considered standard dynamic range images, while images with n>8 (e.g., color 24-bit JPEG images) may be considered enhanced dynamic range images. EDR and HDR images may also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

[0026] As used herein, the term "metadata" refers to any auxiliary information transmitted as part of an encoded bitstream that aids a decoder in rendering a corresponding image or images. For television broadcasting and video streaming, video metadata may be used to provide side information about a particular video and audio stream or file. The metadata is either embedded directly in the video or included as a separate file within a container such as MP4 or MKV. The metadata may include information about an entire video stream or file, or about a particular video frame. Metadata created by cameras, encoders, and other video processing elements (see, e.g., 115, 120 in FIG. 1) may include, but is not limited to, timestamps, video resolution, digital film grain parameters, color space or gamut information, reference display parameters, auxiliary signal parameters, file size, closed captions, audio language, ad insertion points, color space, error messages, and the like. Additional examples of metadata relevant to the disclosed embodiments are described herein below.

[0027] Most consumer desktop displays have a brightness of 200-300 cd / m 2 or nits of luminance. Most consumer HDTVs are in the 300-500 nits range, with newer models supporting 1000 nits (cd / m 2 ). Such conventional displays are representative of low dynamic range (LDR), also referred to as standard dynamic range (SDR), with respect to HDR or EDR. As the availability of HDR content increases due to advances in both image capture devices (e.g., cameras) and HDR displays (e.g., the PRM-4200 Professional Reference Monitor from Dolby Laboratories), HDR content can be color graded and displayed on HDR displays that support a higher dynamic range (e.g., 1,000 nits to 5,000 nits or more).

[0028] Video Encoding According to an Exemplary Embodiment FIG. 1 illustrates an exemplary process of a video delivery pipeline (100) showing various stages from video capture to video content display, according to one embodiment. A sequence of video frames (102) may be captured or generated using an image generation block (105). The video frames (102) may be captured digitally (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide video data (107). Alternatively, the video frames (102) may be captured on film by a film camera. The film may then be converted to a digital format (107) to provide the video data.

[0029] In the production phase (110), the video data (107) may be edited to provide a video production stream (112). The data in the video production stream (112) may then be provided to a processor (or one or more processors, such as a central processing unit CPU) in a post-production block (115) for post-production editing. The post-production editing in block (115) may include, for example, adjusting or modifying the color or brightness in certain areas of the image to improve image quality or achieve a particular look of the image according to the creative intent of the video creator. This part of the post-production editing may be referred to as "color timing" or "color grading". Other editing (e.g., scene selection and ordering, image cropping, adding computer-generated visual special effects, etc.) may be performed in block (115) to generate a "final" version (117) of the production for distribution. During post-production editing (115), the video images may be viewed on a reference display (125).

[0030] Following post-production (115), the final version (117) of the video data may be delivered to an encoding block (120) for delivery to downstream decoding and playback devices such as television sets, set-top boxes, movie theaters, etc. In some embodiments, the encoding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate an encoded bitstream (122). Some methods described herein below may be executed by a corresponding processor in the encoding block (120). At the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a corresponding decoded signal (132) that represents a copy or approximation of the signal (117). The receiver may be attached to a target display (140), which may have characteristics that are somewhat or completely different from the reference display (125). In such cases, a display management block (135) may be used to map the decoded signal (132) to the characteristics of the target display (140) by generating a display-mapped signal (137). Some methods described herein below may be performed by the decode unit (130) and / or the display management block (135). Depending on the embodiment, the decode unit (130) and the display management block (135) may include individual processors or may be based on a single integrated processing unit.

[0031] As already mentioned above, film grain can provide an effective way to improve the appearance of digital video in terms of aesthetics and sharpness. Film grain can also be used to reduce banding artifacts and / or to mask compression artifacts. Film grain may be added in post-production (115). Conventionally, film grain is removed in an encoder in a coding block (120) and the resulting "clean" video signal is compressed, for example using a lossy video compression code, to generate a coded bitstream (122). In a corresponding decoder, the coded bitstream (122) is decoded and film grain is added using a decoding unit (130) to generate a decoded signal (132) including film grain.

[0032] In some use cases, the film grain may not be removed at the encoder in the encoding block (120), or alternatively, basic film grain may be added there to the "clean" video signal before video compression. This type of processing may be performed, for example, to ensure that film grain features are present in the encoded bitstream (122) and allow a "simpler" decoder to provide playback including film grain, even if the decoder itself is not capable of synthesizing and overlaying the film grain. However, if the decoder has film grain synthesis capabilities, the basic film grain of the encoded bitstream (122) may be removed at the decoder using the decode unit (130), where further video processing may be applied. In some use cases, such further video processing may include adding another (e.g., adaptive) type of film grain according to the viewing environment corresponding to the target display (140).

[0033] In one example of the above use case, the base layer of the bitstream (122) carries an SDR signal with SDR-specific film grain injected therein. If the received bitstream is to be converted to HDR, the SDR-specific film grain may be removed in the decode unit (130) and replaced by a different (e.g., HDR-specific) film grain.

[0034] In another example of the above use case, the bitstream (122) may have film grain injected therein to reduce the banding artifacts mentioned above. If the decode unit (130) operates to convert an 8-bit based profile to a 10-bit based profile, the corresponding processing may include removing the existing film grain and then applying a de-banding filter. The resulting filtered signal may be further processed to, among other things, insert a different film grain.

[0035] Exemplary embodiments disclosed herein are generally directed to performing film grain removal from a received bitstream (122) in a decode unit (130). Some embodiments may rely solely on a spatial domain grain removal process or solely on a temporal domain grain removal process. Some other embodiments may rely on both spatial and temporal domain grain removal processes. In exemplary embodiments, the grain removal process is performed using metadata of the received bitstream (122). The metadata may provide useful information regarding the film grain of the received bitstream, thereby facilitating the grain removal process in the decode unit (130).

[0036] In an example embodiment, the pipeline blocks (115, 120) may generate film grain for the encoded bitstream (122) using a reproducible film grain model, where a random or pseudorandom seed and a selected intensity modulation function may be used. The seed information and film grain parameters of the model may be transmitted as metadata in the bitstream 122. As a result, a decoder in the decode unit (130) is provided with certain details of the film grain pattern of the stream, which the decoder may beneficially exploit to generate an accurate approximation of a corresponding "clean" (i.e., grain-free) video signal efficiently, e.g., as described in more detail below.

[0037] FIG. 2 is a block diagram illustrating a video encoder (200) according to one embodiment. The encoder (200) may be used, for example, to implement the encoding block (120) of the pipeline (100) (see also FIG. 1). For illustrative purposes, and without implying any limitation, FIG. 2 uses reference numbers (117) and (122) to better illustrate the exemplary relationships between the block diagrams of FIG. 1 and FIG. 2. Those skilled in the art will readily appreciate that alternative configurations for the encoder (200) within the encoding block (120) are also possible.

[0038] As shown in Figure 2, the encoder (200) operates to convert an input video signal (117) into an output bitstream (122). In this embodiment, the input video signal (117) does not have film grain encoded therein. The processing performed in the encoder (200) causes the corresponding video signal encoded in the output bitstream (122) to have digital film grain therein. Additionally, the encoder (200) typically causes the output bitstream (122) to carry metadata corresponding to and / or characterizing the digital film grain.

[0039] For each image in the output bitstream (122), the encoder (200) generates a film grain image (220) using a film grain model (210). The film grain model (210) is also typically "known" to a corresponding video decoder (see, e.g., FIG. 3). The specific parameter values ​​of the film grain model (210) used by the encoder (200) in generating the film grain image (220) may typically be communicated to the corresponding video decoder as film grain model metadata (212). Thus, the corresponding video decoder can compute an exact copy of the film grain image (220) (see also FIG. 3).

[0040] The film grain injection block (240) of the encoder (200) operates to modulate the noise intensity of the film grain image (220) on a pixel-by-pixel basis based on the corresponding intensity of the input video signal (117). The resulting modulated film grain image is then added to the host image to generate a corresponding film grain injected image. Such a sequence of film grain injected images (242) is compressed in a video compression module (250) of the encoder (200), thereby generating a compressed video bitstream (252). In some embodiments, the video compression module (250) may perform lossy compression according to a video compression standard. The metadata generator (230) having parameter inputs (222, 244, 254) shown in FIG. 2 generates film grain removal metadata (232) including one or more parameters that may be used for efficient and effective film grain removal in a corresponding video decoder. Examples of such metadata (232) are described in more detail below. The combiner module (260) operates to appropriately combine the metadata (212, 232) with the compressed video bitstream (252) to generate the output bitstream (122).

[0041] 3 is a block diagram illustrating a video decoder (300) according to one embodiment. The decoder (300) may be used, for example, to implement the decode unit (130) of the pipeline (100) (see also FIG. 1). The decoder (300) operates to convert a received bitstream (122) into a video signal (342). In one exemplary embodiment, the video signal (342) has the above-mentioned digital film grain substantially completely removed therefrom, and thus may represent a relatively accurate approximation of the source video signal (117) (see also FIG. 2).

[0042] The decoder (300) comprises a separator module (310) configured to appropriately extract the above-mentioned metadata (212, 232) and the compressed video bitstream (252) from the received bitstream (122). As already indicated above, the film grain model (210) is "known" to the decoder (300) (e.g., stored in the memory of the decoder (300)). This knowledge, together with the film grain model metadata (212) extracted by the separator module (310), allows the decoder (300) to recalculate the film grain image (220) as shown in FIG. 3. The film grain image (220) may be referred to as a simulated film grain image.

[0043] The decoder (300) further comprises a video decompression module (320) configured to appropriately decompress the compressed video bitstream (252) received from the separator module (310), thereby generating a decompressed video signal (330). It should be noted that if lossy compression is used in the video compression module (250) of the encoder (200), the decompressed video signal (330) may differ from the sequence (242). Since the film grain injection performed in the film grain injection block (240) of the encoder (200) involves modulating the film grain image (220) with luma, the actual injected film grain does not have to be determined from the decompressed video signal (330) but may instead be estimated. This estimation is performed by a film grain removal module (340) of the decoder (300) and generally relies on the recalculated film grain image (220), the metadata (232) extracted by the separator module (310) from the received bitstream (122), and the decompressed video signal (330). An exemplary embodiment of a film grain removal method implemented in the film grain removal module (340) is described in more detail below.

[0044] Film Grain Injection Process Let the i-th pixel of the j-th host image frame be s ji and the corresponding film grain pixel is n ji The luma modulation function is expressed as f(). The modulation function f(s ji ) is typically a host signal ji The film grain injected pixel v ji teeth,

number

number

number

[0045] Film Grain Removal Process: Spatial Domain 4 is a process flow diagram (400) illustrating a spatial domain grain removal process that may be implemented in a decoder (300) according to one embodiment. The process (400) includes a processing block (440) where a decompressed video signal (330) is converted to an output video signal (342) (see also FIG. 3). The processing block (440) also receives inputs from the processing blocks (420, 430) that aid in the conversion process. In general, the grain removal process (400) operates by converting a compressed film grain injection signal (v ji 〕 (see also equations (1)-(2)) to obtain a host signal s ji Depending on the embodiment, the host signal that is actually restored in the decoder (300) may be a copy of the host signal that was used to generate the bitstream (122) in the encoder (200), or may be a relatively accurate estimate of that host signal.

[0046] The input provided by the processing block (420) to the processing block (440) is based on the luma modulation function f() (410) described above. Relevant parameters describing the function f() may be provided to the decoder (300), for example, as part of the film grain model (210) and / or metadata (212) (see also Figures 2 and 3). In one exemplary embodiment, the processing block (420) operates to calculate a local first-order polynomial approximation of the function f(). This may be used, for example, in the grain removal process implemented in the processing block (440), as described in more detail below.

[0047] The input provided by processing block (430) to processing block (440) is an estimate of the distorted film grain image (220), where the distortion is typically caused by the sequential application of a lossy compression (250) and a corresponding decompression (320) (see also Figures 2 and 3). An example of how to calculate such an estimate is described in more detail below.

[0048] Lossless Compression When the video compression module (250) of the encoder (200) applies lossless compression, the function g( ) defined by equations (2)-(3) is effectively a bypass function with the following properties:

number

[0049] The linear function f() can be expressed as follows:

number

number

number

[0050] When the luma modulation function f() is nonlinear, a closed-form solution for film grain removal such as equation (8) may not be available. However, in view of equation (8), an approximate solution can be constructed based on a local linear approximation of the function f(), which can be expressed as follows:

number

number

[0051] Local linear polynomial parameters (a x ,bx ) into vector m x Presented as:

number

number

number

[0052] Similar to equation (7), equation (1) can now be approximated by equation (16) as follows:

number

number

number

number

[0053] Lossy Compression If the video compression module (250) of the encoder (200) applies lossy compression, the above-described film grain removal process adapted to lossless compression may not produce optimal results. Note, for example, that lossy compression may cause signal distortions, e.g., both the host video signal and the film grain component may lose some of their higher frequency content. Since the above-described "lossless" film grain removal does not take into account the distortions caused by such compression, residual film grain may be very noticeable if such removal is applied in conjunction with lossy compression.

[0054] To address this issue, equation (18) can be further modified to arrive at equation (19).

number

number

[0055] According to one exemplary embodiment, the tilde n ji The values ​​of are estimated using a scaled Gaussian filter G(), which is expressed as:

number

[0056] 5-8 graphically illustrate exemplary film grain removal results that can be achieved using the scaled Gaussian filter of Equation (21). More specifically, each of Figs. 5-7 illustrates the effect of the scaled Gaussian filter on the film grain removal results achieved using the scaled Gaussian filter of Equation (21) for different bit rates. j5 shows the dependence of the peak signal-to-noise ratio (PSNR) on σ. FIG. 5 corresponds to a bit rate of 20 Mbps. FIG. 6 corresponds to a bit rate of 10 Mbps. FIG. 7 corresponds to a bit rate of 5 Mbps. σ j The PSNR value corresponding to σ = 0 represents the result where no filter is applied. j = 0.75 and the bit rate is 20 Mbps. j 5-8 show in graph form the PSNR dependence on .times. ...

[0057] According to another exemplary embodiment, n with a tilde ji is estimated using an FIR filter implemented according to equation (22).

number

number

[0058] Optimal filter coefficient w jk can be obtained, for example, as follows: ji and s ji The value of is known and can be expressed as:

number

number

[0059] An approximate solution to the problem in equation (26) can be obtained using the LMS algorithm, for example, as follows: jk Organize the sets of in vector form.

number

number

number

number

number

[0060] The PSNR gain of the FIR filter embodiment generally increases with increasing filter size, however, this improvement may come at the cost of additional implementation overhead.

[0061] With respect to the guided filter embodiment, it is observed that the most noticeable residual film grain noise may be present in image regions that have relatively flat and / or smooth image patterns. For example, pixel values ​​in flat regions may be approximately constant, and local variations may be primarily due to added or incompletely / poorly removed film grain. As a preliminary observation, in such flat regions, the estimated compressed film grain noise may be

number

number

[0062] Here, the original film grain pattern (n ji ) and reconstructed film grain (^n ji ) is τ ji Therefore, the following holds:

number

number

number

[0063] Then, the average of the local linear model constants in a neighborhood can be calculated, for example as follows:

number

number

number

number

number

[0064] Note that in general, the guided filter embodiment may fall between the Gaussian filter embodiment and the FIR filter embodiment in terms of PSNR gain, however, since the guided filter embodiment works best for flat and smooth image regions, the improvement subjectively perceptible by an observer may typically be more noticeable under the guided filter approach.

[0065] Film Grain Removal Process: Time Domain In an exemplary embodiment, film grain may have a zero-mean property, i.e., an image with film grain and an original image have the same DC value. As a result, the averaged pixel value for a given pixel location in a static scene tends to average the film grain back to "zero." Certain embodiments of the time domain grain removal disclosed herein are designed to take advantage of this property to remove film grain.

[0066] 9 is a process flow diagram (900) illustrating a time-domain particle removal process that can be implemented in a decoder (300) according to one embodiment. For purposes of illustration, and without implying any limitation, the process (900) takes as its input the sequence of images (342) generated by the process (400). j-L ,…,342 j ,…,342 j+L), however, embodiments of the process (900) are not so limited. For example, in another embodiment, the process (900) can operate directly on the sequence of images (330) (see also FIG. 3). The sliding window length L can depend on the maximum amplitude of the film grain pattern. In general, larger values ​​of L can be more beneficial when the maximum amplitude is higher. In operation, the processing block (910) of the process (900) converts an input sequence of images (342) (FIG. 4) or images (330) (FIG. 3) into a corresponding output sequence of images (342′), e.g., as described below. The film grain pattern in the output sequence is less pronounced than in the input sequence.

[0067] According to an exemplary embodiment, the jth frame (342 j ) for the i-th spatially filtered pixel in the temporal sliding window [w ji L ,w ji H ], the time domain filtering of the process (900) is carried out to obtain the next signal ^s for the sequence (342'). ji T Gives.

number

number

[0068] w ji L The value of is, for example, the sequence (342 j-1 ,342 j-2 ,…) by iteratively checking the condition expressed by inequality (45).

number

number

[0069] w ji H The value of is similarly the sequence (342 j+1 ,342 j+2 ,…) by iteratively checking the condition expressed by inequality (47).

number

number

[0070] Once w ji L and w ji H Once the value of is determined as above, the time averaging in processing block (910) can be performed according to equation (42). Note that the time averaging described above is a pixel-based operation. The buffer size D L and D. R may be provided to the decoder (300) as part of the metadata (232).

[0071] Signal Preprocessing for Alternative Film Grain Models Some embodiments may be adapted to be compatible with alternative film grain models, i.e., film grain models different from those described above. Such embodiments generally rely on signal pre-processing to provide a pre-processed signal that is compatible with the grain removal process described above. As a non-limiting example of such pre-processing, the present specification describes an exemplary pre-processing applicable to the MPEG film grain model defined in the corresponding standard(s).

[0072] An example of an MPEG film grain model is a piecewise model according to pixel intensity values. In each piece, different film grain model parameters may be used, e.g., different cutoff frequencies, different scaling factors for scaling the film grain, etc. If there are T such pieces in the jth frame, and the division points for the pieces are {p j t}. The film grain image in the t-th piece is represented as {n ji t}, and the scaling factor for the t-th piece is a j t This is expressed as:

[0073] Fig. 10 shows an example of an intensity-dependent scaling factor that can be accommodated using the pre-processing described above according to an embodiment. In Fig. 10, it can be seen that the scaling factor can take several discrete values ​​from the range between 0 and 30. In this case, for a film grain injected pixel v ji teeth, v ji =s ji +a j t ·n ji t (49) Here, p j t ≦s ji <p j t+1 It is.

[0074] Comparing equation (49) with equation (1), the piecewise scaling factor (a j t ) is the nonlinear modulation function f(s ji ) may be considered as a special use case of the nonlinear modulation function f(s ji The methodology described above with reference to A(s) is applicable to this particular use case. ji ) and B(s ji ) (see also equations (12) to (15)) can be calculated, for example, as follows: A(s ji )=a j t Here, p j t ≦s ji <p j t+1 (50) B(s ji )=0, where p j t ≦s ji <p j t+1 (51)

[0075] On the other hand, the different cutoff frequencies of the noise pattern in each piece add an additional dimension that needs to be addressed in pre-processing. ji t Complete film grain image from {n ji Once a complete film grain image has been constructed through pre-processing, subsequent grain removal may rely on one or more of the grain removal processes described above.

[0076] To construct a complete film grain image, v ji Consider the i-th pixel of (see equation (1)). If the i-th pixel value is not close to one of the boundaries of different intensity pivot points (see, for example, FIG. 10), then the corresponding v ji Values ​​in the range p j t +Δ <v ji <p j t+1 If it is within −Δ, then the film grain value for the complete film grain image can follow equation (52). n ji =n ji t (52) where Δ denotes the film grain amplitude, which is typically relatively small.

[0077] However, if the pixel value is relatively close to the piece boundary, e.g., p j t -Δ <v ji <p j t +Δ, the following ambiguity needs to be resolved: v ji =s ji (α) +a j t-1 ·n ji t-1 (53) v ji =s ji(β) +a j t ·n ji t (54) Here, a j t-1 ·n ji t-1 and a j t ·n ji t can be determined based on the received bitstream and metadata. ji indicates the decoded pixel value. However, s ji The determination of the host pixel value for can be done in two different options, namely, s defined by equations (53) and (54). ji (α) or s ji (β) needs to be dealt with.

[0078] In one exemplary embodiment, there are two possible options s ji (α) or s ji (β) The choice between is made by choosing the local neighborhood / patch (Ω i ), the results of preprocessing in the patch are expected to be smoother (e.g., exhibit lower residual film grain noise) than the results corresponding to the incorrect one of the two alternatives. Based on this observation, an example embodiment is configured to calculate a standard deviation for each of the two alternatives in such a patch. The standard deviation is expected to be smaller for the correctly selected one of the two alternatives.

[0079] Equations (55) to (56) are the above-mentioned standard deviation ω i (α) and ω i (β) Provide a formula for the calculation of

number

number

[0080] According to example embodiments disclosed above, for example in the Overview section and / or with respect to any one or any combination of some or all of FIGS. 1-10, a video processing apparatus may include an input interface (e.g., 310 of FIG. 3) for receiving an encoded bitstream (e.g., FIGS. 1, 122 of FIG. 3) including a compressed video bitstream (e.g., 252 of FIG. 3) and metadata (e.g., 212, 232 of FIG. 3), the compressed video bitstream having encoded therein a sequence of images including digital film grain, the metadata including one or more parameters corresponding to the digital film grain; (A) an input interface (e.g., 310 of FIG. 3) for receiving the compressed video bitstream (e.g., 310 of FIG. 3) including a compressed video bitstream (e.g., 252 of FIG. 3) and metadata (e.g., 212, 232 of FIG. 3), the compressed video bitstream having encoded therein a sequence of images including digital film grain, the metadata including one or more parameters corresponding to the digital film grain; 3 ) to generate a respective decompressed representation of each image (e.g., 330 of FIG. 3 ), each decompressed representation including a respective host image component and a respective film grain component; (B) calculating, based on the metadata, a respective estimate of the respective film grain component for each decompressed representation; and (C) removing (e.g., at 340 of FIG. 3 ) the respective estimate from each decompressed representation to generate a corresponding estimate of the respective host image component (e.g., 342 of FIG. 4 ).

[0081] In some embodiments of the above apparatus, the processor is further configured to calculate (e.g., at 210 in FIG. 3 ) a film grain image (e.g., 220 in FIG. 3 ) based on the metadata, and to calculate the respective estimates using the film grain image.

[0082] In some embodiments of any of the above devices, the processor is further configured to adjust (eg, at 430 in FIG. 4) the film grain image to account for distortion caused by at least one of video compression and decompression.

[0083] In some embodiments of any of the above devices, the processor is configured to adjust the film grain image using a Gaussian filter (e.g., as shown in Figures 5-8).

[0084] In some embodiments of any of the above devices, the processor is configured to adjust the film grain image using a finite impulse response filter (eg, according to equations (22)-(31)).

[0085] In some embodiments of any of the above devices, the processor is configured to adjust the film grain image using a guided filter (eg, according to equations (32)-(41)).

[0086] In some embodiments of any of the above devices, the processor is further configured to calculate a standard deviation corresponding to the estimated host image pixel values ​​for the patch of image pixels (e.g., according to equations (55)-(56)), and to calculate a film grain image based on the standard deviation (e.g., according to equation (58)).

[0087] In some embodiments of any of the above devices, the processor is further configured to calculate (e.g., at 420 in FIG. 4 ) a first-order polynomial approximation of the nonlinear luma modulation function (e.g., f() in Equation (1)) and to calculate the respective estimates using said approximation.

[0088] In some embodiments of any of the above apparatus, the processor is further configured to calculate the approximation using metadata.

[0089] In some embodiments of any of the above devices, the processor is further configured to calculate the respective estimates using a pre-computed look-up table (e.g., 420 in FIG. 4) that stores parameters of a first-order polynomial approximation of the nonlinear luma modulation function (e.g., f() in Equation (1)).

[0090] In some embodiments of any of the above apparatus, the processor is further configured to temporally average the sequence of each decompressed representation using a plurality of temporal sliding windows (e.g., at 910 in FIG. 9 ; according to equation (42)), each of the temporal sliding windows corresponding to a respective image frame pixel, at least some of the temporal sliding windows having different respective lengths (e.g., selected according to equations (45), (47)).

[0091] In some embodiments of any of the above apparatus, the processor may further include a processor that performs the metadata (e.g., τ A and τ B ) the length of the temporal sliding window.

[0092] In some embodiments of any of the above devices, the device further includes a video encoder (e.g., 120 of FIG. 1 ) comprising an output interface (e.g., 260 of FIG. 2 ) for outputting an encoded bitstream for the video encoder, and a video compression module (e.g., 250 of FIG. 2 ) for generating a video bitstream compressed using lossy compression in accordance with a video compression standard.

[0093] According to another exemplary embodiment disclosed above, e.g., in the Overview section and / or with respect to any one or any combination of some or all of FIGS. 1-10, an input interface (e.g., 310 of FIG. 3) for receiving an encoded bitstream (e.g., 122 of FIGS. 1, 3) including a compressed video bitstream (e.g., 252 of FIG. 3) and metadata (e.g., 212, 232 of FIG. 3), the compressed video bitstream having encoded therein a sequence of images including digital film grain, and the metadata including one or more parameters corresponding to the digital film grain, decompressing (e.g., at 320 of FIG. 3 ) the compressed video bitstream to generate a respective decompressed representation (e.g., at 330 of FIG. 3 ) of each image, each decompressed representation including a respective host image component and a respective film grain component; temporally averaging (e.g., at 910 of FIG. 9 ; according to equation (42)) a sequence of the respective decompressed representations using a plurality of temporal sliding windows, each of the temporal sliding windows corresponding to a respective image frame pixel, at least some of the temporal sliding windows being independent of metadata (e.g., τ in equations (43) and (44)). A and τ B In one embodiment, an apparatus is provided that includes a video decoder (e.g., 130 of FIG. 1 ) having a processor (e.g., 320, 340 of FIG. 3 ) configured to perform steps (a) to (c) the first and second inputs of the first and second inputs of the second input, the first inputs having different respective lengths selected based on (e.g., in accordance with equations (45), (47)).

[0094] In some embodiments of the above apparatus, the apparatus further comprises a video encoder (e.g., 120 of FIG. 1 ) comprising an output interface (e.g., 260 of FIG. 2 ) for outputting an encoded bitstream for the video encoder, and a video compression module (e.g., 250 of FIG. 2 ) for generating a video bitstream compressed using lossy compression in accordance with a video compression standard.

[0095] According to yet another exemplary embodiment disclosed above, for example in the Summary section and / or with respect to any one or any combination of some or all of FIGS. 1-10, there is provided a machine-implemented method for removing film grain from video data, the method comprising: (A) receiving an encoded bitstream (e.g., 122 of FIGS. 1, 3) including a compressed video bitstream (e.g., 252 of FIG. 3) and metadata (e.g., 212, 232 of FIG. 3), the compressed video bitstream having encoded therein a sequence of images including digital film grain, the metadata including one or more digital film grains corresponding to the digital film grain; or a plurality of parameters; (B) decompressing (e.g., at 320 of FIG. 3 ) the compressed video bitstream to generate a respective decompressed representation of each image (e.g., at 330 of FIG. 3 ), each decompressed representation including a respective host image component and a respective film grain component; (C) calculating, based on the metadata, a respective estimate of the respective film grain component for each decompressed representation; and (D) removing (e.g., at 340 of FIG. 3 ) the respective estimate from each decompressed representation to generate a corresponding estimate of the respective host image component (e.g., 342 of FIG. 4 ).

[0096] In some embodiments of the above method, the method further includes calculating (e.g., at 210 in FIG. 3 ) a film grain image (e.g., 220 in FIG. 3 ) based on the metadata, and wherein the calculating of the respective estimates includes using the film grain image.

[0097] In some embodiments of any of the above methods, the method further includes calculating (e.g., at 420 of FIG. 4) a first order polynomial approximation of the nonlinear luma modulation function (e.g., f() of FIG. 1 ), and calculating the respective estimates includes using the approximation.

[0098] In some embodiments of any of the above methods, the method further includes temporally averaging the sequence of each decompressed representation using a plurality of temporal sliding windows (e.g., at 910 of FIG. 9 ; according to equation (42)), each of the temporal sliding windows corresponding to a respective image frame pixel, at least some of the temporal sliding windows having different respective lengths (e.g., selected according to equations (45), (47)).

[0099] In some embodiments of any of the above methods, the compressed video bitstream is generated using lossy compression according to a video compression standard.

[0100] According to yet another exemplary embodiment disclosed above, for example in the Summary section and / or with respect to any one or any combination of some or all of Figures 1-10, there is provided a non-transitory machine-readable medium having program code encoded thereon that, when executed by a machine, causes the machine to perform a method. The method includes: (A) receiving an encoded bitstream (e.g., 122 of FIGS. 1, 3) including a compressed video bitstream (e.g., 252 of FIG. 3) and metadata (e.g., 212, 232 of FIG. 3), where the compressed video bitstream has encoded therein a sequence of images including digital film grain, and where the metadata includes one or more parameters corresponding to the digital film grain; (B) decompressing (e.g., at 320 of FIG. 3) the compressed video bitstream to generate a respective decompressed representation (e.g., 330 of FIG. 3) of each image, where each decompressed representation includes a respective host image component and a respective film grain component; (C) calculating a respective estimate of the respective film grain component for each decompressed representation based on the metadata; and (D) removing (e.g., at 340 of FIG. 3) the respective estimate from each decompressed representation to generate a corresponding estimate of the respective host image component (e.g., 342 of FIG. 4).

[0101] With respect to the processes, systems, methods, heuristics, etc. described herein, steps of such processes, etc. are described as occurring according to a certain ordered sequence, but it should be understood that such processes can be implemented with the described steps occurring in orders other than the order described herein. Furthermore, it should be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should not be construed as limiting the scope of the claims in any way.

[0102] Therefore, it should be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided will be apparent upon reading the above description. The scope should not be determined with reference to the above description, but instead with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology described herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that this application is capable of modification and variation.

[0103] All terms used in the claims are intended to be given their broadest reasonable interpretation and their ordinary meaning as understood by one skilled in the art described herein, unless expressly indicated to the contrary herein. In particular, the use of singular articles such as "a," "the," "said," etc., should be read as describing one or more of the indicated elements, unless the claim describes an explicit limitation to the contrary.

[0104] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it will be seen that various features are grouped together in various embodiments for the purpose of improving the flow of the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

[0105] Although the present disclosure includes reference to exemplary embodiments, the specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments, as well as other embodiments within the scope of the present disclosure that are apparent to those skilled in the art to which the present disclosure pertains, are deemed to be within the principles and scope of the present disclosure, as expressed, for example, in the following claims.

[0106] Some embodiments may be implemented as circuit-based processes, including possible implementations on a single integrated circuit.

[0107] Some embodiments may be embodied in the form of methods and apparatus for carrying out those methods. Some embodiments may also be embodied in the form of program code stored on a tangible medium, such as a magnetic recording medium, an optical recording medium, a solid-state memory, a floppy disk, a CD-ROM, a hard drive, or any other non-transitory machine-readable storage medium, which when loaded into a machine, such as a computer, and executed by the machine, the machine becomes an apparatus for carrying out the patented invention. Some embodiments may also be embodied in the form of program code stored on a non-transitory machine-readable storage medium, including, for example, being loaded into and / or executed by a machine, which when loaded into a machine, such as a computer or a processor, and executed by the machine, the machine becomes an apparatus for carrying out the patented invention. When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0108] Unless expressly stated to the contrary, each numerical value and range should be construed as being approximation, as if the word "about" or "approximately" preceded the value or range.

[0109] Use of figure numbers and / or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate claim interpretation, and such use should not be construed as necessarily limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0110] Although elements in the following method claims are described in a particular order with corresponding labeling, the elements are not necessarily intended to be limited to being implemented in that particular order, unless the claim recitations imply a particular order for implementing some or all of the elements.

[0111] Reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of the present disclosure. The appearances of the phrase "in an embodiment" in various places in this specification do not necessarily all refer to the same embodiment, and separate or alternative embodiments are not necessarily exclusive of other embodiments. The same applies to the term "implementation."

[0112] Unless otherwise specified herein, the use of the ordinal adjectives "first," "second," "third," etc. to refer to one object of a plurality of similar objects merely indicates that different instances of such similar objects are being referred to, and does not imply that the similar objects so referred to must be in a corresponding order or sequence in time, space, ranking, or in any way whatsoever.

[0113] Unless otherwise specified herein, in addition to its plain meaning, the conjunction "if" may also or alternatively be interpreted to mean "when" or "upon" or "response to determining" or "response to detecting," which interpretation may depend on the particular context to which it corresponds. For example, the phrase "if determined" or "if [stated condition] is detected" may be interpreted to mean "in determining" or "in response to determining" or "in detecting [stated condition or event]" or "in response to detecting [stated condition or event]."

[0114] Also, for purposes of this document, the terms "couple," "coupled," "coupled," "connect," "connection," or "connected" refer to any manner known in the art or later developed that permits energy to be transferred between two or more elements, where the intervening presence of one or more additional elements is contemplated, but not required. Conversely, terms such as "directly coupled," "directly connected," and the like, imply the absence of such additional elements.

[0115] As used herein with respect to elements and standards, the term conforming means that the element communicates with other elements in a manner specified, either fully or partially, by the standard, and is recognized by other elements as being sufficiently capable of communicating with other elements in the manner specified by the standard. A conforming element does not have to operate internally in the manner specified by the standard.

[0116] The functionality of the various elements illustrated in the figures, including any functional blocks labeled "processor" and / or "controller," may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functionality may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. Furthermore, explicit use of the terms "processor" or "controller" should not be construed to refer only to hardware capable of executing software, but may implicitly include, without limitation, digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, any switches illustrated in the figures are conceptual only. Those functions may be performed through operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, with the particular technique being selectable by the implementer, as more particularly understood from the context.

[0117] As used in this application, the term "circuit", "circuitry" may refer to one or more or all of the following: (a) a hardware-only circuit implementation (such as an implementation with only analog and / or digital circuitry); (b) a combination of hardware circuitry and software, such as (where applicable) (i) a combination of analog and / or digital hardware circuitry with software / firmware, and (ii) any portion of a hardware processor having software (including digital signal processors, software, and memory that cooperate to cause a device such as a mobile phone or server to perform various functions); and (c) a hardware circuit and / or processor, such as a microprocessor or a portion of a microprocessor, that requires software (e.g., firmware) to operate, but that may be absent when not required for operation. This definition of circuitry applies to all uses of the term in this application, including any claims. As a further example, as used in this application, the term circuitry also covers merely a hardware circuit or processor (or processors), or a portion of a hardware circuit or processor, and its (or their) accompanying software and / or firmware implementation. The term circuitry also encompasses, for example, baseband or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices, where applicable to particular claim elements.

[0118] It should be understood by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the present disclosure. Similarly, any flow charts, flow diagrams, state transition diagrams, pseudocode, or the like, may be substantially represented in a computer-readable medium and therefore will be understood to represent various processes that may be performed by such a computer or processor, whether or not a computer or processor is explicitly depicted.

[0119] This Summary is intended to introduce some example embodiments, with additional embodiments described in the Detailed Description and / or with reference to one or more of the Figures. This Summary is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0120] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEE). [EEE1] 1. A video delivery system capable of film grain removal, the system comprising a video decoder, the video decoder comprising: an input interface for receiving an encoded bitstream comprising a compressed video bitstream and metadata, the compressed video bitstream encoding a sequence of images comprising digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; A processor comprising: decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each of the decompressed representations including a respective host image component and a respective film grain component; calculating an estimate of each of the respective film grain components for each of the decompressed representations based on the metadata; removing said respective estimates from said respective decompressed representations to generate corresponding estimates of said respective host image components; A video delivery system comprising: [EEE2] The processor: Calculating a film grain image based on said metadata; Calculate the respective estimates using the film grain image. The video delivery system of EEE1, further configured as follows: [EEE3] The processor: Calculate the standard deviation corresponding to the estimated host image pixel values ​​for a patch of image pixels; Calculating the film grain image based on the standard deviation The video delivery system of EEE2, further configured as follows: [EEE4] The video delivery system of any one of EEE2 and EEE3, wherein the processor is further configured to adjust the film grain image to account for distortion caused by at least one of video compression and decompression. [EEE5] The video delivery system of claim 8, wherein the processor is configured to adjust the film grain image using a Gaussian filter. [EEE6] The video delivery system of any one of EEE4 and EEE5, wherein the processor is configured to adjust the film grain image using a finite impulse response filter. [EEE7] The video delivery system of any one of claims 8 to 10, wherein the processor is configured to adjust the film grain image using a guided filter. [EEE8] The processor: Calculates a first order polynomial approximation of the nonlinear luma modulation function; Calculate the respective estimates using the approximations The video delivery system of any one of EEE1 to EEE7, further configured as follows: [EEE9] The video delivery system of claim 8, wherein the processor is further configured to calculate the approximation using the metadata. [EEE10] The video delivery system of any one of EEE1 to EEE9, wherein the processor is further configured to calculate the respective estimates using a pre-computed look-up table storing parameters of a first order polynomial approximation of a non-linear luma modulation function. [EEE11] The processor: temporally averaging the sequence of each decompressed representation using a plurality of temporal sliding windows, each of the temporal sliding windows corresponding to a respective image frame pixel, and at least some of the temporal sliding windows having different respective lengths; selecting a length of the temporal sliding window based on the metadata; The video delivery system of any one of EEE1 to EEE10, further configured to perform the following: [EEE12] 12. The video delivery system of any one of claims EE1 to 11, further comprising a video encoder, the video encoder comprising: an output interface for outputting the encoded bitstream for the video encoder; a video compression module generating said compressed video bitstream using lossy compression according to a video compression standard; A video delivery system comprising: [EEE13] A video delivery system capable of film grain removal is provided, the system having a video decoder, the video decoder comprising: an input interface for receiving an encoded bitstream comprising a compressed video bitstream and metadata, the compressed video bitstream encoding a sequence of images comprising digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; A processor comprising: decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each of the decompressed representations including a respective host image component and a respective film grain component; a processor configured to perform the steps of: temporally averaging the sequence of each decompressed representation using a plurality of temporal sliding windows, each of the temporal sliding windows corresponding to a respective image frame pixel, at least some of the temporal sliding windows having different respective lengths selected based on the metadata; A video delivery system comprising: [EEE14] 8. The video delivery system of claim 6 further comprising a video encoder, the video encoder comprising: an output interface for outputting the encoded bitstream for the video encoder; a video compression module generating said compressed video bitstream using lossy compression according to a video compression standard; A video delivery system comprising: [EEE15] 1. A machine-implemented method for removing film grain from video data, the method comprising: receiving an encoded bitstream comprising a compressed video bitstream and metadata, the compressed video bitstream encoding a sequence of images comprising digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each of the decompressed representations including a respective host image component and a respective film grain component; calculating an estimate of each of the respective film grain components for each of the decompressed representations based on the metadata; and removing said respective estimates from said respective decompressed representations to generate corresponding estimates of said respective host image components. method. [EEE16] calculating a film grain image based on the metadata; and calculating the respective estimates includes using the film grain image. Method according to EEE15. [EEE17] Calculating a first order polynomial approximation of the nonlinear luma modulation function; calculating the respective estimates includes using the approximations; The method according to any one of claims 8 to 10, [EEE18] 18. The method of any one of EEE15 to 17, further comprising temporally averaging the sequence of respective decompressed representations using a plurality of temporal sliding windows, each of the temporal sliding windows corresponding to a respective image frame pixel, and at least some of the temporal sliding windows having different respective lengths. [EEE19] 8. The method according to any one of claims 8 to 17, wherein the compressed video bitstream has been generated using lossy compression according to a video compression standard. [EEE20] A non-transitory machine-readable medium encoded with program code which, when executed by a machine, causes the machine to perform operations that constitute a method as recited in any one of claims EEE15 to 19.

Claims

1. 1. A video delivery system capable of digital film grain removal, the system comprising a video decoder, the video decoder comprising: an input interface for receiving a compressed video bitstream and an encoded bitstream including metadata, the compressed video bitstream encoding a sequence of images including digital film grain, the metadata including one or more parameters corresponding to the digital film grain; 13. A processor comprising: decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each of the decompressed representations including a respective host image component and a respective film grain component; calculating an estimate of each of the respective film grain components for each of the decompressed representations based on the metadata using a simulated film grain image and an approximation of a modulation function that modulates the digital film grain; removing said respective estimates from said respective decompressed representations to generate corresponding estimates of said respective host image components; A video delivery system comprising:

2. The processor: Calculating the simulated film grain image using a film grain model and the metadata. The video delivery system of claim 1 , further configured to:

3. The processor: Calculate a standard deviation corresponding to the estimated host image pixel values ​​for the patch of image pixels; Calculating the simulated film grain image based on the standard deviation The video delivery system of claim 2 , further configured to:

4. 4. The video delivery system of claim 2 or 3, wherein the processor is further configured to adjust the simulated film grain image using a digital filter to account for distortions caused by at least one of video compression and decompression.

5. The video delivery system of claim 4 , wherein the processor is configured to adjust the simulated film grain image using a Gaussian filter.

6. 6. The video delivery system of claim 4 or 5, wherein the processor is configured to adjust the simulated film grain image using a finite impulse response filter.

7. 7. The video delivery system of claim 4, wherein the processor is configured to adjust the simulated film grain image using a guided filter.

8. The processor: Compute a first order polynomial approximation of the nonlinear luma modulation function; Calculate the respective estimates using the approximations 8. The video delivery system of claim 1 , further configured to:

9. The video delivery system of claim 8 , wherein the processor is further configured to calculate the approximation using the metadata.

10. 10. The video delivery system of claim 1, wherein the processor is further configured to calculate the respective estimates using a pre-computed look-up table storing parameters of a first order polynomial approximation of a non-linear luma modulation function.

11. The processor: Temporally averaging the sequence of each of the decompressed representations using a plurality of temporal sliding windows, each of the temporal sliding windows corresponding to a respective image frame pixel, at least some of the temporal sliding windows having different respective lengths; selecting a length of the temporal sliding window based on the metadata; The video delivery system of claim 1 , further configured to:

12. 12. The video delivery system of claim 1, further comprising a video encoder, the video encoder comprising: an output interface for outputting the encoded bitstream for the video encoder; a video compression module for generating the compressed video bitstream using lossy compression according to a video compression standard; A video delivery system comprising:

13. A video delivery system capable of digital film grain removal is provided, the system having a video decoder, the video decoder comprising: an input interface for receiving a compressed video bitstream and an encoded bitstream including metadata, the compressed video bitstream encoding a sequence of images including digital film grain, the metadata including one or more parameters corresponding to the digital film grain; 13. A processor comprising: decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each of the decompressed representations including a respective host image component and a respective film grain component; a processor configured to perform the steps of: temporally averaging the sequence of each decompressed representation using a plurality of temporal sliding windows, each of the temporal sliding windows corresponding to a respective image frame pixel, at least some of the temporal sliding windows having different respective lengths selected based on the metadata; A video delivery system comprising:

14. 14. The video delivery system of claim 13, further comprising a video encoder, the video encoder comprising: an output interface for outputting the encoded bitstream for the video encoder; a video compression module for generating the compressed video bitstream using lossy compression according to a video compression standard; A video delivery system comprising:

15. 1. A machine-implemented method for removing digital film grain from video data, the method comprising: receiving an encoded bitstream comprising a compressed video bitstream and metadata, the compressed video bitstream encoding a sequence of images comprising digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each of the decompressed representations including a respective host image component and a respective film grain component; calculating an estimate of each of the respective film grain components for each of the decompressed representations based on the metadata using a simulated film grain image and an approximation of a modulation function that modulates the digital film grain; and removing said respective estimates from said respective decompressed representations to generate corresponding estimates of said respective host image components. method.

Images