Metadata assisted film particle removal

Abstract

The invention discloses a metadata-assisted film particle removal method and a corresponding device. One example embodiment allows a video decoder to substantially completely remove film particles from a digital video signal undergoing lossy video compression and then undergoing video decompression. Different embodiments may depend on only spatial domain particle removal processing, only time domain particle removal processing, or a combination of spatial domain and time domain particle removal processing. Both the spatial domain and time domain particle removal processes may use metadata provided by a corresponding video encoder, the metadata including one or more parameters corresponding to digital film particles injected into the host video at the encoder. The video decoder can accommodate different film particle injection formats by signal pre-processing intended to provide an input to a film particle removal module of the video decoder compatible with the film particle removal method implemented therein.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority from the following priority applications: U.S. Provisional Patent Application 63 / 332,332 filed on April 19, 2022 (No.: D22003USP01) and European Patent Application 22168812.0 filed on April 19, 2022 (No.: D22003EP), each of which is hereby incorporated by reference in its entirety.

[0003] Public domain

[0004] Various example embodiments relate to image and video processing, and more particularly, but not exclusively, to digital film grain technology. Background Art

[0005] On physical film, film grain is a random physical texture consisting of small metallic silver particles on processed photographic celluloid. In digital photography, the visual and artistic effect of film grain can be simulated by adding a digital grain pattern to a digital image after it has been captured. Because digital film grain can be difficult to encode, e.g. due to its (quasi-)random nature, a video encoder can often remove film grain during encoding, and then, during playback, a corresponding video decoder can synthesize the film grain and add it back.

[0006] Brief Overview of Specific Embodiments

[0007] Various embodiments of metadata-assisted film grain removal methods and corresponding devices are disclosed herein. An example embodiment allows a video decoder to substantially completely remove film grains from a digital video signal that has undergone lossy video compression and then video decompression. Different embodiments may rely solely on spatial domain grain removal processing, solely on temporal domain grain removal processing, or on a combination of spatial domain and temporal domain grain removal processing. Both spatial domain and temporal domain grain removal processing can use metadata provided by a corresponding video encoder, the metadata including one or more parameters corresponding to digital film grain injected into a host video at the encoder. A video decoder can accommodate different film grain injection formats through signal preprocessing, which is intended to provide a film grain removal module of a video decoder with input compatible with the film grain removal method implemented therein.

[0008] According to an example embodiment, a video delivery system capable of removing film grains is provided, the system comprising a video decoder, the video decoder comprising: an input interface for receiving an encoded bitstream comprising a compressed video bitstream having a sequence of images containing digital film grains encoded therein and metadata, the metadata comprising one or more parameters corresponding to the digital film grains; and a processor configured to: decompress the compressed video bitstream to generate a corresponding decompressed representation of each image, each decompressed representation comprising a corresponding host image component and a corresponding film grain component; calculate a corresponding estimated value of the corresponding film grain component of each decompressed representation based on the metadata; and remove the corresponding estimated value from the corresponding decompressed representation to produce a corresponding estimated value of the corresponding host image component.

[0009] According to another example embodiment, a video delivery system capable of removing film grain is provided, the system comprising a video decoder, the video decoder comprising: an input interface for receiving an encoded bitstream comprising a compressed video bitstream having encoded therein a sequence of images containing digital film grains and metadata, the metadata comprising one or more parameters corresponding to the digital film grains; and a processor configured to: decompress the compressed video bitstream to generate a corresponding decompressed representation of each image, each decompressed representation comprising a corresponding host image component and a corresponding film grain component; and temporally average the sequence of corresponding decompressed representations using a plurality of temporal sliding windows, each temporal sliding window corresponding to a corresponding image frame pixel, at least some of the temporal sliding windows having different corresponding lengths selected based on the metadata.

[0010] According to another example embodiment, a machine-implemented method of removing film grain from video data is provided, the method comprising: receiving an encoded bitstream comprising a compressed video bitstream having encoded therein a sequence of images containing digital film grain and metadata, the metadata comprising one or more parameters corresponding to the digital film grain; decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each decompressed representation comprising a respective host image component and a respective film grain component; calculating a respective estimate of the respective film grain component of each decompressed representation based on the metadata; and removing the respective estimate from the respective decompressed representation to produce a corresponding estimate of the respective host image component.

[0011] According to another example embodiment, a non-transitory machine-readable medium is provided having a program code encoded thereon, wherein when the program code is executed by a machine, the machine implements a method comprising the steps of: receiving an encoded bitstream comprising a compressed video bitstream having encoded therein a sequence of images containing digital film grains and metadata, the metadata comprising one or more parameters corresponding to the digital film grains; decompressing the compressed video bitstream to generate a corresponding decompressed representation of each image, each decompressed representation comprising a corresponding host image component and a corresponding film grain component; calculating a corresponding estimated value of the corresponding film grain component of each decompressed representation based on the metadata; and removing the corresponding estimated value from the corresponding decompressed representation to produce a corresponding estimated value of the corresponding host image component. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other aspects, features, and benefits of the various disclosed embodiments will become more fully apparent from the following detailed description and accompanying drawings, in which:

[0013] Figure 1 An example process of a video delivery pipeline is depicted in which at least some embodiments may be practiced.

[0014] Figure 2 is a diagram showing the Figure 1 Block diagram of a video encoder used in the process.

[0015] Figure 3 is a diagram showing the Figure 1 Block diagram of a video decoder used in the process.

[0016] Figure 4 is a diagram showing the Figure 3 Processing flow chart of the spatial domain particle removal process implemented in a video decoder.

[0017] Figure 5 The diagram shows the use of the first embodiment. Figure 3 Example film grain removal results that may be achieved using a scaled Gaussian filter in a video decoder.

[0018] Figure 6 The diagram shows the use of the second embodiment. Figure 3 Example film grain removal results that may be achieved using a scaled Gaussian filter in a video decoder.

[0019] Figure 7 The diagram shows the use of the third embodiment. Figure 3 Example film grain removal results that may be achieved using a scaled Gaussian filter in a video decoder.

[0020] Figure 8 The diagram shows the use of the fourth embodiment. Figure 3 Example film grain removal results that may be achieved using a scaled Gaussian filter in a video decoder.

[0021] Fig. 9 is a diagram showing the Figure 3 Flowchart of a process for temporal grain removal implemented in a video decoder of FIG.

[0022] Fig.10 It is shown that the use of signal preprocessing according to an embodiment can be adapted to Figure 3 Example of an intensity-dependent scaling factor in a video decoder. DETAILED DESCRIPTION

[0023] The present disclosure and its aspects may be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods; computer program products, computer systems and networks, user interfaces and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The above content is only intended to give a general idea of ​​the various aspects of the present disclosure and does not limit the scope of the disclosure in any way.

[0024] In the following description, numerous details are set forth, such as optical device configuration, timing, operation, etc., in order to provide an understanding of one or more aspects of the present disclosure. It will be apparent to those skilled in the art that these specific details are merely exemplary and are not intended to limit the scope of the present application.

[0025] In addition, although the present disclosure focuses primarily on examples of various circuits for use in digital projection systems, it should be understood that these are merely examples. It will be further understood that the disclosed systems and methods can be used in any device that requires projected light, such as movie theaters, consumer and other commercial projection systems, head-up displays, virtual reality displays, etc. The disclosed systems and methods can be implemented in additional display devices, such as with OLED displays, LCD displays, quantum dot displays, or similar devices.

[0026] Herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance (luma)) in an image, e.g., from the darkest gray (black) to the brightest white (highlight). In this sense, DR relates to "scene-referred" intensities. DR may also relate to the ability of a display device to adequately and / or approximately render a range of intensities of a particular width. In this sense, DR relates to "display-referred" intensities. Unless a particular meaning is explicitly designated as having a particular meaning at any point in this specification, it should be inferred that the term can be used in either sense, e.g., interchangeably.

[0027] Here, the term "high dynamic range" (HDR) refers to a DR width that spans about 14-15 orders of magnitude across the human visual system (HVS). In practice, with respect to HDR, the DR across a wide width of the intensity range that humans can perceive simultaneously may be truncated. As used herein, the terms "enhanced dynamic range" (EDR) or "visual dynamic range" (VDR) may refer, individually or interchangeably, to the DR that is perceptible within a scene or image by the human visual system including eye movements, thereby allowing for some light adaptation variations throughout the scene or image.

[0028] In practice, an image includes one or more color components (e.g., luma Y and chroma Cb and Cr), where each color component is represented by n bits of precision per pixel (e.g., n=8). By using linear luminance encoding, images where n≤8 are considered standard dynamic range images, while images where n>8 (e.g., JPEG images with 24-bit color) can be considered enhanced dynamic range images. EDR and HDR images can also be stored and distributed using high-precision (e.g., 16-bit) floating point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

[0029] Here, the term "metadata" refers to any auxiliary information that is transmitted as part of the coded bitstream and assists a decoder in rendering the corresponding image(s). For television broadcasting and video streaming, video metadata can be used to provide side information about specific video and audio streams or files. Metadata can be embedded directly into the video or included as a separate file in a container such as MP4 or MKV. Metadata can include information about an entire video stream or file or about a specific video frame. Metadata is collected by cameras, encoders, and other video processing elements (see, for example, Figure 1The metadata created by the method 115, 120) may include, but is not limited to, timestamp, video resolution, digital film grain parameters, color space information or color gamut information, reference display parameters, auxiliary signal parameters, file size, closed captioning, audio language, advertising insertion points, color space, error messages, etc. Additional examples of metadata related to the disclosed embodiments are described below.

[0030] Many consumer desktop monitors may support brightness levels of 200 to 300 cd / m 2 Many consumer HDTVs have brightness levels ranging from 300 to 500 nits, with newer models reaching 1,000 nits (cd / m 2 ). Such conventional displays are typically characterized by a lower dynamic range (LDR), also referred to as standard dynamic range (SDR), compared to HDR or EDR. As the availability of HDR content grows due to advances in image capture devices (e.g., cameras) and HDR displays (e.g., the PRM-4200 Professional Reference Monitor from Dolby Laboratories), HDR content can be color graded and displayed on HDR displays that support a higher dynamic range (e.g., from 1,000 nits to 5,000 nits or more).

[0031] Video Coding According to Example Embodiments

[0032] Figure 1 An example process of a video delivery pipeline (100) is depicted, showing various stages from video capture to video content display according to an embodiment. A sequence of video frames (102) can be captured or generated using an image generation block (105). The video frames (102) can be captured digitally (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide video data (107). Alternatively, the video frames (102) can be captured on film using a film camera. The film is then converted to a digital format to provide video data (107).

[0033] During the production stage (110), the video data (107) may be edited to provide a video production stream (112). The data of the video production stream (112) may then be provided to a processor (or one or more processors, such as a central processing unit (CPU)) at a post-production block (115) for post-production editing. Post-production editing at this block (115) may include, for example, adjusting or modifying the color or brightness of specific areas of an image to enhance image quality or achieve a specific look for an image in accordance with the creative intent of the video creator. This portion of post-production editing is sometimes referred to as "color timing" or "color grading." Other edits (e.g., scene selection and sequencing, image cropping, addition of computer-generated visual effects, etc.) may be performed at block (115) to arrive at a "final" version (117) of the production for distribution. During post-production editing (115), the video images may be viewed on a reference display (125).

[0034] After post-production (115), the video data of the final version (117) can be passed to the encoding block (120) for downstream delivery to a decoding and playback device, such as a television, set-top box, movie theater, etc. In some embodiments, the encoding block (120) may include an audio and video encoder, such as those defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats, to generate an encoded bitstream (122). Some of the methods described below in this document can be performed by a corresponding processor at the encoding block (120). In the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a corresponding decoded signal (132) that is a copy or approximation of the representation signal (117). The receiver can be connected to a target display (140), which can have characteristics that are different or completely different from the reference display (125). In this case, the display management block (135) can be used to map the decoded signal (132) to the characteristics of the target display (140) by generating a display mapping signal (137). Some of the methods described herein below may be performed by the decoding unit (130) and / or the display management block (135). Depending on the embodiment, the decoding unit (130) and the display management block (135) may include separate processors, or may be based on a single integrated processing unit.

[0035] As described above, film grain can provide an effective method to enhance the appearance of digital video in terms of aesthetics and clarity. Film grain can also be used to mitigate banding artifacts and / or mask compression artifacts. Film grain can be added in post-production (115). Traditionally, film grain is removed at an encoder at an encoding block (120), and the resulting "clean" video signal is compressed, for example, using a lossy video compression code to generate an encoded bitstream (122). At a corresponding decoder, the encoded bitstream (122) is decoded, and the film grain is added back using a decoding unit (130) to generate a decoded signal (132) that includes film grain.

[0036] In some instances, film grain may not be removed at the encoder in the encoding block (120), or, alternatively, basic film grain may be added to a "clean" video signal prior to video compression. For example, this type of processing may be performed to ensure that film grain characteristics are present in the encoded bitstream (122) and to enable a "simpler" decoder to provide playback that includes film grain, even though the decoder itself may not be capable of synthesizing and overlaying film grain. However, if the decoder has film grain synthesis capabilities, the basic film grain of the encoded bitstream (122) may be removed at the decoder using a decoding unit (130), and further video processing may be applied thereon. In some use cases, such further video processing may include adding another (e.g., adaptive) type of film grain based on the viewing environment corresponding to the target display (140).

[0037] In one example of the above use case, the base layer of the bitstream (122) carries an SDR signal with SDR-specific film grain injected in. If the received bitstream is to be converted to HDR, the SDR-specific film grain may be removed at the decoding unit (130) and replaced with different (e.g., HDR-specific) film grain.

[0038] In another example of the above use case, the bitstream (122) may have film grain injected therein to reduce the above banding artifacts. If the decoding unit (130) operates to convert an 8-bit based profile to a 10-bit based profile, the corresponding processing may include removing the existing film grain and then applying a de-banding filter. The resulting filtered signal may be further processed to, among other things, insert different film grain.

[0039] Example embodiments disclosed herein are generally directed to performing film grain removal from a received bitstream (122) at a decoding unit (130). Some embodiments may rely only on spatial domain grain removal processing or only on temporal domain grain removal processing. Some other embodiments may rely on both spatial domain and temporal domain grain removal processing. In example embodiments, grain removal processing is performed using metadata of a received bitstream (122), which metadata may provide useful information about film grain of the received bitstream, thereby facilitating grain removal processing at the decoding unit (130).

[0040] In an example embodiment, the pipeline blocks (115, 120) may generate film grain for an encoded bitstream (122) using a reproducible film grain model, where a random or pseudo-random seed and a selected intensity modulation function may be used. The seed information and the film grain parameters of the model may be transmitted as metadata in the bitstream 122. Thus, a decoder at the decoding unit (130) may be provided with certain specific details of the film grain pattern of the stream, which the decoder may beneficially utilize to efficiently generate an accurate approximation of a corresponding "clean" (i.e., grain-free) video signal, e.g., as described in more detail below.

[0041] Figure 2 is a block diagram showing a video encoder (200) according to an embodiment. The encoder (200) can be used, for example, to implement the encoding block (120) of the pipeline (100) (see also Figure 1 ). For illustrative purposes and without any implied limitation, Figure 2 Reference numerals (117) and (122) are used to better indicate Figure 1 and Figure 2 A person skilled in the relevant art will readily appreciate that alternative arrangements of the encoder (200) within the encoding block (120) are also possible.

[0042] like Figure 2 As shown, an encoder (200) operates to convert an input video signal (117) into an output bitstream (122). In this embodiment, the input video signal (117) has no film grain encoded therein. The processing performed in the encoder (200) causes the corresponding video signal encoded in the output bitstream (122) to have digital film grain therein. In addition, the encoder (200) typically causes the output bitstream (122) to carry metadata corresponding to and / or characterizing the digital film grain.

[0043] For each picture of the output bitstream (122), the encoder (200) generates a film grain image (220) using a film grain model (210). The film grain model (210) is also typically "known" to the corresponding video decoder (see, for example, Figure 3 ). The specific parameter values ​​of the film grain model (210) used by the encoder (200) in generating the film grain image (220) can generally be communicated to the corresponding video decoder as film grain model metadata (212). Thus, the corresponding video decoder can compute an exact copy of the film grain image (220) (see also Figure 3 ).

[0044] A film grain injection block (240) of the encoder (200) operates to modulate the noise intensity of a film grain image (220) pixel by pixel based on the corresponding intensity of the input video signal (117). The resulting modulated film grain image is then added to a host image to produce a corresponding film grain injected image. A sequence (242) of such film grain injected images is compressed in a video compression module (250) of the encoder (200) to produce a compressed video bitstream (252). In some embodiments, the video compression module (250) may perform lossy compression according to a video compression standard. Figure 2 The metadata generator (230) of the illustrated parameter inputs (222, 244, 254) generates film grain removal metadata (232) including one or more parameters that can be used to efficiently and effectively remove film grain at a corresponding video decoder. Examples of such metadata (232) are described in more detail below. A merger module (260) operates to appropriately merge the metadata (212, 232) with the compressed video bitstream (252) to generate an output bitstream (122).

[0045] Figure 3 is a block diagram showing a video decoder (300) according to an embodiment. The decoder (300) can be used, for example, to implement the decoding unit (130) of the pipeline (100) (see also Figure 1 The decoder (300) operates to convert the received bitstream (122) into a video signal (342). In an example embodiment, the video signal (342) has the above-described digital film grain substantially completely removed therefrom and, therefore, may represent a relatively accurate approximation of the source video signal (117) (see also Figure 2 ).

[0046] The decoder (300) includes a separator module (310) configured to appropriately extract the above-mentioned metadata (212, 232) and the compressed video bitstream (252) from the received bitstream (122). As described above, the film grain model (210) is "known" to the decoder (300) (e.g., stored in a memory of the decoder (300)). This knowledge, together with the film grain model metadata (212) extracted by the separator module (310), enables the decoder (300) to Figure 3 The recalculated film grain image (220) is shown. The film grain image (220) may be referred to as a simulated film grain image.

[0047] The decoder (300) further includes a video decompression module (320) configured to appropriately decompress the compressed video bitstream (252) received from the separator module (310) to generate a decompressed video signal (330). Note that if lossy compression is used at the video compression module (250) of the encoder (200), the decompressed video signal (330) may be different from the sequence (242). Since the film grain injection performed at the film grain injection block (240) of the encoder (200) involves modulation of the film grain image (220) by luminance, the actual injected film grain may not be determined from the decompressed video signal (330), but may be estimated. The estimation is performed by a film grain removal module (340) of the decoder (300) and typically relies on the recomputed film grain image (220), metadata (232) extracted from the received bitstream (122) by the separator module (310), and the decompressed video signal (330). An example embodiment of a film grain removal method implemented in the film grain removal module (340) is described in more detail below.

[0048] Film grain injection process

[0049] Let us denote the i-th pixel of the j-th host image frame as s ji , the corresponding film grain pixel is represented by n ji The brightness modulation function is represented by f(). The modulation function, f(s ji ), which may be a typical nonlinear function of the host signal. Film grain is injected into the pixel, v ji , which can be expressed as:

[0050] v ji =s ji +f(s ji )·n ji (1)

[0051] The film grain injection signal after video compression, for example by the video compression module (250) of the encoder (200), is represented as:

[0052]

[0053] Where g() represents the compression function. The compressed signal without film grain is expressed as:

[0054]

[0055] As described above, the film grain removal process may rely on spatial domain processing only, temporal domain processing only, or a combination of spatial and temporal domain processing.Example embodiments of a film grain removal process are described in more detail below.

[0056] Film Grain Removal Process: Spatial Domain

[0057] Figure 4 4 is a process flow diagram (400) illustrating a spatial domain grain removal process implemented in a decoder (300) according to an embodiment. The process (400) includes a processing block (440) in which a decompressed video signal (330) is converted into an output video signal (342) (see also Figure 3 ). The processing block (440) also receives inputs from the processing blocks (420, 430) which will assist in the conversion process. In general, the purpose of the grain removal process (400) is to remove the grain from the signal by injecting compressed film grain into the signal. Recover the host signal from the signal obtained by applying video decompression ji (See also equations (1) to (2)) According to an embodiment, the host signal actually restored at the decoder (300) may be a copy of the host signal used to generate the bitstream (122) at the encoder (200), or may be a relatively accurate estimate of the host signal.

[0058] The input provided by the processing block (420) to the processing block (440) is based on the above-mentioned brightness modulation function f() (410). The relevant parameters representing the function f() can be provided to the decoder (300), for example, as part of the film grain model (210) and / or metadata (212) (see also Figure 2 , Figure 3 ). In an example embodiment, processing block (420) operates to compute local first-order polynomial approximations of function f(), which approximations may then be used in a particle removal process implemented in processing block (440), for example, as described in more detail below.

[0059] The input provided by processing block (430) to processing block (440) is an estimate of a distorted film grain image (220), wherein the distortion is typically caused by the serial application of lossy compression (250) and corresponding decompression (320) (see also Figure 2 , Figure 3 ). Examples of how these estimates are calculated are described in more detail below.

[0060] Lossless compression

[0061] When the video compression module (250) of the encoder (200) applies lossless compression, the function defined by the function g() defined by equations (2) to (3) is actually a bypass function with the following properties:

[0062] v ji =g(v ji ) (4)

[0063] s ji =g(s ji ) (5)

[0064] In some embodiments, the brightness modulation function f() may be linear. These embodiments are described with reference to equations (6) to (8). In some other embodiments, the brightness modulation function f() may be nonlinear. The latter embodiments are described with reference to equations (9) to (18).

[0065] The linear function f() can be expressed as follows:

[0066] f(s ji )=a+b·s ji (6)

[0067] Where a and b are constants. Using equation (6), equation (1) can be rewritten as follows:

[0068] v ji =s ji +(a+b·s ji )·n ji =(1+b·n ji )s ji +a·n ji (7)

[0069] The original host signal s can be recovered by rearranging equation (7) to obtain the following equation: ji :

[0070]

[0071] Since the constants a and b can be passed to the decoder (300), for example as part of the metadata (212), the film grain removal module (340) can be programmed to recover the original host signal s using equation (8): ji .

[0072] When the brightness modulation function f() is a nonlinear function, an approximate solution for film grain removal, such as the solution of equation (8), may not exist. However, given equation (8), an approximate solution can be constructed based on the local linear approximation of the function f(), which is expressed as follows:

[0073] f(x)≈a x +b x ·x (9)

[0074] where a x and b x are (local) coefficients that depend on x. More specifically, for each input codeword x, we can calculate the coefficients based on the neighboring codewords x∈[x L ,x H ] and the first-order polynomial regression of the corresponding modulation function output y = f(x) to calculate the local linear function. Range boundary x L and x H It can be determined, for example, using equations (10) to (11):

[0075] x L =clip3(xd,0,2 B -1) (10)

[0076] x H =clip3(x+d,0,2 B -1) (11)

[0077] Where B is the bit depth of the video signal; d is the neighborhood range. In some embodiments, the value of d can be selected to approach the maximum film grain size.

[0078] Let us replace the local first-order polynomial parameter (a x ,b x ) is represented by vector m x :

[0079]

[0080] Then, let us use the following matrix S x and vector v x Rearrange the input and output of function f():

[0081]

[0082]

[0083] Then, (a x , b x ) is an approximate solution This can be obtained by using an appropriate least mean square (LMS) algorithm applied to the following equation:

[0084]

[0085] where the superscript T denotes transposition. Note that equations (12) to (15) are such that for every possible value x∈[0, 2 B -1] has a corresponding pair of parameters (a x , b x In one example embodiment, equations (12) to (15) can be used to construct a lookup table (LUT), where subtable A(x) maps x to a x , and the subtable B(x) maps x to b x .

[0086] Similar to equation (7), equation (1) can now be approximated by equation (16) as follows:

[0087] v ji ≈s ji +(A(s ji )+B(s ji )·s ji )·n ji =(1+B(s ji )·n ji )s ji +A(s ji )·n ji (16)

[0088] The original host signal s can be obtained by rearranging equation (16) to obtain the following equation: ji Approximate value of :

[0089]

[0090] Note that the right side of (17) depends on s ji , which may not be available at the decoder. Therefore, further approximation is required to adapt equation (17) to the process of the decoder (300). In an example embodiment, this further approximation can be obtained by replacing: ji ≈v ji Qualitatively, this substitution can be justified by the fact that the injected film grain is a relatively weak signal and s ji and v jiThe difference between the two is usually small. After substitution, equation (17) becomes equation (18):

[0091]

[0092] Because v ji The value of is usually available at the decoder, so equation (18) can be used in the decoder (300) procedure to obtain an estimate of the host image frame s ji .

[0093] Lossy Compression

[0094] When the video compression module (250) of the encoder (200) applies lossy compression, the film grain removal process described above for lossless compression may not produce optimal results. For example, it can be noted that lossy compression may cause signal distortion, such as the host video signal and the film grain component may both lose some of their higher frequency components. Since the "lossless" film grain removal described above does not account for such compression-induced distortion, if such removal is used in conjunction with lossy compression, the residual film grain may be quite noticeable.

[0095] In order to solve this problem, we can further modify equation (18) to obtain equation (19):

[0096]

[0097] in Represents the estimated compression signal; the film grain injection signal after video compression Given by equation (2); and the compressed film grain component Given by formula (20):

[0098]

[0099] Note that (19) depends on the value that needs to be estimated at the decoder According to the example embodiments disclosed herein, such estimation can be performed using a suitable digital filter. Such a digital filter can be, for example, a Gaussian filter, a finite impulse response (FIR) filter, a guided filter, or another suitable filter. As an example, several exemplary filter embodiments are described below. Based on the description provided, a person of ordinary skill in the relevant art will be able to make and use additional filter embodiments without any excessive experimentation.

[0100] According to an example embodiment, a scaled Gaussian filter G() is used to estimate The value of is represented as follows:

[0101]

[0102] where k j is the scaling factor; and σ j is the spectral width of the filter. j and σ j May be included in metadata (232).

[0103] Figures 5 to 8 An example film grain removal result that can be achieved using the scaled Gaussian filter of equation (21) is shown graphically. More specifically, Figures 5 to 7 Each of them graphically shows the peak signal-to-noise ratio (PSNR) versus parameter σ at different corresponding bit rates. j dependency. Figure 5 Corresponding to a bit rate of 20Mbps. Figure 6 Corresponding to a bit rate of 10Mbps. Figure 7 Corresponds to a bit rate of 5 Mbps. Corresponds to σ j A PSNR value of = 0 represents a result where no filter is applied. Figure 8 The graph shows that when σ j = 0.75 and the bit rate is 20Mbps j In general, Figures 5 to 8 The data shown in show that according to these embodiments, a PSNR gain of up to about 2.5 dB can be achieved.

[0104] According to another example embodiment, the FIR filter implemented according to equation (22) is used to estimate Values:

[0105]

[0106] The index k is in Within the range; w jk are the filter coefficients; and ω ji represents a BxB pixel module, the center of which is located at the i-th pixel.

[0107] The optimal filter coefficient w can be obtained jk , for example in the following manner. At the encoder (200), and ji The value of is known and can be expressed as follows:

[0108]

[0109] Rearrangement of equation (23) provides equation (24):

[0110]

[0111] Using the approximate g(s ji )≈s ji , transform equation (24) into equation (25) representing the target signal of the filtering result:

[0112]

[0113] Then, the problem of optimizing the FIR filter coefficients can be formulated as follows:

[0114]

[0115] An approximate solution to the problem of equation (26) can be obtained using the LMS method, for example, as follows. First, let us set the filter coefficients w jk The collection of is arranged into vector form:

[0116]

[0117] Similarly, the i-th pixel (with value n ji ) can be expressed as a vector, ω ji , and these vectors ω of P neighboring pixels ji can be stacked to form a corresponding matrix Ω j , which is expressed as follows:

[0118]

[0119] The corresponding target reference signal is also in vector form, given by equation (29):

[0120]

[0121] Approximate solution for FIR filter coefficients It can be obtained by applying the LMS algorithm to the following equation:

[0122]

[0123] The FIR filter coefficients obtained in this way from equation (30) can then be used in equation (22) to calculate Finally, Equation (31) can be used to calculate the estimated compressed signal

[0124]

[0125] The PSNR gain of FIR filter embodiments generally increases as the filter size increases. However, this improvement may be achieved at the expense of additional implementation overhead.

[0126] With respect to embodiments of guided filters, we observe that the most noticeable residual film grain noise may be located in image regions with relatively flat and / or smooth image patterns. For example, pixel values ​​in flat regions may be approximately constant, with local variations being primarily due to added or incompletely / imperfectly removed film grain. As a preliminary observation, we note that in such flat regions, the estimated compressed film grain A decoded film grain based signal can be injected In one example embodiment, such a model can be expressed as follows:

[0127]

[0128] in, is the B×B local pixel region Φ of the image subjected to lossy compression ji The local linear model constant within .

[0129] Here, the original film grain pattern (n ji ) and reconstructed film grain The difference between ji . Therefore, the following is true:

[0130]

[0131] Applying the LMS method in a similar manner to the above method, the optimal linear model constants can be found using equation (34):

[0132]

[0133] Where ∈ is the regularization parameter. The approximate solution of equation (34) is given by equation (35) and equation (36):

[0134]

[0135] in, and μ i and (σ i ) 2 is Φ i middle The mean and variance of .

[0136] Next, the average of the local linear model constants within the neighborhood can be calculated, for example as follows:

[0137]

[0138] Substituting these values ​​into equation (32), we can obtain:

[0139]

[0140] In an exemplary embodiment, the calculation corresponding to equation (39) may use a guided filter function GF() known to those skilled in the art. Function GF() may be accelerated using an integral image, which is also known to those skilled in the art. Using this function, the j-th frame A two-dimensional image composed of values It can be calculated as follows:

[0141]

[0142] although Represents the jth frame 2D image composed of values; and N j Indicates that the j-th frame n ji The neighborhood size B is a parameter of this method. As mentioned above, the value of B may be related to the film grain size. For example, for larger film grains, a larger value of B may be more appropriate. The linear model constant is the value of the film grain injected into the image. and film grain model {n ji} calculated, for example, as described above. Finally, equation (41) can be used to calculate the estimated compressed signal

[0143]

[0144] It should be noted here that, generally, the guided filter embodiment may be between the Gaussian filter embodiment and the FIR filter embodiment in terms of PSNR gain. However, because the guided filter embodiment works best for flat and smooth image regions, the improvement that is subjectively perceptible to the observer may generally be more significant under the guided filter approach.

[0145] Film Grain Removal Process: Time Domain

[0146] In an example embodiment, film grain may have a zero mean property, which means that an image with film grain has the same DC value as the original image. Therefore, the average pixel value at a given pixel location in a static scene tends to average the film grain back to "zero". Certain embodiments of temporal grain removal disclosed herein are designed to take advantage of this property to remove film grain.

[0147] Fig. 9 900 is a process flow diagram illustrating a temporal grain removal process that may be implemented in a decoder (300) according to an embodiment. For purposes of illustration and without any implicit limitation, the process (900) is shown and described as receiving an image sequence (342) generated by the process (400). j-L ,…,342j ,…,342 j+L ) as input. However, embodiments of the process (900) are not so limited. For example, in another embodiment, the process (900) can operate directly on a sequence of images (330) (see also Figure 3 ). The sliding window length L may depend on the maximum amplitude of the film grain pattern. Generally, when the maximum amplitude is larger, a larger value of L may be more advantageous. In operation, the processing block (910) of the process (900) converts the image (342) ( Figure 4 ) or image (330)( Figure 3 ) is converted into a corresponding output sequence of images (342′), for example, as described below, where the film grain pattern in the output sequence is less obvious than in the input sequence.

[0148] According to an example embodiment, for the j-th frame (342 j )’s i-th spatial filter pixel, in the temporal sliding window The time domain filtering of the process (900) within provides the following signal for the sequence (342'):

[0149]

[0150] It can be based on two thresholds and Adaptively determine and The value of and is based on the metadata parameter τ A and τ B It is derived as follows:

[0151]

[0152] Use function f() to adjust the threshold and One motivation for this is that the injected film grain intensity is also adjusted using the function f() (see, for example, equation (1)).

[0153] It can be determined The value of, for example, by iteratively checking the inequality (45) for the sequence (342 j-1 ,342 j-2 ,...) conditions:

[0154]

[0155] For each iteration, the parameter d is increased. If inequality (45) is not satisfied, and the previous iteration value of d satisfies this inequality, then To reduce the multi-frame search time and frame buffer, the upper limit of the sliding window size may be limited by the left buffer size D L and the restrictions on the available frames at the beginning of a video sequence, for example, as described below:

[0156]

[0157] Similarly, we can iteratively check the sequence (342 j+1 ,342 j+2 ,…) is determined by the condition expressed by inequality (47) Values:

[0158]

[0159] For each iteration, the parameter d is increased. If condition (47) is not satisfied, and the previous value of d satisfied this condition, then To reduce the multi-frame search time and frame buffer, the upper limit of the window size can be determined by the right buffer size D R and the available frames at the end of the video sequence, for example, as follows:

[0160]

[0161] Where F is the number of video frames in the sequence.

[0162] once and The value of is determined as described above, and the time averaging of the processing block (910) can be performed according to equation (42). Note that the time averaging described above is a pixel-based operation. The buffer size D L and D R May be provided to the decoder (300) as part of the metadata (232).

[0163] Signal preprocessing corresponding to alternative film grain models

[0164] Some embodiments may be adapted to be compatible with alternative film grain models, i.e., film grain models that differ from the above-described film grain models. Such embodiments typically rely on signal preprocessing to provide a preprocessed signal that is compatible with the above-described grain removal process. As a non-limiting example of such preprocessing, an example of preprocessing adapted for an MPEG film grain model as defined in a corresponding standard is described herein.

[0165] The example MPEG film grain model is a segmented model based on pixel intensity values. In each segment, different film grain model parameters can be used, such as different cutoff frequencies, different scaling factors for scaling film grain, etc. Suppose there are T such segments in the jth frame, and the segmentation points of these segments are represented as The film grain image in the tth film is represented as And the scaling factor of the tth slice is expressed as

[0166] Fig.10 An example of an intensity-dependent scaling factor adapted using the above-described preprocessing is shown in accordance with an embodiment. Fig.10 It can be noted that the scaling factor can take several discrete values ​​ranging from 0 to 30. In this case, the film grain injected into the pixel v ji It can be expressed as:

[0167]

[0168] in,

[0169] Comparing (49) with (1), we can notice that the segmented scaling factor It can be regarded as the nonlinear modulation function f(s ji ) is a special case of . Therefore, the above nonlinear modulation function f(s ji ) is suitable for this special use case. The corresponding LUT, A(s ji ) and B(s ji ), see also equations (12) to (15), can be calculated, for example, in the following manner:

[0170]

[0171] B(s ji )=0where

[0172] On the other hand, the different cutoff frequencies of the noise patterns in each segment add an extra dimension that needs to be handled in preprocessing. Here, we describe the Construct a complete film grain image {n ji Once a complete film grain image is constructed through preprocessing, subsequent grain removal may rely on one or more of the above-mentioned grain removal processes.

[0173] For building a complete film grain image, consider the i-th pixel at the decoder (300) (see equation (1)). If the i-th pixel value is not close to one of the boundaries of different intensity branches (e.g., see Fig.10 ), for example, the corresponding v ji Value in range If , then the film grain value of the complete film grain image can meet the formula (52):

[0174]

[0175] Here, Δ represents the film grain amplitude which is usually relatively small.

[0176] However, when the pixel value is relatively close to the segment boundary, e.g. The following ambiguity needs to be resolved:

[0177]

[0178] Here, it can be determined based on the received bit stream and metadata and value; and v ji represents the decoded pixel value. However, the host pixel value s ji The determination of must deal with two different options, namely, defined by equations (53) and (54) or

[0179] In an example embodiment, the local neighborhood / patch (Ω i ) in two possible options or More specifically, the result corresponding to the correctly selected alternative is expected to be smoother (e.g., show less residual film grain noise) than the result corresponding to the incorrect one of the two alternatives. Based on this observation, the example embodiment is configured to calculate the standard deviation for each of the two alternatives in such a patch. For the correctly selected one of the two alternatives, the standard deviation is expected to be smaller.

[0180] Formula (55) to (56) are used to calculate the above standard deviation and The following mathematical expressions are provided:

[0181]

[0182] Calculated standard deviation and Then it can be used to select the host image pixel value s ji and the film grain pixel value nji , as shown below:

[0183]

[0184] By performing the above calculation pixel by pixel, we can get the decoded value v ji and split point Create a complete film grain image {n ji In some embodiments, the preprocessing results, including LUTA (s ji ), B(s ji )(see equation (50), equation (51)), which can then be used for further processing according to a suitable embodiment of the above-mentioned particle removal method.

[0185] According to the exemplary embodiments disclosed above, for example, in the brief overview section and / or reference Figures 1 to 10 According to any one or any combination of some or all of the above, a device is provided, which includes a video decoder (e.g., Figure 1 130), the video decoder includes: an input interface (for example, Figure 3 310 in ), for receiving a coded bit stream (eg, Figures 1 to 3 122 in ), the coded bitstream includes a compressed video bitstream (e.g., Figure 3 252) and metadata (e.g. Figure 3 212, 232), the compressed video bitstream having encoded therein a sequence of images containing digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; and a processor (e.g., Figure 3 320, 340), the processor is configured to: (A) decompress (e.g., Figure 3 320 in the compressed video bitstream to generate a corresponding decompressed representation of each image (e.g., Figure 3 330 in (B), each decompressed representation includes a corresponding host image component and a corresponding film grain component; (B) calculating a corresponding estimate of the corresponding film grain component for each decompressed representation based on the metadata; and (C) removing from the corresponding decompressed representation (e.g., Figure 3 340 ) to generate corresponding estimated values ​​of corresponding host image components (eg, Figure 4 342 in ).

[0186] In some embodiments of the above apparatus, the processor is further configured to: calculate based on the metadata (for example, Figure 3 210 ) film grain image (e.g., Figure 3 220); and calculating a corresponding estimate using the film grain image.

[0187] In some embodiments of any of the above apparatus, the processor is further configured to adjust (e.g., Figure 4 430) film grain image to account for distortion caused by at least one of video compression and decompression.

[0188] In some embodiments of any of the above apparatus, the processor is configured to use a Gaussian filter (e.g., Figures 5 to 8 to adjust the film grain image.

[0189] In some embodiments of any of the above apparatus, the processor is configured to adjust the film grain image using a finite impulse response filter (e.g., according to equations (22) to (31)).

[0190] In some embodiments of any of the above apparatus, the processor is configured to adjust the film grain image using a guided filter (e.g., according to equations (32) to (41)).

[0191] In some embodiments of any of the above-mentioned devices, the processor is further configured to: for an image pixel patch, calculate a standard deviation corresponding to an estimated host image pixel value (e.g., according to equations (55) to (56)); and calculate a film grain image based on the standard deviation (e.g., according to equation (58)).

[0192] In some embodiments of any of the above apparatus, the processor is further configured to: calculate (for example, in Figure 4 420 ) a first-order polynomial approximation of a nonlinear brightness modulation function (eg, f() in equation (1)); and using the approximation to calculate a corresponding estimate.

[0193] In some embodiments of any of the above apparatus, the processor is further configured to use metadata to calculate the approximation.

[0194] In some embodiments of any of the above apparatus, the processor is further configured to use a pre-computed lookup table (e.g. Figure 4 420) calculates the corresponding estimated value, and the lookup table stores the parameters of the first-order polynomial approximation of the nonlinear brightness modulation function (for example, f() in equation (1)).

[0195] In some embodiments of any of the above apparatus, the processor is further configured to time average the sequence of corresponding decompressed representations using multiple time sliding windows (e.g., Fig. 9 At 910 in ; according to equation (42)), each time sliding window corresponds to a corresponding image frame pixel, and at least some of the time sliding windows have different corresponding lengths (for example, selected according to equations (45) and (47)).

[0196] In some embodiments of any of the above apparatus, the processor is further configured to calculate the value of the τ in equation (43) and equation (44) based on the metadata (e.g., A and τ B ) Select the length of the time sliding window.

[0197] In some embodiments of any of the above apparatus, the apparatus further comprises a video encoder (e.g., Figure 1 120), the video encoder includes: an output interface (for example, Figure 2 260) to output a coded bit stream for a video encoder; and a video compression module (e.g., Figure 2 250) to generate a compressed video bitstream using lossy compression according to a video compression standard.

[0198] According to another exemplary embodiment disclosed above, for example, in the brief overview section and / or with reference to Figures 1 to 10 Any one or any combination of some or all of the , providing an apparatus including a video decoder (e.g., Figure 1 130), the video decoder includes: an input interface (for example, Figure 3 310 in ), for receiving a coded bit stream (eg, Figures 1 to 3 122 in ), the coded bitstream includes a compressed video bitstream (e.g., Figure 3 252) and metadata (e.g. Figure 3 212, 232), the compressed video bitstream having encoded therein a sequence of images containing digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; and a processor (e.g., Figure 3 320, 340), the processor is configured to: decompress (for example, in Figure 3 320 in the compressed video bitstream to generate a corresponding decompressed representation of each image (e.g., Figure 3 330 in ), each decompressed representation includes a corresponding host image component and a corresponding film grain component; and temporally averaging the sequence of corresponding decompressed representations using multiple temporal sliding windows (e.g., in Fig. 9 910; according to equation (42), each time sliding window corresponds to a corresponding image frame pixel, and at least some time sliding windows have a τ based on metadata (e.g., equations (43) and (44) A and τ B ) selected from different corresponding lengths (for example, according to equations (45) and (47)).

[0199] In some embodiments of the above apparatus, the apparatus further comprises a video encoder (eg, Figure 1 120), the video encoder includes: an output interface (for example, Figure 2 260), for outputting a coded bit stream for a video encoder; and a video compression module (e.g., Figure 2 250) for generating a compressed video bitstream using lossy compression according to a video compression standard.

[0200] According to yet another exemplary embodiment disclosed above, for example, in the brief overview section and / or with reference to Figures 1 to 10 According to any one or any combination of some or all of the foregoing, there is provided a machine-implemented method for removing film grain from video data, the method comprising the steps of: (A) receiving a compressed video bitstream (e.g., Figure 3 252) and metadata (e.g. Figure 3 212, 232) of the coded bit stream (for example, Figures 1 to 3 122 in ), the compressed video bitstream having encoded therein a sequence of images including digital film grain, the metadata including one or more parameters corresponding to the digital film grain; (B) decompressing (e.g., in Figure 3 320 in the compressed video bitstream to generate a corresponding decompressed representation of each image (e.g., Figure 3 330 in, each decompressed representation includes a corresponding host image component and a corresponding film grain component; (C) calculating a corresponding estimate of the corresponding film grain component for each decompressed representation based on the metadata; and (D) removing from the corresponding decompressed representation (e.g., Figure 3 340 in the figure) to generate corresponding estimated values ​​of corresponding host image components (e.g., Figure 4 342 in ).

[0201] In some embodiments of the above method, the method further includes calculating based on the metadata (e.g., Figure 3 210 in the figure) film grain image (e.g., Figure 3 220); and wherein said calculating the corresponding estimate comprises using a film grain image.

[0202] In some embodiments of any of the above methods, the method further includes calculating (e.g., in Figure 4 At 420 in ), a first-order polynomial approximation of a nonlinear brightness modulation function (e.g., f(), equation (1)); and wherein said calculating a corresponding estimate includes using said approximation.

[0203] In some embodiments of any of the above methods, the method further comprises temporally averaging the sequence of corresponding decompressed representations using a plurality of temporal sliding windows (e.g., Fig. 9 At 910 in ; according to equation (42)), each time sliding window corresponds to a corresponding image frame pixel, and at least some of the time sliding windows have different corresponding lengths (for example, selected according to equations (45) and (47)).

[0204] In some embodiments of any of the methods described above, the compressed video bitstream has been generated using lossy compression according to a video compression standard.

[0205] According to another exemplary embodiment disclosed above, for example, in the brief overview section and / or with reference to Figures 1 to 10 A non-transitory machine-readable medium having a program code encoded thereon is provided, wherein when the program code is executed by a machine, the machine implements a method comprising the following steps: (A) receiving a compressed video bitstream (e.g., Figure 3 252) and metadata (e.g. Figure 3 212, 232) of the coded bit stream (for example, Figures 1 to 3 122 in ), the compressed video bitstream having encoded therein a sequence of images containing digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; (B) decompressing (e.g., in Figure 3 320 in the compressed video bitstream to generate a corresponding decompressed representation of each image (e.g., Figure 3 330 in, each decompressed representation includes a corresponding host image component and a corresponding film grain component; (C) calculating a corresponding estimate of the corresponding film grain component for each decompressed representation based on the metadata; and (D) removing from the corresponding decompressed representation (e.g., Figure 3 340 in the figure) to generate corresponding estimated values ​​of corresponding host image components (e.g., Figure 4 342 in ).

[0206] With respect to the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of the processes, etc. have been described as occurring in a certain ordered sequence, the processes may be practiced with the described steps performed in other sequences than those described herein. It should be further understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the descriptions of the processes herein are provided for the purpose of illustrating certain embodiments and should not be construed in any way as limiting the claims.

[0207] Accordingly, it will be appreciated that the foregoing description is intended to be illustrative and not restrictive. Upon reading the foregoing description, many embodiments and applications will be apparent in addition to the examples provided. The scope should not be determined with reference to the foregoing description, but with reference to the appended claims and the full range of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology discussed herein, and the disclosed systems and methods will be incorporated into these future embodiments. In summary, it will be appreciated that the present application is capable of modification and variation.

[0208] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those familiar with the technology described herein unless expressly indicated to the contrary herein. In particular, use of the singular articles such as "a," "the," "said," etc. should be understood to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0209] The abstract of the present disclosure is provided to enable the reader to quickly ascertain the nature of the technical disclosure. The abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above detailed description section, it can be seen that various features are grouped together in different embodiments for the purpose of simplifying the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments include more features than are expressly recited in each claim. On the contrary, as reflected in the following claims, inventive subject matter exists in less than all the features of a single disclosed embodiment. Therefore, the following claims are incorporated into the detailed description section, with each claim as a separate claimed subject matter.

[0210] Although the present disclosure includes references to illustrative embodiments, the description is not intended to be interpreted in a limiting sense. Various modifications to the described embodiments and other embodiments within the scope of the present disclosure will be apparent to those skilled in the art to which the present disclosure belongs, and these modifications are considered to be within the principles and scope of the present disclosure, for example, as described in the following claims.

[0211] Some embodiments may be implemented as a circuit-based process, including possible implementation on a single integrated circuit.

[0212] Some embodiments may be embodied in the form of methods and apparatus for practicing these methods. Some embodiments may also be embodied in the form of program code recorded in a tangible medium, such as a magnetic recording medium, an optical recording medium, a solid-state memory, a floppy disk, a CD-ROM, a hard disk drive, or any other non-transitory machine-readable storage medium, wherein when the program code is loaded into a machine such as a computer and executed by the machine, the machine becomes an apparatus for practicing the patented invention. Some embodiments may also be embodied in the form of program code, such as a program code stored in a non-transitory machine-readable storage medium, including program code that is loaded and / or executed by a machine, wherein when the program code is loaded and executed by a machine such as a computer or a processor, the machine becomes an apparatus for practicing the patented invention. When implemented on a general-purpose processor, the program code segments are combined with the processor to provide a unique device that operates similarly to a specific logic circuit.

[0213] Unless expressly stated otherwise, each numerical value and range should be interpreted as approximations as if the word "about" or "approximately" were preceding such value or range.

[0214] The use of numerical numbers and / or numerical reference signs in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate the interpretation of the claims. Such use should not be interpreted as necessarily limiting the scope of these claims to the corresponding numbered embodiments.

[0215] Although the elements of the following method claims, if any, are recited in a specific order with corresponding labels, unless the claim recitation implies a specific order for implementing some or all of these elements, these elements are not necessarily intended to be limited to being implemented in this specific order.

[0216] References to "one embodiment" or "embodiment" herein mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present disclosure. The appearances of the phrase "in one embodiment" in different places in the specification do not necessarily all refer to the same embodiment, nor do they necessarily refer to separate or alternative embodiments that are mutually exclusive of other embodiments. The same applies to the term "implementation".

[0217] Unless otherwise specified herein, the use of ordinal adjectives "first", "second", "third", etc. to refer to one of multiple similar objects merely indicates that the reference is to different instances of these similar objects, and is not intended to imply that the referred similar objects must have a corresponding order or sequence in time, space, sorting or any other manner.

[0218] Unless otherwise specified herein, the conjunction "if" may also or alternatively be interpreted as "when", "upon", "in response to determination", or "in response to detection" in addition to its literal meaning, and these interpretations may depend on the corresponding specific context. For example, the phrase "if it is determined" or "if [the stated condition] is detected" may be interpreted as "upon determining" or "in response to determination", or "upon detecting [the stated condition or event]", or "in response to detecting [the stated condition or event]".

[0219] Likewise, for the purposes of this description, the terms "couple," "coupling," "coupled," "connect," "connecting," "connected" refer to any means known in the art or later developed that allows energy to be transferred between two or more elements and that contemplates, although not required, the insertion of one or more additional elements. Conversely, the terms "directly coupled," "directly connected," and the like, imply the absence of such additional elements.

[0220] The term "compatible" as used herein in reference to an element and a standard means that the element communicates with other elements in whole or in part in a manner specified by the standard and will be recognized by other elements as being sufficiently capable of communicating with other elements in a manner specified by the standard. The compliant element need not function internally in a manner specified by the standard.

[0221] The functions of the various elements shown in the figure, including any functional modules marked as "processors" and / or "controllers", can be provided by using dedicated hardware and hardware capable of executing software together with appropriate software. When the function is provided by a processor, the function can be provided by a single dedicated processor, by a single shared processor, or by multiple separate processors, some of which can be shared. In addition, the explicit use of the term "processor" or "controller" should not be interpreted as specifically referring to hardware capable of executing software, and may implicitly include but is not limited to digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read-only memory (ROM) for storing software, random access memory (RAM) and non-volatile storage devices. Other hardware, traditional and / or customized, may also be included. Similarly, the switches shown in the figure are only conceptual. Their functions can be implemented by the operation of program logic, by dedicated logic, by the interaction of program control and dedicated logic, or even by manual means, and the implementer can specifically understand the selectable specific technology according to the context.

[0222] As used in this application, the terms "circuit" and "circuitry" may refer to one or more or all of the following: (a) a hardware-only implementation of a circuit (e.g., an implementation of an analog and / or digital circuit system only); (b) a combination of hardware circuits and software, such as (as applicable) (i) a combination of analog and / or digital hardware circuits and software / firmware; and (ii) a hardware processor and any portion of software (including a digital signal processor, software and memory that operate together to enable a device such as a mobile phone or server to perform various functions); and (c) a hardware circuit and / or processor, such as a microprocessor or a portion of a microprocessor, that requires software (e.g., firmware) to operate, but the software may not be present when it is not required for operation. This definition of "circuitry" applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term "circuitry" also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. For example, if applicable to a particular claim element, the term "circuitry" also includes a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0223] For those of ordinary skill in the art, any block diagram herein represents a conceptual view of a schematic circuit system embodying the principles of the present disclosure. Similarly, it is understood that any flow charts (flowdiagrams), state transition diagrams, pseudocodes, etc. represent various processes, which can be substantially represented by computer-readable media and executed by computers or processors, whether or not these computers or processors are explicitly shown.

[0224] The "Brief Summary of Specific Embodiments" in this specification is intended to introduce some example embodiments, and other additional embodiments are described in the "Detailed Description" and / or with reference to one or more drawings. The "Brief Summary of Specific Embodiments" is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0225] Various aspects of the present invention may be understood from the following exemplary embodiments (EEE): EEE1. A video transmission system capable of removing film grain, the system comprising a video decoder, the video decoder comprising:

[0226] an input interface that receives a coded bitstream, the coded bitstream comprising a compressed video bitstream, the compressed video bitstream having encoded therein a sequence of images containing digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; and

[0227] A processor, the processor being configured to:

[0228] decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each decompressed representation comprising a respective host image component and a respective film grain component;

[0229] calculating a respective estimate of a respective film grain component of each decompressed representation based on the metadata;

[0230] as well as

[0231] The respective estimate is removed from the respective decompressed representation to produce a corresponding estimate of the respective host image component.

[0232] EEE2. A video transmission system according to EEE1, wherein the processor is further configured to:

[0233] calculating a film grain image based on the metadata; and

[0234] A corresponding estimate is calculated using the film grain image.

[0235] EEE3. A video transmission system according to EEE2, wherein the processor is further configured to:

[0236] For a patch of image pixels, computing the standard deviation corresponding to the estimated host image pixel value; and

[0237] Based on the standard deviation, the film grain image is calculated.

[0238] EEE4. A video delivery system according to EEE2 or EEE3, wherein the processor is further configured to adjust the film grain image to account for distortion caused by at least one of video compression and decompression.

[0239] EEE5. A video transmission system according to EEE4, wherein the processor is configured to adjust the film grain image using a Gaussian filter.

[0240] EEE6. A video transmission system according to EEE4 or EEE5, wherein the processor is configured to adjust the film grain image using a finite impulse response filter.

[0241] EEE7. A video transmission system according to any one of EEE4 to EEE6, wherein the processor is configured to adjust the film grain image using a guided filter.

[0242] EEE8. A video transmission system according to any one of EEE1 to EEE7, wherein the processor is further configured to:

[0243] calculating a first order polynomial approximation of a nonlinear brightness modulation function; and

[0244] The approximation is used to calculate a corresponding estimate.

[0245] EEE9. A video delivery system according to EEE 8, wherein the processor is further configured to calculate the approximation using the metadata.

[0246] EEE10. A video transmission system according to any one of EEE1 to EEE9, wherein the processor is further configured to calculate the corresponding estimated value using a pre-calculated lookup table, in which parameters of a first-order polynomial approximation of a nonlinear brightness modulation function are stored.

[0247] EEE11. The video transmission system according to any one of EEE1 to EEE 10, wherein the processor is further configured to:

[0248] Temporally averaging a sequence of corresponding decompressed representations using a plurality of temporal sliding windows, each temporal sliding window corresponding to a corresponding image frame pixel, at least some of the temporal sliding windows having different corresponding lengths; and selecting the length of the temporal sliding window based on the metadata.

[0249] EEE12. The video transmission system according to any one of EEE1 to EEE 11, further comprising a video encoder, wherein the video encoder comprises:

[0250] an output interface for outputting the encoded bit stream for the video encoder; and

[0251] A video compression module is used to generate the compressed video bit stream using lossy compression according to a video compression standard.

[0252] EEE13. A video transmission system capable of removing film grain, the system comprising a video decoder, the video decoder comprising:

[0253] an input interface that receives a coded bitstream, the coded bitstream comprising a compressed video bitstream, the compressed video bitstream having encoded therein a sequence of images containing digital film grain, the metadata comprising one or more parameters corresponding to the digital film grain; and

[0254] A processor, the processor being configured to:

[0255] decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each decompressed representation comprising a respective host image component and a respective film grain component; and

[0256] The sequence of corresponding decompressed representations is temporally averaged using a plurality of temporal sliding windows, each temporal sliding window corresponding to a corresponding image frame pixel, at least some of the temporal sliding windows having different corresponding lengths selected based on the metadata.

[0257] EEE14. The video transmission system according to EEE13 further comprises a video encoder, wherein the video encoder comprises:

[0258] an output interface for outputting the encoded bit stream for the video encoder; and

[0259] A video compression module is used to generate the compressed video bit stream using lossy compression according to a video compression standard.

[0260] EEE15. A machine-implemented method for removing film grain from video data, the method comprising:

[0261] receiving an encoded bitstream including a compressed video bitstream having encoded therein a sequence of images including digital film grain and metadata including one or more parameters corresponding to the digital film grain;

[0262] decompressing the compressed video bitstream to generate a respective decompressed representation of each image, each decompressed representation comprising a respective host image component and a respective film grain component;

[0263] computing a respective estimate of a respective film grain component of each decompressed representation based on the metadata; and

[0264] The respective estimate is removed from the respective decompressed representation to produce a corresponding estimate of the respective host image component.

[0265] EEE16. The method according to EEE15 further includes calculating a film grain image based on the metadata; and wherein the calculating the corresponding estimate includes utilizing the film grain image.

[0266] EEE17. The method according to EEE 15 or EEE 16, further comprising calculating a first-order polynomial approximation of the nonlinear brightness modulation function; and

[0267] Wherein said calculating the corresponding estimated value comprises utilizing said approximate value.

[0268] EEE18. The method according to any one of EEE15 to EEE17 further includes using multiple time sliding windows to temporally average the corresponding decompressed representation sequence, each time sliding window corresponding to a corresponding image frame pixel, and at least some of the time sliding windows have different corresponding lengths.

[0269] EEE19. A method according to any one of EEE15 to EEE18, wherein the compressed video bitstream has been generated using lossy compression according to a video compression standard.

[0270] EEE20. A non-transitory machine-readable medium having a program code encoded thereon, wherein when the program code is executed by a machine, the machine performs operations including the method described in any one of EEE15 to EEE19.

Claims

1. A video transmission system capable of removing digital film grain, the system comprising a video decoder (130), the video decoder comprising: an input interface (310) receiving an encoded bitstream (122), the encoded bitstream comprising a compressed video bitstream (252) having encoded therein a sequence of images containing digital film grain, and metadata (212, 232), the metadata comprising one or more parameters corresponding to the digital film grain; and A processor (320, 340), the processor being configured to: decompressing the compressed video bitstream to generate a respective decompressed representation of each of the images (330), each decompressed representation comprising a respective host image component and a respective film grain component; computing, based on the metadata, a respective estimate of the respective film grain component of each decompressed representation using a simulated film grain image (430) and an approximation of a modulation function used to modulate the digital film grain (420); and The respective estimate is removed from the respective decompressed representation to produce a corresponding estimate of the respective host image component (342).

2. The video delivery system according to claim 1, wherein the processor is further configured to: The simulated film image is calculated using a film grain model (210) and the metadata (212).

3. The video delivery system according to claim 2, wherein the processor is further configured to: For a patch of image pixels, computing a standard deviation corresponding to an estimated host image pixel value; and Based on the standard deviation, the simulated film grain image is calculated.

4. A video delivery system according to claim 2 or claim 3, wherein the processor is further configured to adjust the simulated film grain image using a digital filter to account for distortion caused by at least one of video compression and decompression.

5. The video delivery system of claim 4, wherein the processor is configured to adjust the simulated film grain image using a Gaussian filter.

6. A video delivery system according to claim 4 or claim 5, wherein the processor is configured to adjust the simulated film grain image using a finite impulse response filter.

7. The video delivery system of any one of claims 4-6, wherein the processor is configured to adjust the simulated film grain image using a guided filter.

8. The video transmission system according to any one of claims 1 to 7, wherein the processor is further configured to: calculating a first order polynomial approximation of a nonlinear brightness modulation function; and The corresponding estimate is calculated using the approximation.

9. The video delivery system of claim 8, wherein the processor is further configured to utilize the metadata to calculate the approximation.

10. The video transmission system according to any one of claims 1-9, wherein the processor is further configured to calculate the corresponding estimated value using a pre-calculated lookup table, in which parameters of a first-order polynomial approximation of a non-linear brightness modulation function are stored.

11. The video transmission system according to any one of claims 1 to 10, wherein the processor is further configured to: temporally averaging the sequence of corresponding decompressed representations using a plurality of temporal sliding windows, each of the temporal sliding windows corresponding to a corresponding image frame pixel, at least some of the temporal sliding windows having different corresponding lengths; and The length of the temporal sliding window is selected based on the metadata.

12. The video transmission system according to any one of claims 1 to 11, further comprising a video encoder, wherein the video encoder comprises: an output interface, the output interface outputting the encoded bit stream for the video encoder; as well as A video compression module is used to generate the compressed video bit stream using lossy compression according to a video compression standard.

13. A video transmission system capable of removing digital film grain, the system comprising a video decoder (130), the video decoder comprising: an input interface (310) receiving an encoded bitstream (122), the encoded bitstream comprising a compressed video bitstream (252) having encoded therein a sequence of images containing digital film grain, and metadata (212, 232), the metadata comprising one or more parameters corresponding to the digital film grain; and A processor (320, 340), the processor being configured to: decompressing the compressed video bitstream to generate a respective decompressed representation of each of the images (330), each decompressed representation comprising a respective host image component and a respective film grain component; as well as A sequence of corresponding decompressed representations is temporally averaged (910) using a plurality of temporal sliding windows, each of the temporal sliding windows corresponding to a corresponding image frame pixel, at least some of the temporal sliding windows having different corresponding lengths selected based on the metadata.

14. The video transmission system according to claim 13, further comprising a video encoder, the video encoder comprising: an output interface, the output interface outputting the encoded bit stream for the video encoder; as well as A video compression module is used to generate the compressed video bit stream using lossy compression according to a video compression standard.

15. A machine-implemented method for removing digital film grain from video data, the method comprising: receiving an encoded bitstream (122) comprising a compressed video bitstream (252) having encoded therein a sequence of images including digital film grain and metadata (212, 232) including one or more parameters corresponding to the digital film grain; decompressing (320) the compressed video bitstream to generate a respective decompressed representation (330) of each of the images, each decompressed representation comprising a respective host image component and a respective film grain component; computing, based on the metadata, a respective estimate of the respective film grain component of each decompressed representation using a simulated film grain image (430) and an approximation of a modulation function used to modulate the digital film grain (420); and The respective estimate is removed (340) from the respective decompressed representation to produce a corresponding estimate (342) of the respective host image component.

Images