Method and apparatus for transforming film grain metadata - Patents.com

Abstract

A method and apparatus for converting film grain metadata is provided. In one embodiment, first film grain metadata is obtained, the first film grain metadata conforming to a first film grain syntax and representing a first film grain model. At least a portion of the first film grain metadata is converted to at least a portion of second film grain metadata, the second film grain metadata conforming to a second film grain syntax and representing a second film grain model, the second film grain syntax being different from the first film grain syntax, or the second film grain model being different from the first film grain model, or both the second film grain syntax and the second film grain model being different from the first film grain syntax and the first film grain model.

Description

[Technical field]

[0001] The present embodiments relate generally to video compression, distribution and rendering, and more particularly to film grain modeling. The present embodiments relate to methods and apparatus for film grain metadata conversion between codecs and / or models. [Background technology]

[0002] To achieve high compression efficiency, image and video coding schemes usually employ prediction and transformation to exploit spatial and temporal redundancy in video content. In general, intra- or inter-prediction is used to exploit intra- or inter-picture correlation, and then the difference between the original block and the predicted block, often called the prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to the entropy encoding, quantization, transformation, and prediction. Summary of the Invention

[0003] According to one aspect, a method is provided for converting film grain metadata, the method including obtaining first film grain metadata, the first film grain metadata conforming to a first film grain syntax and representing a first film grain model, and converting at least a portion of the first film grain metadata to at least a portion of second film grain metadata, the second film grain metadata conforming to a second film grain syntax and representing a second film grain model, the second film grain syntax being different from the first film grain syntax, or the second film grain model being different from the first film grain model, or both the second film grain syntax and the second film grain model being different from the first film grain syntax and the first film grain model.

[0004] According to another aspect, an apparatus for converting film grain metadata is provided, the apparatus comprising one or more processors configured to: obtain first film grain metadata, the first film grain metadata conforming to a first film grain syntax and representing a first film grain model; and convert at least a portion of the first film grain metadata to at least a portion of second film grain metadata, the second film grain metadata conforming to a second film grain syntax and representing a second film grain model, the second film grain syntax being different from the first film grain syntax, or the second film grain model being different from the first film grain model, or both the second film grain syntax and the second film grain model being different from the first film grain syntax and the first film grain model.

[0005] One or more embodiments also provide a computer program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method for transforming film grain metadata according to any of the embodiments described herein. One or more of the embodiments also provide a computer-readable storage medium having stored thereon instructions for transforming film grain metadata according to the methods described herein. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 illustrates a block diagram of a system in which aspects of the present embodiments may be implemented. [Diagram 2] FIG. 2 shows a block diagram of one embodiment of a video encoder. [Diagram 3] FIG. 3 shows a block diagram of one embodiment of a video decoder. [Figure 4] FIG. 4 shows an example of a film grain model within a video coding standard. [Diagram 5] FIG. 5 shows possible transformation directions for film grain metadata. [Figure 6] FIG. 6 shows possible conversion directions for cross-codec and cross-model conversion. [Figure 7] FIG. 7 shows a block diagram of an example of film grain usage in a video coding framework. [Figure 8] FIG. 8 illustrates a block diagram of an example method for converting film grain metadata from a first format to a second format, according to one embodiment. [Figure 9] FIG. 9 shows an example of a possible location of the film grain transformation in the encoding / decoding chain. [Figure 10] FIG. 10 illustrates a block diagram of an example of a method for converting film grain metadata from a first format to a second format according to another embodiment. [Figure 11] Figure 11 shows an example of the use of the update_grain and film_grain_params_ref_idx syntax elements for AV1 film grain metadata. [Figure 12] FIG. 12 shows another example of the use of the update_grain and film_grain_params_ref_idx syntax elements for AV1 film grain metadata. [Figure 13] FIG. 13 illustrates a block diagram of an example of a method for converting film grain metadata from a first format to a second format according to another embodiment. [Figure 14] FIG. 14 illustrates a block diagram of an example of a method for converting film grain metadata from a first format to a second format according to another embodiment. [Figure 15] FIG. 15 shows an example of matching between film grain patterns obtained from different film grain model parameter sets. [Figure 16] FIG. 16 illustrates a block diagram of a system in which aspects of the present embodiment may be implemented, according to another embodiment. [Figure 17] FIG. 17 shows two remote devices communicating over a communications network according to an example of the present principles. [Figure 18] FIG. 18 shows the syntax of a signal according to an example of the present principles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] In this application, various aspects are described, including tools, features, embodiments, models, methods, and the like. Many of these aspects are described with specificity and in what may often sound definitive terms to at least indicate their individual characteristics. However, this is for purposes of clarity of description and does not limit the applicability or scope of the aspects. In fact, all of the different aspects can be combined and substituted to provide further aspects. Moreover, these aspects can also be combined and substituted with aspects described in previous applications as well.

[0008] The aspects described and contemplated in this application can be implemented in many different forms. Figures 1, 2, and 3 below provide some embodiments, but other embodiments are contemplated, and discussion of Figures 1, 2, and 3 is not intended to limit the scope of implementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0009] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture", and "frame" may be used interchangeably.

[0010] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. In addition, terms such as "first", "second", etc. may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding". The use of such terms does not imply a modified ordering of operations, unless specifically required. Thus, in this embodiment, the first decoding need not be performed before the second decoding, but may occur, for example, before, during, or during an overlapping time with the second decoding.

[0011] Various methods and other aspects described herein can be used to modify modules, such as the forward transform module (225), the inverse transform module (250), and the entropy coding module (245) of the video encoder 200 shown in FIG. 2, and the inverse transform module (350) and the entropy decoding module (330) of the video decoder 300 shown in FIG. 3. Furthermore, aspects of the present disclosure are not limited to VVC or HEVC, but can be applied to other standards and recommendations, such as existing or future developments, and extensions of any such standards and recommendations, including VVC and HEVC. Unless otherwise specified or technically precluded, aspects described herein can be used individually or in combination.

[0012] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 100, alone or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 100 are distributed across multiple ICs and / or separate components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described herein.

[0013] The system 100 includes at least one processor 110 configured to execute instructions loaded therein, for example, to implement various aspects described herein. The processor 110 may include embedded memory, input / output interfaces, and various other circuits as known in the art. The system 100 includes at least one memory 120 (e.g., volatile and / or non-volatile memory devices). The system 100 includes a storage device 140, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 140 may include, by way of non-limiting example, an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0014] The system 100 includes, for example, an encoder / decoder module 130 configured to process data to provide encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 130 represents a module(s) that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding and decoding module. Additionally, the encoder / decoder module 130 may be implemented as a separate element of the system 100 or may be incorporated within the processor 110 as a combination of hardware and software, as is known to those skilled in the art.

[0015] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and then loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, expressions, operations, and operational logic.

[0016] In some embodiments, memory internal to the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 also refers to ISO / IEC 13818, 13818-1 also known as H.222, and 13818-2 also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, H.265 and MPEG-H Part 2 are also known), or VVC (Versatile Video Coding, a new standard being developed by the Joint Video Experts Team (JVET)).

[0017] Inputs to the elements of system 100 may be provided through a variety of input devices, as shown in block 105. Such input devices may include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, an RF signal transmitted throughout a broadcast by a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples include composite video, although not shown in FIG. 1.

[0018] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF section may be associated with appropriate elements to (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) band-limit again to a narrower frequency band to select a signal frequency band that may be referred to (for example) as a channel in a particular embodiment, (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs these various functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and refiltering RF signals transmitted over a wired (e.g., cable) medium to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0019] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 110, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 110, as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and an encoder / decoder 130 operating in combination with memory and storage elements that process the data streams as desired for presentation on an output device.

[0020] The various elements of system 100 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between each other using suitable connection arrangements 115, e.g., internal buses known in the art, including I2C buses, wiring, and printed circuit boards.

[0021] System 100 includes a communication interface 150 that enables communication with other devices over a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented in a wired and / or wireless medium, for example.

[0022] Data is streamed to the system 100 in various embodiments using a Wi-Fi network such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received by a communication channel 190 and communication interface 150 adapted for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, the streamed data is provided to the system 100 using a set-top box that delivers data through an HDMI connection of the input block 105. In yet other embodiments, the streamed data is provided to the system 100 using an RF connection of the input block 105. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0023] The system 100 may provide output signals to various output devices, including a display 165, speakers 175, and other peripheral devices 185. The display 165 in various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 may be for a television, a tablet, a laptop, a mobile phone, or other device. The display 165 may also be integrated with other components (e.g., as in a smart phone) or may be separate (e.g., an external monitor for a laptop). The other peripheral devices 185, in various example embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (DVR in both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 185 that provide functionality based on the output of the system 100. For example, a disc player performs the functionality of playing the output of the system 100.

[0024] In various embodiments, control signals are communicated between system 100 and display 165, speaker 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that allow inter-device control with or without user intervention. Output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speaker 175 may be integrated into a single unit with other components of system 100, for example, in an electronic device such as a television. In various embodiments, display interface 160 includes a display driver, for example, a timing controller (TCon) chip.

[0025] Display 165 and speakers 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which display 165 and speakers 175 are external components, output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0026] The embodiments may be performed by computer software implemented by the processor 110, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 120 may be of any type suitable for the technology environment, and may be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memory devices. The processor 110 may be of any type suitable for the technology environment, and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0027] 2 shows an encoder 200. Variations of this encoder 200 are contemplated, but for clarity, the encoder 200 is described below without describing all possible variations.

[0028] In some embodiments, FIG. 2 also illustrates an encoder that improves on the HEVC standard or employs technology similar to HEVC, such as the Versatile Video Coding (VVC) encoder under development by the Joint Video Exploration Team (JVET).

[0029] Before being encoded, a video sequence may undergo encoding pre-processing (201), such as applying a color transformation to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components to obtain a signal distribution that is more resistant to compression (e.g., using histogram equalization of color components) or resizing (ex: downscaling) the picture. Metadata may be associated with the pre-processing and attached to the bitstream.

[0030] In the encoder 200, a picture is coded by the encoder elements as described below. The picture to be coded is divided (202) into units, e.g., CUs, and processed. Each unit is coded, e.g., using either intra mode or inter mode. When a unit is coded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and motion compensation (270) are performed. The encoder decides (205) whether to use intra mode or inter mode to code the unit, and indicates the intra / inter decision, e.g., by a prediction mode flag. The encoder may also mix (263) intra and inter prediction results, or mix results from different intra / inter prediction methods. A prediction residual is calculated (210), e.g., by subtracting the predicted block from the original image block.

[0031] The motion refinement module (272) uses already available reference pictures to refine the motion field of a block without referring to the original block. The motion field for a region can be considered as a collection of motion vectors for all pixels that comprise the region. If the motion vectors are subblock-based, the motion field can also be represented as a collection of all subblock motion vectors in the region (all pixels in a subblock have the same motion vector, and the motion vectors can be different for each subblock). If a single motion vector is used for a region, the motion field for the region can also be represented by a single motion vector (the same motion vector for all pixels in the region).

[0032] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, as well as the motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying a transform or quantization process.

[0033] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (255) to reconstruct an image block. An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / Sample Adaptive Offset (SAO) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0034] Figure 3 shows a block diagram of a video decoder 300. In the decoder 300, the bitstream is decoded by decoder elements as described below. The video decoder 300 generally performs a decoding path that is the inverse of the encoding path described in Figure 2. The encoder 200 also generally performs video decoding as part of encoding the video data.

[0035] In particular, the decoder's input includes a video bitstream, which may be generated by video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. The decoder may then partition the picture according to the decoded picture partition information (335). The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (355) to reconstruct an image block.

[0036] A prediction block can be obtained (370) from intra prediction (360) or motion compensated prediction (i.e., inter prediction) (375). The decoder may mix (373) intra and inter prediction results, or mix results from multiple intra / inter prediction methods. Before motion compensation, the motion field may be improved (372) by using already available reference pictures. An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0037] The decoded picture may further undergo post-decoding processing (385), such as an inverse color conversion (e.g., from YCbCr 4:2:0 to RGB 4:4:4), or an inverse remapping or resizing (ex: upscaling) of the reconstructed picture that performs the inverse of the remapping process performed in the pre-encoding processing (201). The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0038] Film grain is a specific type of noise that appears in video that is very pleasing and very distinctive. In its essence, film grain is the result of the physical process of analog film stock exposure and development. Silver halide crystals, a light-sensitive compound that is part of the film stock, become tiny specks of metallic silver when exposed to sufficient light. Exposure to light, controlled by the camera lens, induces a chemical process in the silver halide crystals that is proportional to the amount of light absorbed by each crystal. Because not all crystal specks are exposed equally, and because they appear in different sizes and shapes and are randomly distributed within the film emulsion, some of them develop into metallic silver and some do not. Furthermore, within uniformly exposed areas, some crystals grow while others do not. Underexposed and undeveloped crystals undergo chemical removal during film development, leaving small gaps between the reacted ones. During negative printing, the empty gaps allow light to pass through, creating the visual sensation of film grain. Furthermore, the growing crystals block the light from passing through, and the more reactions that occur in the first place, the more light is blocked by the growing specks, resulting in different intensity levels in the final palette. Finally, these atomic changes are seen as a larger phenomenon, which, when all combined, is perceived as an image with a grainy texture.

[0039] Moreover, the formation of film grain is random due to the (quasi-)random formation of metallic silver. More precisely, this process is quasi-random because the physical properties of photographic film are controlled by design and during manufacturing. For example, silver crystals are not perfectly and uniformly distributed in a lattice pattern, and film stock has many different sizes of crystals in it. The larger the crystals are, the more sensitive they are to light. The more sensitive the film is to light, the larger and more visible the grain will appear. Also, the size of the silver crystals in the original emulsion, the physical granularity of the crystals, the amount of exposure variation a given film can tolerate, the development time, and even the development temperature are some parameters that affect the visibility and subjective sensation of film grain.

[0040] Technically speaking, film stock is a product of the past and can be considered primitive from today's point of view. The digital age has offered us vast new possibilities for producing and consuming content. However, in the digital age, film grain is no longer present during the creation of content. A crisp video without noise can be too perfect, which, contrary to expectations, leads to a lower subjective experience for the viewer. Therefore, film grain is often added in post-processing to improve the sharpness of digitally captured scenes and / or to better contribute to the expression of creative intent in the digital world.

[0041] It is widely known today that moderately grainy textures exhibit desirable characteristics. Digital artists choose film grain patterns that fit their desired look and feel. However, it is difficult to preserve film grain throughout the entire video processing and distribution chain. Visually lossless encoding that does not change or destroy the film grain is only possible at very high bit rates. This is only feasible for high-quality applications, e.g., cinema-grade movie quality. For general video applications, it consumes a lot of bandwidth. Therefore, stronger lossy compression is an inevitable step within current content distribution networks. It should be noted that before distribution, videos also usually undergo various preprocessing steps, where noise filtering is presented to improve coding efficiency. As a result, in the various steps of filtering and lossy compression, the film grain is suppressed without reconstruction.

[0042] To address this issue, modern video compression and delivery systems can provide mechanisms to remove film grain before and / or during compression, and to add it back automatically and in a controllable manner at the user's end by using a parameter model. Figure 7 shows a block diagram of an example of film grain usage in a video coding framework. Typically, an encoder removes the original film grain from the video (700), adjusts the film grain model parameters (automatically or manually at 701), and a decoder replaces it with a synthetic substitute that is visually closer to the one that was removed (704). In this way, information about the film grain is communicated as additional metadata.

[0043] The automatic analysis of the original film grain (film grain analysis consisting of pre-processing 700 for filtering and encoding the input video and film grain (FG) estimation 701) is an optional step. The model parameters can be estimated using an algorithm / automatic technique or can be manually adjusted by an expert. In many cases, a person skilled in the art provides a manually selected set of film grain parameters (FG parameters) to the encoder (702) that encodes the filtered video into a bitstream. However, in some cases, the automatic estimation of parameters can be used directly to automate the process or can be used as an initialization step before manual adjustment. In this way, the goal is to add synthetic grain back to the video while keeping in mind that the artistic intent remains unchanged. The FG parameters are then communicated to the decoder (703) as additional metadata, for example as a film grain characteristics (FGC) SEI message. The decoder (703) decodes the bitstream and outputs the decoded video and the decoded FG parameters. Then, the FG synthesis (704) adds a film grain pattern to the decoded video by using the FG parameters to provide a decoded video with film grain as an output.

[0044] Film grain has also been receiving more attention in the compression community, where another, more technical use of film grain is presented. It can be synthesized for video compression purposes to mask compression artifacts. In such a scenario, when stronger compression is used, the film grain is removed by the compression itself, and other compression artifacts appear at the same time. Film grain can then be added to the compressed video to mask compression artifacts, even if they are not present in the original content. Adding noise to the video during playback disguises the compression, making the video more visual without bandwidth cost. Note that in this scenario, it is not very important that the film grain matches what was removed when presented in the source video. The important aspect is to re-noise the video by adding film grain to mask flat, monotonous areas and other compression artifacts resulting from compression. Today's activity within the video compression community is directed towards compression masking use cases, rather than preserving the same look of film grain for artistic purposes.

[0045] Currently, the latest and most cutting-edge video coding standards support film grain metadata. For example, AV1 (AV1 Bitstream & Decoding Process Specification, The Alliance for Open Media, Version 1.0.0 with Errata 1, January 18, 2019) and Versatile Video Coding (VVC, Versatile Video Coding, Recommendation ITU-T H.266(08 / 2020) | International Standard ISO / IEC 23090-3) are two different coding standards that allow for the delivery of film grain metadata. HEVC (High efficiency video coding, Recommendation ITU-T H.265(08 / 2021)) and H.264 / AVC (Advanced video coding for generic audiovisual services, Recommendation ITU-T H.264(08 / 2021)) also provide a means to transfer film grain model parameters to the decoder, similar to VVC (these standards share the same film grain metadata).

[0046] Regarding film grain modeling, various approaches have been researched. In the context of VVC, frequency filtering solutions can be used to parameterize and resynthesize film grain to recover film grain in compressed video. VVC also supports the same autoregressive models as AV1. Even though both VVC and AV1 support autoregressive techniques, the models are not the same.

[0047] AV1 and VVC define film grain metadata in their specifications. VVC does so through the Versatile Supplemental Enhancement Information messages for coded video bitstreams (VSEI) specification (Versatile Supplemental Enhancement Information messages for coded video bitstreams, Recommendation ITU-T H.274 (08 / 2020) | International Standard ISO / IEC 23002-7), while AV1 incorporates film grain syntax in the frame headers and film grain metadata is part of the specification itself. Thus, support for film grain is specified in significantly different ways in these two standards. Depending on the codec and / or model, different syntaxes and semantics are used. As shown in FIG. 4, different models can be used, so the compositing process and film grain mixing are also defined differently.

[0048] Some embodiments of the present disclosure provide a means for converting film grain model parameters and associated syntax from one format to another suitable format having a standardized syntax.

[0049] Some embodiments described herein transform a set of metadata or syntax elements into a modified set of syntax elements that better meets the needs of an end user (eg, a decoder).

[0050] 5 and 6 provide examples of transformation directions. There are several possible transformations. 1. Convert AV1 film grain metadata to VVC film grain metadata and vice versa for the autoregressive model. 2. Convert AV1 film grain metadata to VVC film grain metadata and vice versa for the frequency filtering model. 3. Convert VVC film grain metadata for the autoregressive model to VVC film grain metadata for the frequency filtering model and vice versa. The same can be applied for HEVC and H.264 / AVC instead of VVC. 4. Conversion from frequency filtering model to autoregressive model, and vice versa, applied to any combination of H.264 / AVC, HEVC, and VVC.

[0051] According to one embodiment, a framework is provided for converting film grain syntax and film grain model parameters from one format to another. The conversion process is not a simple copy process because different syntax elements and semantics, including different models, are used. Also, the conversion does not necessarily have to be between codecs. For example, VVC film grain metadata supports two different film grain models, and in some situations it is useful to enable conversion between models of the same codec. It is noted that other video coding standards, such as High Efficiency Video Coding (HEVC) and H.264 / AVC, also define film grain metadata similar to VVC. Thus, the conversion framework provided herein applies to them as well as for VVC. In addition, if model conversion is required, film grain metadata conversion between VVC, HEVC, or H.264 / AVC can be defined. Such a scenario is illustrated in FIG. 6.

[0052] On the other hand, if conversion of film grain metadata between VVC, HEVC or H.264 / AVC is required but model conversion is not required, a simple copy can be performed. In fact, if model conversion is not performed, film grain metadata can be shared between VVC, HEVC or H.264 / AVC since the metadata format is the same.

[0053] Also, after transformation of the metadata including the model parameters, the synthesized new film grain pattern should match (at least visually) the original film grain pattern. However, in some embodiments, this requirement can be relaxed. Note that the transcoding process does not introduce any new syntax. The idea is to produce metadata that will be embedded directly in the bitstream after transformation.

[0054] In the following, for simplicity, the framework provided herein is described for the case of AV1 and VVC standards. However, the general principles described herein also apply to other video coding standards, such as HEVC and H.264 / AVC. It should be noted that HEVC and H.264 / AVC define the same film grain metadata as VVC. Therefore, it is not necessary to define all the transformation directions from FIG. 6. More precisely, all the transformation directions from FIG. 6 can be clearly defined by only defining the transformation between VVC models. Other transformation directions can reuse the transformations defined between VVC models. For example, transforming a HEVC frequency filtering model to a VVC autoregressive model variant-2 is the same as transforming a VVC frequency filtering model to a VVC autoregressive model variant-2. Or, transforming from H.264 / AVC autoregressive model variant-2 to a HEVC frequency filtering model is the same as transforming from VVC autoregressive model variant-2 to a VVC frequency filtering model. Therefore, in the following, all definitions and examples that apply to VVC also apply to H.264 / AVC and HEVC.

[0055] It is an objective of this disclosure to provide a framework for converting film grain metadata and associated film grain model parameters into an appropriate format (eg, a format supported by a video decoder).

[0056] As shown in FIG. 9, the transformation (FG transformation) can be achieved at different positions in the video encoding / decoding chain depending on the use case. 1. A video encoder (901) provides film grain metadata in format 1. A video decoder (903) supports metadata format 2. In this example, format 1 is not understandable by the decoder, and a conversion from format 1 to format 2 can be performed by the encoder before embedding the film grain metadata into the bitstream. This can be implemented in the encoder (901) (or immediately after as in step 902), but the conversion can also be performed before the encoder, for example in a pre-processing step (900). 2. In another example, the decoder (903) supports format 2, but the metadata coming from the bitstream is in format 1. Therefore, processing is needed in the decoder to produce a proper output (decoded video) with film grain. The conversion can be performed at a stage between the encoder and the decoder, before the bitstream is parsed by the decoder (902). 3. In another example, the decoder (903) supports format 1 coming from the bitstream. The decoder can extract film grain metadata from the bitstream. However, the implementation of film grain synthesis does not support film grain modeling using format 1. The transformation of the film grain metadata can be done in the post-processing stage (904 or 905) to perform synthesis and produce a film grain output with a target quality. For example, such a use case can occur in VVC, where the FGC SEI is sent to the decoder by incorporating a frequency filtering model. The decoder can parse the syntax, but on the decoder (or post-processing) side, synthesis by using an autoregressive model is implemented. 4. In another example, an encoder is provided with film grain metadata in format 1, which is embedded in the bitstream during encoding. However, during bitstream transcoding, some decoders may require format 2. In such a case, film grain metadata conversion may be performed after encoding and before decoding (e.g., during bitstream transcoding). In such a case, film grain metadata conversion is performed in step 902. An example of such a use case may be transcoding an AV1 bitstream to a VVC bitstream. In such a case, if film grain metadata is present in the AV1 bitstream, it should be converted rather than discarded.

[0057] In all cases, film grain metadata transformation should be done such that the resulting synthesized film grain visually matches (or closely resembles) the appearance of the original film grain.

[0058] In some embodiments, this requirement may be relaxed.

[0059] Some other requirements regarding complexity or memory usage may be imposed as well. Note that the transcoding process does not introduce any new syntax. The aim of the present embodiment is to produce metadata that will be embedded directly in the bitstream after transcoding.

[0060] Some embodiments relate to converting metadata coming from one film grain metadata format to another film grain metadata format, where the format differs from the video coding standard and / or film grain model used to synthesize the film grain pattern.

[0061] The principles described herein are based on examples taken from currently available standards and models supported therein, however the principles may also be applied to other new standards and new models that may become available in the future.

[0062] As mentioned above, the conversion process described herein can be implemented before encoding of the video content, during encoding, after encoding but before decoding, during decoding, or after decoding of the video and before creating and mixing film grain.

[0063] The use of such a mechanism may be as follows. 1. Allow and ensure that signaling of film grain parameters to the decoder before or during encoding is supported by the decoder itself. 2. At the decoder or post-processing stage when the film grain synthesis implementation does not support the required model, i.e. when the decoder receives a bitstream with film grain parameters that are not supported by the film grain synthesis implementation, the decoder can convert the received film grain metadata and model parameters into an appropriate / understandable format / model. 3. During transcoding of bitstream to ensure film grain synthesis, it is performed at the decoder side after transcoding is finished.

[0064] It is important to note that transformation of metadata does not necessarily require encoding and / or decoding of the video bitstream. For example, transformation of film grain syntax can be done based on metadata provided in a configuration file. It can also be done by analyzing the encoded bitstream without decoding it, since existing metadata can be extracted from the bitstream (and added back) without actually decoding the bitstream. By doing so, only the film grain portion can be extracted, decoded, and then processed and backfilled into the bitstream in the right places.

[0065] 8 illustrates a method for converting film grain metadata from a first format to a second format. At 800, a first set of film grain metadata is obtained. The first set of film grain metadata conforms to a first film grain metadata format. The film grain metadata format may include a specific syntax for transmitting syntax elements related to film grain data, the syntax conforming to a video coding standard and / or parameters of a film grain model. In some embodiments, the first set of film grain metadata conforms to a first film grain syntax and represents a first film grain model.

[0066] Depending on the embodiment, as described above, the transformation can be performed at different locations within the video coding framework, so obtaining the first set of film grain metadata may include reading film grain metadata stored in a memory, or estimating the film grain metadata from the input video, or receiving and parsing a bitstream including the film grain metadata, or receiving decoded film grain metadata from a decoder.

[0067] At 801, at least a portion of the first set of film grain metadata is converted to at least a portion of a second set of film grain metadata that conforms to a second format, such that the second set of film grain metadata conforms to a second film grain syntax and represents a second film grain model.

[0068] In a variation, the method may also include transmitting a second set of the converted film grain metadata together with the coded video to a decoder or post-processing device.

[0069] As described above, the conversion may include converting the first set of film grain metadata from a first video standard to a second video standard, where the video standard defines a film grain syntax, and / or converting the first set of film grain metadata from a first film grain model to a second film grain model, where the first and second film grain models belong to the same or different video compression standards.

[0070] Below, some examples of film grain syntax and models are described for the AV1 and VVC video coding standards.

[0071] In AV1, film grain is based on an autoregressive model (autoregressive model variant-1). The syntax is defined in AV1 Bitstream & Decoding Process Specification, The Alliance for Open Media, Version 1.0.0 with Errata 1, January 18, 2019, Section 5.9.30: Film grain params syntax. The parameters semantics are given in Section 6.8.20: Film grain params semantics. The synthesis process, how to generate film grain patterns and how to blend them into the image, is described in Section 7.18.3: Film grain synthesis process.

[0072] Note that in AV1, film grain compositing is a mandatory part of post-processing, which means that if film grain parameters are present in the bitstream, the compositing process will be invoked without exception, resulting in an output video with film grain.

[0073] The MPEG codec family (VVC, HEVC, H264 / AVC) supports two different models in its structure: the autoregressive model (autoregressive model variant-2) and the frequency filtering model.

[0074] Film grain metadata in VVC (as well as in HEVC and H.264 / AVC) is communicated to the decoder via the Supplementary Enhancement Information (SEI) message. The syntax is defined in Versatile Video Coding, Recommendation ITU-T H.266 (08 / 2020) | International Standard ISO / IEC 23090-3, Section 8.5: Film grain characteristics SEI message. The complete syntax and semantics are defined for both available models. In contrast to AV1 film grain, the VVC film grain specification only provides a means of how to transfer film grain parameters to the decoder. However, it does not provide a precise description of the compositing process. An example of a compositing implementation based on a frequency filtering model using the FGC SEI message as defined in VVC is provided in Sean McCarthy, Peng Yin, Walt Husak, Fangjun Pu, Taoran Lu, Tao Chen, Edouard Francois, Milos Radosavljevic, Vijayakumar GR, Kaustubh Patankar, Shireesh Kadaramandalgi, Ajayshyam, "AHG9: Fixed-point grain blending process for film grain characteristics SEI message", JVET-W0095, Teleconference, 7-16 July, 2021. Also, adding film grain is not a mandatory process in VVC even if film grain SEI is present in the bitstream (film grain SEI can simply be discarded by the decoder).

[0075] In general, there are two subsets of syntax elements used in film grain metadata. The first subset consists of syntax elements that are not directly related to model parameters. They are considered as additional syntax used to support the transfer of model parameters. This can be called general film grain metadata.

[0076] The second subset consists of model-specific film grain parameters.

[0077] Examples of these two sets for VVC are given below in Table 1. Note that the model-specific parameters are the same for frequency filtering and autoregressive model variant-2. However, the interpretation of the syntax differs based on the model.

[0078] [Table 1-1]

[0079] [Table 1-2]

[0080] For simplicity, we assume here that fg_separate_colour_description_present_flag is set to 0. The given flag is needed in a specific use case where the decoded image is not in the same colour representation gamut as the simulated film grain, where we can assume that the film grain and video colour spaces are the same.

[0081] The film grain metadata for AV1 is different, as shown in Table 2 below.

[0082] [Table 2-1]

[0083] [Table 2-2]

[0084] [Table 2-3]

[0085] Based on the semantics of some film grain syntax elements, a mapping between them can be defined. For general film grain metadata, a simple relationship can be defined. For model-specific parameters, the conversion is more complex and should be implemented with respect to requirements on the quality of the film grain and with respect to preserving the look of the original film grain (if it is one of the requirements). Other requirements related to complexity or memory requirements may be imposed. For example, if computational resources are limited, the conversion process can only convert the general film grain metadata and use default values ​​for the model-specific parameters.

[0086] Several embodiments for transforming the film grain metadata given in Tables 1 and 2 are described below. Different transformations depending on the input and output formats are shown in Fig. 4. The embodiments provided herein can be implemented in different ways with different requirements regarding memory, complexity, and / or final quality of the synthesized film grain (after metadata transformation).

[0087] According to one embodiment, different conversion modes are provided, each corresponding to a variant of the present principles, which are summarized as follows: 1. Discard - According to this variant, the transformation is performed such that a minimal subset of parameters is provided so that the decoder can handle the bitstream syntax, but does not synthesize the film grain. This is an alternative method that does not render the film grain and can produce an output that is significantly different from the expected one. However, it is a less demanding process in terms of computation and memory. 2. Random - According to this variant, the general film grain metadata is transformed, but for the specific model parameters, random values ​​are used. 3. Default - According to this variant, the general film grain metadata is converted, but for model-specific parameters default values ​​are used that are defined a priori for each model type. In total, three groups of default parameters can be provided, one for each model (frequency filtering, autoregressive variant-1 and autoregressive variant-2). Furthermore, for each model several different configurations can be defined a priori, reflecting different film grain appearances. 4. Mapping - According to this variant, the complete film grain metadata is transformed with as much accuracy as possible. The aim is to achieve a visual match after transformation with the original film grain pattern before transformation.

[0088] The different modes reflect different complexity, memory and quality requirements. In all cases, the original video coding stream remains intact, only the metadata is transformed or discarded / ignored.

[0089] According to this embodiment, the conversion mode can be selected by the device implementing the conversion based on the requirements. The selection can be made among the conversion modes mentioned above or among a subset of these modes. According to another embodiment, the selection can be made by the user.

[0090] According to another embodiment, each of the variations described herein may be implemented independently of the other variations, in which case there is no need for selection of the conversion mode by the device, since the device may implement only one of the variations.

[0091] Variation: Discard Mode In a variant called discard mode, the process simply ignores the existing film grain metadata and does not perform film grain compositing. Instead, it removes / ignores the incoming film grain metadata or sets an appropriate flag so that film grain compositing is not performed later. There are several ways to implement this depending on the conversion direction:

[0092] In a first variant, if VVC film grain metadata needs to be converted to AV1 metadata (the model type is not important in this particular mode), then either the film_grain_params_present or apply_grain flag should be set to 0. Since film_grain_params_present or apply_grain is 0, no other syntax elements are required.

[0093] In another variant, if AV1 film grain metadata needs to be converted to VVC film grain metadata in discard mode, one option is to simply ignore all metadata and not embed it in the VVC bitstream. This is possible because film grain metadata is not a mandatory part of the VVC bitstream and there is no strict requirement to create a film grain SEI and have it coded in the bitstream. Another option is to create an SEI and embed it in the bitstream, but fg_characteristics_cancel_flag should be set to 1. Since fg_characteristics_cancel_flag is 1, no other syntax elements are required.

[0094] In another variant, when the conversion between VVC models is performed in discard mode (VVC autoregressive variant-2 to VVC frequency filtering or vice versa), the same as in the previous example applies. By simply discarding the film grain SEI from the bitstream, or by setting fg_characteristics_cancel_flag to 1, the decoder can handle the bitstream without other syntax elements having to be converted. However, it will not produce an output with film grain.

[0095] The discard mode is a simple solution that does not require a lot of processing power or memory, and therefore may produce undesirable results when film grain is expected to be synthesized and added to the decoded video.

[0096] Variation: Random mode FIG. 10 shows an example of a method for converting film grain metadata from a first format to a second format according to another variant, referred to herein as a random mode. In this variant, only the general film grain metadata is converted and the model-specific film grain parameters are set randomly. In 1000, the general film grain metadata is converted by mapping one or more syntax elements of the first format to one or more syntax elements of the second format. In 1001, random values ​​are assigned to the model-specific parameters. In such a case, the model-specific parameters may be limited by a supported range and the parameters are initialized by taking a random value from that range.

[0097] VVC~VVC In one example, the simplest conversion is between VVC models: all the general film grain metadata is kept the same except for the fg_model_id flag. It should be changed so that the new value is represented by ~fg_model_id, where ~ is the negation of the flag. Then the model-specific film grain parameters are set randomly with respect to the supported value range.

[0098] VVC~AV1 and vice versa Conversion between VVC and AV1 or vice versa is more complex. It requires mapping of generic film grain metadata. Model specific film grain parameters are set randomly with respect to the range of supported values ​​as in the previous example. In such an example, the conversion is established as follows: There are two stages: a mapping stage and an extension stage. The mapping stage produces relationships between existing parameters as well as between parameters where logical and / or mathematical relationships can be observed. The extension stage adds other parameters not mapped in the first stage.

[0099] VVC~AV1 In another example, in the mapping stage, fg_characteristics_cancel_flag maps to film_grain_params_present and apply_grain. If fg_characteristics_cancel_flag is set to 1, film_grain_params_present and apply_grain are set to 0. Then, there is no need to send other syntax elements and therefore no need to convert them. If fg_characteristics_cancel_flag is set to 0, film_grain_params_present and apply_grain are set to 1.

[0100] Since only one model is available in AV1, there is no need to convert the fg_model_id flag. Whether the VVC model needs to be converted to an AV1 model or not, it is converted in the same way. Therefore, this conversion is applied to both the frequency filtering model and the autoregressive model variant-2 when they need to be converted to the autoregressive model variant-1.

[0101] Also, since only additive blending is defined in AV1, there is no need to convert fg_blending_mode_id. Even if fg_blending_mode_id indicates that multiplicative blending is used within the VVC film grain modeling, the result of this random mode conversion should not be affected since random parameters are used to generate the film grain.

[0102] If fg_characteristics_persistence_flag is 0 (meaning that the film grain applies only to the current decoded picture), it indicates that update_grain should be 1 (meaning that new film grain parameters need to be sent).

[0103] If fg_characteristics_persistence_flag is 1, then update_grain is still 1 for the current frame. However, it does affect future frames: for future frames, update_grain is set to 0 and film_grain_params_ref_idx is set to the index of the current frame (as defined in the VSEI film grain specification) until one of the following conditions is met: - A new CLVS for the current layer begins - The bitstream ends - The pictures in the current layer in the access unit associated with the film grain characteristics SEI message that follow the current picture in output order are output.

[0104] Thus, after one of the given conditions is met, a possibly new film grain SEI is sent in the VVC bitstream and the process is repeated. Essentially, this means that the film grain metadata is reused for all future frames until one of the given conditions applies. Thus, for future frames that do not carry a film grain SEI, the film grain is still applied until one of the conditions is met. The given condition essentially resets the "scope" of the previous film grain SEI message. In AV1, it is equivalent to setting update_grain to 0, which means that the parameters from one of the previous frames are used. Compared to VVC, in AV1, the new set of parameters needs to be set for every frame until it is embedded in the bitstream (the new set of parameters is indicated when update_grain is set to 1).

[0105] In the mapping stage, for some syntax elements it is not possible to provide a direct (or possibly indirect) mapping. In such cases, a priori defined "best effort" values ​​can be provided in the extension stage. In other words, some specific implementations of the conversion process can use predefined values ​​for parameters for which no mapping exists. These parameters are selected to produce the best possible film grain quality.

[0106] The missing parameters are overlap_flag and grain_seed. overlap_flag defines whether deblocking between film grain blocks is performed. For example, in Sean McCarthy, Peng Yin, Walt Husak, Fangjun Pu, Taoran Lu, Tao Chen, Edouard Francois, Milos Radosavljevic, Vijayakumar GR, Kaustubh Patankar, Shireesh Kadaramandalgi, Ajayshyam, "AHG9: Fixed-point grain blending process for film grain characteristics SEI message", JVET-W0095, Teleconference, 7-16 July, 2021, which represents one possible implementation design for VVC film grain compositing, deblocking is used by default without an option to disable it. In such a case, it makes sense to set overlap_flag to 1. However, this may be a design choice since it uses additional resources. grain_seed is used in the pseudo-number generator during the blending process. Both compositing algorithms (VVC and AV1) require a seed to be initialized for each frame to ensure bit-accurate playback, so it can be reused between algorithms. The only difference is that VVC does not provide the seed as a parameter in the SEI, but rather it is an implementation-oriented choice, while AV1 still embeds the seed in the film grain metadata structure.

[0107] Also, clip_to_restricted_range is equivalent to fg_full_range_flag and should be set based on the input video range.

[0108] AV1~VVC In another example, in the mapping stage, if film_grain_params_present and apply_grain are both set to 1, then fg_characteristics_cancel_flag is set to 0. If one of the film_grain_params_present or apply_grain flags is 0, then fg_characteristics_cancel_flag is set to 1, and in such case no other parameters are sent so no further mapping is required.

[0109] There is no seed parameter in the VVC syntax, so grain_seed does not need to be mapped. Depending on the implementation, compositing may still need to use a seed to ensure bit-accurate film grain generation, but that is implementation dependent and is not part of the film grain metadata.

[0110] To map update_grain and film_grain_params_ref_idx to fg_characteristics_persistence_flag, it is useful to look at not only the current frame but also other future frames. If update_grain is set to 1, it means that a new set of film grain parameters is sent. For such frames, a new film grain SEI with appropriately transformed parameters needs to be created. However, some future frames may have update_grain set to 0 and film_grain_params_ref_idx pointing to the current frame. In such cases, fg_characteristics_persistence_flag should be set to 1, which means that the current film grain SEI is applied to the current decoded frame, but also to output frames in output order.

[0111] For example, in the situation given in Fig. 11, it is straightforward to perform the metadata conversion. For the current (first) frame with update_grain equal to 1, the mapped and converted flag fg_characteristics_persistence_flag needs to be set to 1. Then, for all subsequent frames with update_grain equal to 0, there is no need to create an associated film grain SEI. If fg_characteristics_persistence_flag is set to 1 and subsequent frames do not have a film grain SEI embedded in their structure, they will reuse the last known set of parameters, as was the case before the conversion from AV1. A new SEI will be created next time for the next frame with update_grain equal to 1. Then the process is repeated.

[0112] Another variant is shown in Fig. 12. In such a case, the mapping described above with reference to Fig. 11 cannot be applied. The problem is caused by the connections represented by dashed lines. AV1 allows a frame to reference film grain metadata from any previous frame by setting the appropriate film_grain_params_ref_idx. However, in VVC, it is only possible to reuse the latest available SEI. In the case shown in Fig. 12, two frames with dashed connections cannot reuse the indicated metadata because in VVC they are rewritten by a new SEI message. One solution could be to create an SEI "transform time" for each frame and embed the SEI for each frame by transforming the parameters pointed to by film_grain_params_ref_idx. Having an SEI for each frame is even more desirable, since it is possible in VVC and can guarantee bit-accurate playback in trick modes. In such a case, when an SEI needs to be created for each frame, fg_characteristics_persistence_flag is set to 0 for each of those SEI messages.

[0113] In summary, if the situation shown in Fig. 11 occurs, it is possible to set fg_characteristics_persistence_flag to 1 for the current frame and skip the conversion of parameters and creation of SEI messages for any frame with update_grain equal to 0. A new SEI is created in the same way when the next frame occurs with update_grain equal to 1. If the situation shown in Fig. 12 occurs, it is possible to set fg_characteristics_persistence_flag to 0 for the current frame and all other frames and create SEI messages for all frames, even if update_grain is 0. In such a case, fg_characteristics_persistence_flag needs to be set to 0 and other parameters are converted as before. Note that the converted parameters are taken from the frame indicated by film_grain_params_ref_idx. Then the random mode is used and other model specific parameters are set randomly as described above.

[0114] Note that this is just one possible illustration of the given problem: potentially there may be other situations where similar logic may be applied.

[0115] Then, clip_to_restricted_range is equivalent to fg_full_range_flag and should be set based on the input video range. And overlap_flag indicates whether deblocking is performed or not. In VVC SEI, there is no such parameter, so it is left to the implementer to choose whether deblocking of film grain blocks is applied during compositing or not.

[0116] Then in the expansion stage fg_blending_mode_id is set to 0, indicating additive blending. In some embodiments it can be set to 1, indicating multiplicative mode, in which case model-specific parameters need to be adjusted for such blending mode. fg_model_id is set to either 0 or 1 depending on the transformed target model. Since random mode is used, model-specific parameters are set according to the selected model and blending mode in a random manner.

[0117] Variation: Default mode In another variant, called default mode, the process is essentially similar to the random mode, except that the model-specific parameters are defined such that several predefined parameter sets are used. Figure 13 shows an example of a method for converting film grain metadata from a first format to a second format according to a variant of the default mode. In 1300, for the random mode, the general film grain metadata is mapped from the first format to the second format. In 1301, default values ​​for the model-specific parameters are obtained. Several sets (different predefined sets of film grain-specific model parameters) can be used, and an index of the set can be input to the conversion process. Thus, according to this variant, obtaining values ​​in 1301 can include determining a set of specific model parameters and / or selecting a particular set of model parameters from among the predefined sets. In 1302, the selected values ​​are assigned to the specific model parameters of the second format.

[0118] In this way, more flexibility is given to select the most appropriate set of parameters. Compared to the random mode, the default mode can provide a more meaningful transformation, since model-specific parameters can be manually selected and stored for further use. Compared to the random mode, this mode achieves a more controllable and more meaningful transformation, but requires predefined parameters to be stored.

[0119] Different predefined metadata are selected for different models and blending modes, which are selected based on different complexity or memory requirements, and with respect to the final film grain look.

[0120] General film grain metadata is interpreted as in random mode.

[0121] Variation: First Mapping Mode Another variant, named the first mapping mode, is provided below: Note that the names given to all variants for film grain metadata conversion are for illustrative and naming purposes, and the name given to any one of the variants is not limiting and does not induce any inherent characteristics to the variant.

[0122] The first mapping mode variant is the most demanding mode in terms of complexity. However, it aims to result in the best possible transformation in terms of preserving the initial film grain appearance. The process aims to establish an optimal mapping between all film grain syntax elements.

[0123] 14 illustrates an example of a method for converting film grain metadata from a first format to a second format according to a first mapping mode variant. At 1400, general film grain metadata is mapped from a first format to a second format for random and default modes.

[0124] Next, the conversion of the model-specific film grain parameters is performed.

[0125] According to a modification example, in 1401, the first set of film grain patterns is determined from different parameter sets of the first film grain model, and the second set of film grain patterns is determined from different parameter sets of the second film grain model. The purpose is to match the film grain patterns obtained from each film grain model. In 1402, a mapping between the first set of film grain patterns and the second set of film grain patterns is obtained. This can be implemented in various ways.

[0126] According to one example, all possible film grain patterns are calculated offline for different models. In the offline stage, after the film grain patterns are calculated, experts can visually compare different patterns from different models and match them with similar patterns. For example, FIG. 15 shows possible visual matches and relationships with different patterns of two models. Note that the number of different patterns that can be represented by one model is limited. In the left part of FIG. 15, N patterns are mapped to M patterns, and N < M. Therefore, each pattern N can establish a unique relationship with a pattern from another model. On the right side of FIG. 15, M patterns of one model need to establish a relationship with N patterns of another model, and M > N. Therefore, some patterns of one model are mapped to the same pattern of another model. Even if there is no unique mapping, in some cases, this situation cannot be omitted and a compromise has to be made. For example, AV1 supports a much larger number of patterns that can be modeled compared to VVC. Therefore, some AV1 film grain patterns are connected to one VVC pattern.

[0127] Since the model specific parameters for each pattern are known, visual matching between the patterns leads to parameter conversion between the models. Thus, in 1403, a pattern from the film grain model in the second format is selected as the pattern that matches the pattern obtained from the film grain model parameters in the first format. Thanks to the matching, each set of model parameters can be represented by another set of another model by simply replacing the model parameters of the input film grain metadata with the parameters of its visual equivalent from another model. Since a given model is deterministic, the visual matching is performed only once and stored for further use.

[0128] This technique can be used with any of the transformation directions mentioned above.

[0129] In another variant, numerical techniques are proposed to find relations instead of visual matching of patterns. For example, in 1402, visual agreement is replaced by minimization (or maximization) of some objective criterion. To quantify the similarity between film grain patterns, a divergence measure between two empirical probability distributions can be used. One possibility is to use the Jensen-Shannon divergence (J. Lin, "Divergence measures based on the Shannon entropy," IEEE Trans. Inf. Theory, vol. 37, no. 1, pp. 145-151, January 1991), which provides a way to measure the similarity between two distributions. The more similar the patterns are, the smaller the JSD. Thus, for one pattern, a match is associated with the minimum JSD between that pattern and other patterns coming from different models. For a particular pattern i, the associated match j, which represents the most similar pattern coming from another model, is given by the following relation:

[0130]

number

[0131] Instead of the JSD, other divergence measures can be used. As mentioned before, this technique can be used in any of the transformation directions mentioned above.

[0132] Variation: Second Mapping Mode As an alternative, a second mapping mode is proposed to match values ​​of a given number of model-specific film grain parameters by finding an analytical mapping between sets of parameter values ​​that give similar outputs (subject to a minimization criterion) thanks to a neural network.

[0133] In this variant, the goal is to determine the parameters of a neural network. The neural network does not necessarily have to be deep and different types of network architectures can be used. It requires a training phase to determine the parameters of the neural network. However, once the neural network model is trained, it can be used on-fly (at 1403) to convert model-specific film grain parameters.

[0134] In the training phase, a predefined film grain pattern (e.g., determined as in 1401 above) and associated model-specific parameters are used. The input attributes containing the parameters of the film grain model are typically given in the form of a vector. The output attributes containing the parameters of the mapped film grain are also vectors that do not necessarily have the same size as the input vectors. The cost function (i.e., optimization criterion) of the neural network can be any objective metric that quantifies the similarity between two patterns obtained from the parameters of the film grain model input to the neural network and the parameters of the film grain model obtained from the output of the neural network.

[0135] Once the neural network is trained, parameters of the film grain model for the second set of film grain metadata are obtained as an output of the neural network fed with the parameters of the film grain model for the first set of film grain metadata.

[0136] By adopting one of the variants provided above, the bitstream delivered after decoding can produce an output picture with film grain. If mapping mode is performed, the same visual appearance of film grain is observed.

[0137] 16 illustrates a block diagram of a system in which aspects of the present embodiment may be implemented, according to another embodiment. FIG 16 illustrates an embodiment of an apparatus 1600 for converting a first set of film grain metadata in a first format to a second set of metadata in a second format, according to any one of the embodiments described herein.

[0138] The device comprises a processor 1610 and may be interconnected through at least one port to a memory 1620. Both the processor 1610 and the memory 1620 may also have one or more additional interconnects to external connections.

[0139] The processor 1610 is also configured to receive a first set of film grain metadata in a first format, convert the first set of film grain metadata in the first format to a second set of metadata in a second format, and transmit the second set of metadata in the second format using any one of the embodiments described herein. For example, the processor 1610 is configured using a computer program product including code instructions implementing any one of the embodiments described herein.

[0140] In one embodiment shown in Fig. 17, in a transmission context between two remote devices A and B over a communication network NET, device A comprises a processor associated with memory RAM and ROM configured to implement a method for encoding video as described in Figs. 1 and 2, and device B comprises a processor associated with memory RAM and ROM configured to implement a method for decoding video as described in relation to Figs. 1 and 3. Depending on the embodiment, device A and / or B are also configured to convert film grain metadata transmitted together with the coded video in either a first format or a second format as described in relation to Figs. 4-15. In some embodiments, device B is also configured to use the converted film grain metadata to composite film grain onto the received video.

[0141] According to one embodiment, the network is a broadcast network adapted to broadcast / transmit the encoded video and the converted film grain metadata from device A to decoding devices including device B.

[0142] 18 shows an example of the syntax of a signal transmitted over a packet-based transmission protocol. Each transmission packet P includes a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may include coded video data and converted film grain metadata encoded according to any one of the embodiments described above.

[0143] Various implementations involve decoding. As used herein, "decoding" can encompass all or some of the processes performed on a received encoded sequence to provide a final output suitable for, e.g., a display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as, e.g., entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by decoders in various implementations described herein, such as, e.g., decoding resampling filter coefficients, resampling a decoded picture, etc.

[0144] As a further example, in one embodiment, "decoding" refers to entropy decoding only, in another embodiment, "decoding" refers to differential decoding only, in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding, and in another embodiment, "decoding" refers to the entire reconstruction picture process including entropy decoding. Whether the phrase "decoding process" is intended to refer to a task subset specifically or to the broader decoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0145] Various implementations involve encoding. Similar to the above discussion regarding "decoding," "encoding" as used herein can encompass all or part of the processes performed on an input video sequence to produce, for example, an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as, for example, partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by encoders in various implementations, such as those described herein, such as, for example, determining resampling filter coefficients, resampling decoded pictures, etc.

[0146] As a further example, in one embodiment, "encoding" refers to entropy encoding only, in another embodiment, "encoding" refers to differential encoding only, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer to a task subset specifically or to the broader encoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0147] It should be noted that syntax elements as used herein are descriptive terms, and therefore do not exclude the use of other syntax element names.

[0148] This disclosure has described various information, such as, for example, syntax, that may be transmitted or stored. This information may be packaged or arranged in various manners, including, for example, common to video standards, such as placing the information in an SPS, PPS, NAL unit, header (e.g., a NAL unit header or slice header), or SEI message. Other manners are also available, including, for example, common to system-level or application-level standards, such as placing the information in one or more of the following: a. SDP (session description protocol), a format for describing multimedia communication sessions for purposes of session announcement and session invitation, e.g., as described in the RFCs and used in conjunction with RTP (Real-time Transport Protocol) transport. b. DASH Media Presentation Description (MPD) descriptors, for example as described in DASH and transmitted over HTTP, which are associated with a representation or a collection of representations to provide additional characteristics to the content representations. c. RTP header extensions, for example as used during RTP streaming. d. The ISO Base Media File Format, such as that used in OMAF and in some specifications, which uses boxes, object-oriented building blocks defined by a unique type identifier and a length, also known as "atoms". e. HTTP Live Streaming (HLS) manifests transmitted over HTTP. A manifest can be associated with a version or collection of versions of content, for example to provide characteristics of the version or collection of versions.

[0149] Where a figure is presented as a flow diagram, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow diagram of the corresponding method / process.

[0150] Some embodiments refer to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, often due to computational complexity constraints. Rate-distortion optimization is usually formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, but with a full evaluation of their encoding costs and associated distortion of the reconstructed signal after encoding and decoding. To keep the encoding complexity down, faster approaches can also be used, especially with the calculation of approximate distortion based on the prediction or prediction residual signal rather than the reconstructed signal. A mixture of these two approaches can also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the encoding cost and the associated distortion.

[0151] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single implementation form (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus may be implemented in appropriate hardware, software, and firmware. The method may be implemented in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0152] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, means that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "an embodiment" or "in one implementation" or "in an implementation," as well as other variations thereof, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0153] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0154] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0155] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves in some manner, for example, an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0156] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This may be expanded as many times as the number of items listed, as would be apparent to one of ordinary skill in the art and related arts.

[0157] Also, as used herein, the term "signaling" specifically means to indicate something to a corresponding decoder. For example, in a particular embodiment, an encoder signals a particular one of a plurality of resampling filter coefficients. Thus, in an embodiment, the same parameters are used at both the encoder and decoder sides. Thus, for example, an encoder can transmit a particular parameter to a decoder (explicit signaling) so that the decoder can use the same particular parameter. In contrast, if the decoder already has the particular parameter as well as other parameters, a non-transmitting signaling (implicit signaling) can be used to simply allow the decoder to know and select the particular parameter. By avoiding transmitting any actual function, bit savings are realized in various embodiments. It will be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. The above relates to the verb form of the word "signal", which may also be used as a noun in this specification.

[0158] As will be apparent to one skilled in the art, implementations can result in a variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. For example, such a signal can be formatted as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries can be, for example, analog information or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored in a processor-readable medium.

[0159] A number of embodiments have been described above, and the features of these embodiments may be provided alone or in any combination across the various claim categories and types.

Claims

1. 1. A method comprising: - obtaining first film grain metadata, said first film grain metadata conforming to a first film grain syntax and representing a first film grain model; - converting at least a portion of the first film grain metadata into at least a portion of second film grain metadata, the second film grain metadata conforming to a second film grain syntax and representing a second film grain model; - the second film grain syntax is different from the first film grain syntax, or the second film grain model is different from the first film grain model, or both the second film grain syntax and the second film grain model are different from the first film grain syntax and the first film grain model.

2. 1. An apparatus comprising one or more processors, the one or more processors comprising: - obtaining first film grain metadata, said first film grain metadata conforming to a first film grain syntax and representing a first film grain model; - translating at least a portion of said first film grain metadata into at least a portion of second film grain metadata, said second set of film grain metadata being operative to conform to a second film grain syntax and to represent a second film grain model; - an apparatus, wherein the second film grain syntax is different from the first film grain syntax, or the second film grain model is different from the first film grain model, or both the second film grain syntax and the second film grain model are different from the first film grain syntax and the first film grain model.

3. The method of claim 1 or the apparatus of claim 2, wherein at least one of the first film grain syntax or the second film grain syntax conforms to a video compression standard.

4. Transforming at least a portion of the first film grain metadata into at least a portion of second film grain metadata includes: - a discard mode in which film grain compositing is disabled; a random mode in which syntax elements representing parameters of the film grain model are randomly assigned; a default mode in which syntax elements representing parameters of a film grain model are assigned based on a set of parameters of the film grain model selected from among a plurality of sets of parameters for said film grain model; a first mapping mode in which syntax elements representing parameters of a film grain model are assigned based on matching of film grain patterns obtained for said first film grain model and said second film grain model, respectively; - a second mapping mode, the syntax elements representing parameters of the film grain model being assigned based on the parameters of the film grain model output by the neural network.

5. 5. The method of claim 1, 3 or 4, or the apparatus of any one of claims 2 to 4, wherein converting at least a portion of the first film grain metadata into at least a portion of the second film grain metadata comprises assigning a given value to a given syntax element of the second film grain metadata that disables film grain synthesis.

6. 5. The method of claim 1, 3 or 4 or the apparatus of any one of claims 2 to 4, wherein converting at least a portion of the first film grain metadata into at least a portion of the second film grain metadata comprises mapping at least one syntax element of the first film grain metadata to at least one other syntax element of the second film grain metadata.

7. The method or apparatus of claim 6 , wherein the at least one syntax element and the at least one other syntax element relate to general film grain metadata that is not a film grain model parameter.

8. 8. The method or apparatus of claim 6 or 7, wherein converting at least a portion of the first film grain metadata into at least a portion of the second film grain metadata further comprises assigning random values ​​to syntax elements of the second film grain metadata associated with film grain model parameters.

9. Transforming at least a portion of the first film grain metadata into at least a portion of second film grain metadata includes: - obtaining values ​​of film grain model parameters; The method or apparatus of claim 6 or 7, further comprising: - assigning said obtained value to a syntax element of said second film grain metadata related to a film grain model parameter.

10. The method or apparatus of claim 9 , wherein obtaining values ​​for the film grain parameters comprises selecting a set of values ​​for the film grain model parameters from among a plurality of sets of values ​​for the film grain model parameters.

11. Transforming at least a portion of the first film grain metadata into at least a portion of second film grain metadata includes: - identifying a film grain pattern within a set of film grain patterns obtained from different parameters of the second film grain model that matches a film grain pattern obtained from parameters of the first film grain model of the first film grain metadata; - assigning film grain model parameters that provided the identified film grain pattern to a syntax element representing the second film grain model of the second film grain metadata.

12. Transforming at least a portion of the first film grain metadata into at least a portion of second film grain metadata includes: - determining a first set of film grain patterns from different parameter sets of the first film grain model and a second set of film grain patterns from different parameter sets of the second film grain model, respectively; The method or apparatus of claim 11, further comprising: obtaining a mapping between said first set of film grain patterns and said second set of film grain patterns.

13. 13. The method or apparatus of claim 12, wherein the mapping between the first set of film grain patterns and the second set of film grain patterns is obtained using a similarity metric determined between the respective first and second sets of film grain patterns.

14. To get the value of the film grain parameter, - providing said film grain model parameters of said first film grain metadata as input to a neural network; - obtaining film grain model parameters of the second film grain metadata as an output of the neural network.

15. the one or more processors: The method of claim 1 or any one of claims 3 to 14, further comprising transmitting the converted second film grain metadata together with the coded video, or an apparatus of any one of claims 2 to 13 further configured for transmitting the same.

16. A computer readable storage medium storing instructions for causing one or more processors to perform the method of claim 1 or any one of claims 3-15.

17. 16. A computer program product comprising instructions, which when executed by one or more processors, cause the one or more processors to perform a method according to any one of claims 1 or 3 to 15.

18. A device, comprising: - a device according to any one of claims 2 to 15, A device comprising at least one of: (i) an antenna configured to receive a signal including data representing a video and at least one of the first set or the second set of film grain metadata; (ii) a band limiter configured to limit the received signal to a frequency band including the data representing a video; or (iii) a display configured to display the decoded video having film grain synthesized using the second set of film grain metadata. －

19. 20. The device of claim 18, comprising a TV, a mobile phone, a tablet or a set-top box.

Images