Method and apparatus for converting film particle metadata

By converting film particle metadata from one format to another in the video compression system, the problem of film particle noise transmission and reconstruction in the video system is solved, and the controllability of film particles is achieved, improving the controllability of film particles Video quality and artistic expression.

CN119094797APending Publication Date: 2024-12-06INTERDIGITAL CE PATENT HOLDINGS SAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411315162.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2021-12-17
Filing Date
2022-12-08
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

It is difficult for existing video compression systems to effectively preserve and reconstruct film particle noise during transmission and distribution, resulting in a lack of vividness and artistic expression in digital video, and it is difficult to reconstruct film particles at high compression rates after removing them.

Method used

Provide a method and device to ensure efficient transmission and replication of film particle information in the video encoding and decoding chain by converting film particle metadata from one format to another, adapting to different video codecs and models. The structure, including the conversion between AV1 and VVC standards, supports autoregression and frequency filtering models.

Benefits of technology

The controllability of film particles is achieved during video compression and distribution, ensuring the maintenance of video quality and artistic intentions, improving the vividness and visual effects of digital videos, while reducing bandwidth consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094797A_ABST
    Figure CN119094797A_ABST
Patent Text Reader

Abstract

A method and apparatus for converting film particle metadata is provided. In an embodiment, first film particle metadata is obtained, the first film particle metadata conforming to a first film particle syntax and representing a first film particle model. Converting at least a portion of the first film particle metadata into at least a portion of second film particle metadata, the second film particle metadata conforming to a second film particle syntax and representing a second film particle model, where the second film particle syntax is different from the first film particle syntax, or the second film particle model is different from the first film particle model, or both the second film particle grammar and the second film particle model are different from the first film particle grammar and the first film particle model.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of Chinese patent application No. 202280083420.3, filed on December 8, 2022, and entitled “Method and Apparatus for Converting Film Grain Metadata”, the contents of which are incorporated herein by reference. Technical Field

[0002] Embodiments of the present invention generally relate to video compression, distribution and rendering, and more particularly to film grain modeling. Embodiments of the present invention relate to methods and apparatus for film grain metadata conversion between codecs and / or between models. Background Art

[0003] In order to achieve high compression efficiency, image and video decoding schemes usually use prediction and transformation to fully exploit the spatial and temporal redundancy in the video content. In general, intra-frame or inter-frame prediction is used to exploit the correlation within or between pictures, and then the difference between the original block and the predicted block (usually expressed as a prediction error or prediction residual) is transformed, quantized, and entropy decoded. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy decoding, quantization, transformation, and prediction. Summary of the invention

[0004] According to one aspect, a method for converting film grain metadata is provided. The method includes: obtaining first film grain metadata, the first film grain metadata conforming to a first film grain syntax and representing a first film grain model; converting at least a portion of the first film grain metadata into at least a portion of second film grain metadata, the second film grain metadata conforming to a second film grain syntax and representing a second film grain model, wherein the second film grain syntax is different from the first film grain syntax, or the second film grain model is different from the first film grain model, or both the second film grain syntax and the second film grain model are different from the first film grain syntax and the first film grain model.

[0005] According to another aspect, an apparatus for converting film grain metadata is provided. The apparatus includes one or more processors, wherein the one or more processors are configured to: obtain first film grain metadata, the first film grain metadata conforming to a first film grain syntax and representing a first film grain model; convert at least a portion of the first film grain metadata into at least a portion of second film grain metadata, the second film grain metadata conforming to a second film grain syntax and representing a second film grain model, wherein the second film grain syntax is different from the first film grain syntax, or the second film grain model is different from the first film grain model, or both the second film grain syntax and the second film grain model are different from the first film grain syntax and the first film grain model.

[0006] One or more embodiments also provide a computer program, the computer program comprising instructions, which when executed by one or more processors, cause the one or more processors to perform the method for converting film grain metadata according to any of the embodiments described herein. One or more embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium having stored thereon instructions for converting film grain metadata according to the method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A block diagram is illustrated of a system in which aspects of embodiments of the present invention may be implemented.

[0008] Figure 2 A block diagram of an implementation of a video encoder is illustrated.

[0009] Figure 3 A block diagram illustrating an implementation of a video decoder is illustrated.

[0010] Figure 4 An example of a film grain model within a video coding standard is illustrated.

[0011] Figure 5 Illustrate possible transformation directions for film grain metadata.

[0012] Figure 6 Possible conversion directions for cross-codec and cross-model conversion are illustrated.

[0013] Figure 7 A block diagram illustrating an example of the use of film grain in a video coding framework.

[0014] Figure 8 A block diagram illustrating an example of a method for converting film grain metadata from a first format to a second format according to an embodiment.

[0015] Fig. 9 Examples of possible locations of film grain conversion in the encoding / decoding chain are illustrated.

[0016] Fig.10 A block diagram illustrating an example of a method for converting film grain metadata from a first format to a second format according to another embodiment.

[0017] Fig.11 An example of the use of update_grain and film_grain_params_ref_idx syntax elements for AV1 film grain metadata is illustrated.

[0018] Fig.12 Another example of the use of update_grain and film_grain_params_ref_idx syntax elements for AV1 film grain metadata is illustrated.

[0019] Fig.13 A block diagram illustrating an example of a method for converting film grain metadata from a first format to a second format according to another embodiment.

[0020] Fig.14 A block diagram illustrating an example of a method for converting film grain metadata from a first format to a second format according to another embodiment.

[0021] Fig.15 An example of matching between film grain patterns obtained from parameter sets of different film grain models is illustrated.

[0022] Fig.16 A block diagram of a system in which aspects of embodiments of the present invention may be implemented is illustrated according to another embodiment.

[0023] Fig.17 Two remote devices communicating over a communication network are shown in accordance with an example of the principles of the present invention.

[0024] Fig.18 The syntax of an example signal according to the principles of the present invention is shown. DETAILED DESCRIPTION

[0025] This application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and at least individual characteristics are shown, usually in a manner that may sound limited. However, this is for the purpose of describing clearly and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with the aspects described in the previous submission.

[0026] The aspects described and contemplated in this application can be implemented in many different forms. Figure 1 , Figure 2 and Figure 3 Some embodiments are provided, but other embodiments are contemplated, and Figure 1 , Figure 2 and Figure 3The discussion of the present invention does not limit the breadth of specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to sending a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the methods.

[0027] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably.

[0028] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless the correct operation method requires a specific order of steps or actions, the order and / or purposes of specific steps and / or actions can be modified or combined. In addition, terms such as "first", "second" etc. can be used in various embodiments to modify elements, parts, steps, operations, etc., such as "first decoding" and "second decoding". Unless specifically required, the use of such terms does not imply the ordering of the modification operation. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can, for example, occur before, during, or in overlapping time periods of the second decoding.

[0029] Various methods and other aspects described in this application can be used to modify modules, such as Figure 2 The forward transform module (225), the inverse transform module (250) and the entropy decoding (245) of the video encoder 200 shown in FIG. Figure 3 300 and the entropy decoding (330). Furthermore, aspects of the present invention are not limited to VVC or HEVC and may be applied, for example, to other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application may be used alone or in combination.

[0030] Figure 1Illustrated is a block diagram of an example of a system in which various aspects and embodiments can be implemented. System 100 may be embodied as a device, which includes various components described below, and is configured to perform one or more aspects of various aspects described in the present application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptops, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 100 may be embodied in a single integrated circuit, multiple ICs and / or discrete components individually or in combination. For example, in at least one embodiment, the processing of system 100 and encoder / decoder elements are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 100 is configured to implement one or more aspects of the aspects described in the present application.

[0031] The system 100 includes at least one processor 110, which is configured to execute instructions loaded therein to implement various aspects described in the present application, for example. The processor 110 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a disk drive, and / or an optical drive. As a non-limiting example, the storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device.

[0032] The system 100 includes an encoder / decoder module 130, which is configured to process data, for example, to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module that can be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. In addition, the encoder / decoder module 130 may be implemented as an independent element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software known to those skilled in the art.

[0033] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described in this application may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of the various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operation logic.

[0034] In some embodiments, memory inside the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory outside the processing device (e.g., the processing device can be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory can be a memory 120 and / or a storage device 140, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory (such as RAM) is used as working memory for video coding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Experts Group (JVET)).

[0035] Inputs to the elements of system 100 may be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal, such as that transmitted over the air by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples ( Figure 1 ) includes composite video.

[0036] In various embodiments, the input device of block 105 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) again band-limiting the frequency band to a narrower frequency band to select a signal band that may be referred to as a channel in certain embodiments; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal that sends by wired (for example, cable) medium, and by filtering, down-conversion and filtering to the frequency band of expectation again to perform frequency selection.Various embodiments rearrange the order of (and other) element described above, remove some elements in these elements, and / or add other elements that perform similar or different functions.Adding element can be included in and inserts element between existing element, for example, inserts amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0037] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 100 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as desired, for example, within a separate input processing IC or within the processor 110. Similarly, various aspects of USB or HDMI interface processing may be implemented as desired within a separate interface IC or within the processor 110. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including (e.g.) the processor 110 and the encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on an output device.

[0038] The various components of system 100 may be disposed within an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data between using a suitable connection arrangement 115 (eg, an internal bus known in the art, including an I2C bus, wiring, and printed circuit boards).

[0039] The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented within a wired and / or wireless medium, etc.

[0040] In various embodiments, a Wi-Fi network (such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream data to the system 100. The Wi-Fi signals of these embodiments are received by a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 in these embodiments is usually connected to an access point or a router, which provides access to an external network including the Internet to allow streaming applications and other OTT communications. Other embodiments use a set-top box to provide streaming data to the system 100, which is connected to the HDMI of the input box 105 to pass data. There are also other embodiments that use the RF connection of the input box 105 to provide streaming data to the system 100. As described above, various embodiments provide data in a non-streaming mode. Additionally, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0041] The system 100 can provide output signals to various output devices, including a display 165, a speaker 175, and other peripherals 185. The display 165 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 165 can be used for a television, a tablet computer, a laptop, a cellular phone (mobile phone), or other devices. The display 165 can also be integrated with other components (e.g., as in a smart phone), or it can be independent (e.g., an external monitor for a laptop). In various examples of the embodiments, other peripherals 185 include one or more of an independent digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripherals 185 that provide functions based on the output of the system 100. For example, a disc player performs the function of playing the output of the system 100.

[0042] In various embodiments, control signals are communicated between the system 100 and the display 165, speaker 175, or other peripherals 185 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to the system 100 via the communication interface 150 using a communication channel 190. The display 165 and speaker 175 may be integrated into a single unit with other components of the system 100 in an electronic device (e.g., a television). In various embodiments, the display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0043] Alternatively, for example, if the RF portion of input 105 is part of a separate set-top box, the display 165 and speaker 175 can be separated from one or more of the other components. In various embodiments in which the display 165 and speaker 175 are external components, the output signal can be provided via a dedicated output connection (including, for example, an HDMI port, a USB port, or a COMP output).

[0044] These embodiments may be implemented by computer software implemented by processor 110, or by hardware, or by a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. As a non-limiting example, memory 120 may be of any type suitable for the technical environment, and may be implemented using any appropriate data storage technology (such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory). As a non-limiting example, processor 110 may be of any type suitable for the technical environment, and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0045] Figure 2 An encoder 200 is illustrated. Variations of the encoder 200 are contemplated, but for clarity, the encoder 200 is described below without describing all contemplated variations.

[0046] In some embodiments, Figure 2 An encoder in which the HEVC standard is improved or an encoder adopting a technique similar to HEVC, such as a VVC (Versatile Video Coding) encoder developed by JVET (Joint Video Exploration Team) is also exemplified.

[0047] Before being encoded, the video sequence may undergo a pre-encoding process 201, for example, applying a color transform to an input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of color components), or resizing the picture (e.g., downscaling). Metadata may be associated with the pre-processing and attached to the bitstream.

[0048] In encoder 200, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (202) and processed in units such as CUs. For example, each unit is encoded using intra mode or inter mode. When a unit is encoded in intra mode, the unit performs intra prediction (260). Motion estimation (275) and compensation (270) are performed in inter mode. The encoder decides (205) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. The encoder can also mix (263) intra prediction results and inter prediction results, or mix results from different intra / inter prediction methods. The prediction residual is calculated, for example, by subtracting (210) the prediction block from the original image block.

[0049] The motion correction module (272) uses the already available reference pictures in order to correct the motion field of the block without referring to the original block. The motion field of a region can be considered as the set of motion vectors for all pixels in the region. If the motion vectors are based on sub-blocks, the motion field can also be represented as the set of motion vectors for all sub-blocks in the region (all pixels within a sub-block have the same motion vector, and the motion vectors may be different from sub-block to sub-block). If a single motion vector is used for the region, the motion field of the region can also be represented by a single motion vector (the same motion vector for all pixels in the region).

[0050] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., directly code the residual without applying the transform or quantization process.

[0051] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The image block is reconstructed by combining the decoded prediction residual and the prediction block (255). An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored at a reference picture buffer (280).

[0052] Figure 3 A block diagram of a video decoder 300 is illustrated. In the decoder 300, the bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs the same Figure 2 The decoding process is the reverse of the encoding process described in . The encoder 200 also typically performs video decoding as part of encoding the video data.

[0053] Specifically, the input of the decoder includes a video bitstream, which may be generated by the video encoder 200. First, the bitstream is entropy decoded (330) to obtain transform coefficients, motion vectors, and other decoding information. The picture partition information indicates how the picture is partitioned. Therefore, the decoder can divide (335) the picture according to the decoded picture partition information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The image block is reconstructed by combining the decoded prediction residual and the prediction block (355).

[0054] The prediction block may be obtained (370) from intra prediction (360) or motion compensated prediction (i.e., inter prediction) (375). The decoder may blend (373) intra prediction results and inter prediction results, or blend results from multiple intra / inter prediction methods. Prior to motion compensation, the motion field may be corrected (372) by using an already available reference picture. An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored at a reference picture buffer (380).

[0055] The decoded picture may also undergo post-decoding processing 385, such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4), or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process 201, or resizing (e.g., upscaling) the reconstructed picture. The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0056] Film grain is a special type of noise that appears in videos and is very pleasant and very unique. Essentially, film grain is the result of a physical process that simulates the exposure and development of film. When exposed to enough light, the crystals of silver halide, a light-sensitive compound that is part of the film, transform into tiny specks of metallic silver. The exposure, controlled by the camera lens, triggers a chemical process in the silver halide crystals that is proportional to the amount of light absorbed by each crystal. Since not all crystal specks are exposed equally, and since they have different sizes and shapes and are randomly distributed within the film emulsion, some of them develop into metallic silver and some do not. Even, within an evenly exposed area, some crystals develop while others do not. The underexposed and undeveloped crystals undergo chemical removal during film development, leaving tiny gaps between the already reacted crystals. During negative printing, the empty gaps allow light to pass through, which creates the visual perception of film grain. Additionally, the developed crystals block light from passing through, and the more reactions that occur in the first place, the more light is blocked by the developed specks, resulting in different intensity levels in the final tray. Ultimately, these atomic changes are seen as larger phenomena that, when all are combined, are perceived as an image with a grainy texture.

[0057] Furthermore, the formation of film grain is random due to the (quasi) random formation of metallic silver. More precisely, the process is quasi-random because the physical properties of photographic film are controlled by design and during the manufacturing process. For example, silver crystals are not perfectly and evenly distributed in a grid pattern, and there are many different sizes of crystals in the film. The larger the crystals, the more sensitive they are to light. The more sensitive the film is to light, the larger and more visible the grains appear. Furthermore, the size of the silver crystals in the original emulsion, the physical particle size of the crystals, the amount of exposure variation that can be tolerated by a given film, the development time, and even the development temperature are some of the parameters that affect the visibility and subjective perception of film grain.

[0058] Technically speaking, film is a product of the past and is primitive from today's perspective. The digital age has opened up a wealth of new possibilities for creating and consuming content. However, in the digital age, film grain no longer exists during content creation. A noise-free and clear video can be too perfect, contrary to expectations, resulting in a lower subjective experience for the viewer. Therefore, in order to increase the vividness of a digitally captured scene and / or to better facilitate the expression of creative intent in the digital world, film grain is often added in post-processing.

[0059] It is now widely known that medium-grained textures present desirable characteristics. Digital artists choose film grain patterns that match the look and feel they want. However, it is challenging to preserve film grain throughout the video processing and delivery chain. Visually lossless encoding that does not change and destroy film grain is only possible at very high bit rates. It is only feasible in high-quality applications (e.g., cinematic movie quality). For ordinary video applications, it will consume a lot of bandwidth. Therefore, stronger lossy compression is an inevitable step within current content distribution networks. It should be noted that before delivery, the video is usually also subjected to various preprocessing steps, in which noise filtering is presented to improve decoding efficiency. Therefore, in various filtering and lossy compression steps, film grains are suppressed and cannot be reconstructed.

[0060] To address this issue, modern video compression and distribution systems may provide a mechanism to remove film grain before and / or during compression and automatically add it back in a controllable manner on the user side by using parametric models. Figure 7 A block diagram illustrating an example of film grain usage in a video coding framework. Typically, an encoder removes original film grain from a video (700) and tunes film grain model parameters (automatically or manually in 701), and a decoder replaces it with a synthetic substitute that is visually close to the removed one (704). In this way, information about film grain is conveyed as additional metadata.

[0061] Automatic analysis of the original film grain (film grain analysis consisting of preprocessing 700 for filtering the input video for encoding and film grain FG estimation 701) is an optional step. The model parameters can be estimated using an algorithmic / automatic method, or they can be manually adjusted by an expert. In many cases, a person skilled in the art provides a manually selected set of film grain parameters (FG parameters) (702) to an encoder, which encodes the filtered video into a bitstream. However, in some cases, the automatic estimation of the parameters can be used directly to automate the process, or it can be used as an initialization step before manual tuning. In this way, the goal is to add synthetic grain back to the video, keeping in mind that the artistic intent remains unchanged. The FG parameters are then communicated to the decoder (703) as additional metadata (e.g., as a Film Grain Characteristic (FGC) SEI message). The decoder (703) decodes the bitstream and outputs a decoded video and decoded FG parameters. The FG synthesis (704) then adds the film grain pattern to the decoded video by using the FG parameters to provide a decoded video with film grain as output.

[0062] Film grain is also gaining more attention in the compression community, where another more technical use of film grain is presented. It can be synthesized for video compression purposes to mask compression artifacts. In this scenario, film grain is removed by the compression itself, while other compression artifacts appear at the same time when stronger compression is used. Thereafter, film grain can be added to the compressed video to mask the compression artifacts, even though it was not present in the original content. Adding noise on the video during playback masks the compression, making the video look more aesthetically pleasing without the bandwidth cost. Note that in this scenario, it is not very important that the film grain matches the removed film grain if the removed film grain was present in the source video. If the removed film grain was present in the source video, it is not important that the film grain matches the removed film grain. The important aspect is that the video is re-noised by adding film grain to mask flat monotonous areas and other compression artifacts caused by compression. Today's activity within the video compression community is more towards compression masking use cases, rather than keeping the same look of film grain for artistic purposes.

[0063] Currently, the latest and most advanced video coding standards support film grain metadata. For example, AV1 (AV1 Bitstream and Decoding Process Specification, Alliance for Open Media, Version 1.0.0 with Errata 1, January 18, 2019) and Versatile Video Coding (VVC, Versatile Video Coding, Recommendation ITU-T H.266 (08 / 2020) | International Standard ISO / IEC 23090-3) are two different coding standards that enable the delivery of film grain metadata. HEVC (High Efficiency Video Coding, Recommendation ITU-T H.265 (08 / 2021)) and H.264 / AVC (Advanced Video Coding for Generic Audiovisual Services, Recommendation ITU-T H.264 (08 / 2021)) also provide means to transmit film grain model parameters to the decoder in a similar manner to VVC (these standards share the same film grain metadata).

[0064] Different approaches for film grain modeling have been investigated. In the case of VVC, to restore film grain in compressed video, film grain is parameterized and resynthesized using a frequency filtering solution. VVC also supports autoregressive models, the same as AV1. Even though both VVC and AV1 support autoregressive methods, the models are different.

[0065] AV1 and VVC define film grain metadata in their specifications. VVC does this via the Versatile Supplementary Enhancement Information message (VSEI) for decoded video bitstream specification (Versatile Supplementary Enhancement Information message for decoded video bitstream, Recommendation ITU-T H.274 (08 / 2020) | International Standard ISO / IEC 23002-7), while AV1 integrates the film grain syntax into the frame header and the film grain metadata is part of the specification itself. Therefore, support for film grain is specified in significantly different ways in the two standards. Different syntax and semantics are used depending on the codec and / or model. Since different models can be used, such as Figure 4 As illustrated, the compositing process and film grain blending are also defined differently.

[0066] Some embodiments of the present disclosure provide means for converting film grain model parameters and associated syntax from one format to another suitable format having a standardized syntax.

[0067] Some implementations described herein convert metadata or a set of syntax elements into a modified set of syntax elements that better meets the needs of an end user (eg, a decoder).

[0068] Figure 5 and Figure 6 An example of the direction of the transition is provided. There are several possible transitions:

[0069] 1. Convert AV1 film grain metadata to VVC film grain metadata for autoregressive models and vice versa.

[0070] 2. Convert AV1 film grain metadata to VVC film grain metadata for frequency filter model and vice versa.

[0071] 3. Convert VVC film grain metadata for the autoregressive model to VVC film grain metadata for the frequency filter model, and vice versa. Similar situations can be applied to HEVC and H.264 / AVC instead of VVC.

[0072] 4. Convert frequency filter models to autoregressive models and vice versa, where any combination of H.264 / AVC, HEVC and VVC is applied.

[0073] According to an embodiment, a framework for converting film grain syntax and film grain model parameters from one format to another is provided. The conversion process is not a simple copy process because different syntax elements and semantics are used, including different models. In addition, the conversion may not necessarily occur between codecs. For example, VVC film grain metadata supports two different film grain models, and in some cases, it is useful to implement conversion between models of the same codec. It should be noted that other video coding standards such as High Efficiency Video Coding (HEVC) and H.264 / AVC (for example) also define similar film grain metadata as VVC. Therefore, the conversion framework provided herein is applied to them similarly to VVC. In addition, if model conversion is required, film grain metadata conversion between VVC, HEVC or H.264 / AVC can be defined. Figure 6 Such a scenario is illustrated in .

[0074] On the other hand, if film grain metadata needs to be converted between VVC, HEVC, or H.264 / AVC, but no model conversion is requested, a simple copy can be performed. In fact, if no model conversion is performed, film grain metadata can be shared between VVC, HEVC, or H.264 / AVC because the format of the metadata is the same.

[0075] Furthermore, after converting the metadata including the model parameters, the newly synthesized film grain pattern should (at least visually) match the original film grain pattern. However, in some embodiments, this requirement can be relaxed. It is to be noted that the transcoding process does not introduce new syntax. The idea is to generate metadata that will be embedded directly into the bitstream after conversion.

[0076] In the following, for simplicity, the framework provided in this article is described in the context of the AV1 and VVC standards. However, the general principles described in this article also apply to other video coding standards, such as HEVC and H.264 / AVC. Note that HEVC and H.264 / AVC define the same film grain metadata as VVC. Therefore, there is no need to define the Figure 6 More precisely, all transformation directions from Figure 6All conversion directions can be explicitly defined by defining only conversions between VVC models. Other conversion directions can reuse conversions defined between VVC models. For example, converting the HEVC frequency filter model to the VVC autoregressive model variant-2 is the same as converting the VVC frequency filter model to the VVC autoregressive model variant-2. Or the conversion between the H.264 / AVC autoregressive model variant-2 to the HEVC frequency filter model is the same as the conversion between the VVC autoregressive model variant-2 to the VVC frequency filter model. Therefore, in the following, all definitions and examples applied to VVC also apply to H.264 / AVC and HEVC.

[0077] An object of the present disclosure is to provide a framework for converting film grain metadata and related film grain model parameters into a suitable format (eg, a format supported by a video decoder).

[0078] like Fig. 9 As illustrated, the conversion (FG conversion) can be implemented at different positions of the video encoding / decoding chain depending on the use case:

[0079] 1. The video encoder (901) provides film grain metadata in format 1. The video decoder (903) supports metadata format 2. In this example, the decoder does not understand format 1, and a conversion from format 1 to format 2 may be performed before the encoder embeds the film grain metadata into the bitstream. This may be done at the encoder (901) (or immediately thereafter, as in step 902), but may also be done before the encoder (e.g., in a pre-processing step (900)).

[0080] ) to perform the conversion.

[0081] 2. In another example, the decoder (903) supports format 2, but the metadata from the bitstream is in format 1. Therefore, a process is required at the decoder in order to produce a proper output (decoded video) with film grain. At the stage between the encoder and decoder (902), the conversion can be performed before the decoder parses the bitstream.

[0082] 3. In another example, the decoder (903) supports Format 1 from the bitstream. The decoder can extract film grain metadata from the bitstream. However, the specific implementation of film grain synthesis does not support film grain modeling using Format 1. The conversion of film grain metadata can occur in a post-processing stage (904 or 905) to perform the synthesis run and have it produce film grain output with a target quality. For example, this use case may occur in VVC if the FGC SEI is transmitted to the decoder by incorporating a frequency filtering model. The decoder is able to parse the syntax, but at the decoder (or post-processing)

[0083] The synthesis is achieved by using an autoregressive model.

[0084] 4. In another example, film grain metadata in format 1 is provided to the encoder and format 1 is embedded into the bitstream during encoding. However, during transcoding of the bitstream, some decoders may require format 2. In this case, conversion of the film grain metadata may occur after encoding and before decoding (e.g., during transcoding of the bitstream). In this case, film grain metadata conversion is performed at step 902.

[0085] An example of such a use case could be the transcoding of an AV1 bitstream to a VVC bitstream. In this case, if film grain metadata is present in the AV1 bitstream, it should be converted rather than discarded.

[0086] In all cases, film grain metadata conversion should be done in such a way that the converted composited film grain visually matches (or is very close to) the appearance of the original film grain.

[0087] In some embodiments, this requirement may be relaxed.

[0088] Some other requirements may also be imposed in terms of complexity or memory usage. Note that the transcoding process does not introduce new syntax. The purpose of embodiments of the present invention is to generate metadata that will be embedded directly into the bitstream after transcoding.

[0089] Some embodiments relate to converting metadata from one film grain metadata format to another film grain metadata format, where the formats differ from a video coding standard and / or a film grain model used to synthesize a film grain pattern.

[0090] The principles described herein are based on examples from currently available standards and models supported therein. However, the principles of the present invention may also be applied to other emerging standards and new models that may be available in the future.

[0091] As discussed above, the conversion process described herein may be implemented before encoding of video content, during encoding, after encoding but before decoding, during or after decoding of the video, and before creating and blending film grain.

[0092] The use of this mechanism can be:

[0093] 1. Before or during encoding, enable and make sure the decoder itself supports the film grain parameters signaled to the decoder.

[0094] 2. At the decoder or in a post-processing stage, if the specific implementation of film grain synthesis does not support the required model - If a decoder receives a bitstream with film grain parameters that are not supported by the specific implementation of film grain synthesis, it can convert the received film grain metadata and model parameters into an appropriate / understandable format / model.

[0095] 3. During transcoding of the bitstream, ensure that film grain synthesis is performed on the decoder side after transcoding is completed.

[0096] It is important to note that conversion of metadata does not necessarily require encoding and / or decoding of the video bitstream. For example, conversion of film grain syntax can be done based on metadata provided in a configuration file. It can also be done by analyzing the encoded bitstreams without decoding them, since existing metadata can be extracted (and added back) from the bitstream without actually decoding it. By doing so, only the film grain portion can be extracted and decoded, processed afterwards, and embedded into the bitstream at the correct location.

[0097] Figure 8 A method for converting film grain metadata from a first format to a second format is illustrated. At 800, a first set of film grain metadata is obtained. The first set of film grain metadata conforms to film grain metadata in a first format. The format of the film grain metadata may include a specific syntax for transmitting syntax elements related to film grain data, the syntax conforming to parameters of a video coding standard and / or a film grain model. In some embodiments, the first set of film grain metadata conforms to a first film grain syntax and represents a first film grain model.

[0098] According to an embodiment, as described above, since the conversion can be performed at different locations in the video decoding framework, obtaining the first set of film grain metadata can include reading film grain metadata stored in a memory, or estimating film grain metadata from an input video, or receiving and parsing a bitstream including film grain metadata, or receiving decoded film grain metadata from a decoder.

[0099] At 801, at least a portion of a first set of film grain metadata is converted into at least a portion of a second set of film grain metadata conforming to a second format. Thus, the second set of film grain metadata conforms to a second film grain syntax and represents a second film grain model.

[0100] In one variation, the method may further include: sending the converted second film grain metadata set together with the decoded video to a decoder or a post-processing device.

[0101] As described above, the conversion may include converting the first set of film grain metadata from a first video standard to a second video standard, wherein the video standard defines a film grain syntax, and / or converting the first set of film grain metadata from a first film grain model to a second film grain model, wherein the first film grain model and the second film grain model belong to the same or different video compression standards.

[0102] In the following, some examples of film grain syntax and models are described for the AV1 and VVC video coding standards.

[0103] In AV1, film grain is based on an autoregressive model (Autoregressive Model Variant-1). The syntax is defined in Section 5.9.30 "Film Grain Parameter Syntax" of the AV1 Bitstream and Decoding Process specification (Alliance for Open Media, Version 1.0.0 with Errata 1, January 18, 2019). The semantics of the parameters are given in Section 6.8.20 "Film Grain Parameter Semantics". The synthesis process of how film grain patterns are created and how they are blended into images is described in Section 7.18.3 "Film Grain Synthesis Process".

[0104] Note that in AV1, film grain compositing is part of mandatory post-processing. This means that if film grain parameters are present in the bitstream, the compositing process is called without exception and produces an output video with film grain.

[0105] The MPEG codec family (VVC, HEVC, H264 / AVC) supports two different models in its architecture. These are the autoregressive model (autoregressive model variant-2) and the frequency filter model.

[0106] Film grain metadata in VVC (as well as HEVC and H.264 / AVC) is communicated to the decoder via Supplemental Enhancement Information (SEI) messages. The syntax is defined in Versatile Video Coding, Recommendation ITU-T H.266 (08 / 2020) | International Standard ISO / IEC 23090-3, Section 8.5 "Film grain characteristics SEI message". The complete syntax and semantics are defined for two available models. In contrast to AV1 film grain, the VVC film grain specification only provides building blocks on how to transmit film grain parameters to the decoder. However, it does not provide an exact description of the synthesis process. An example of a specific implementation of synthesis based on a frequency filtering model using the FGC SEI message as defined in VVC is provided below: Sean McCarthy, Peng Yin, Walt Husak, Fangjun Pu, Taoran Lu, Tao Chen, Edouard JVET-W0095 report "AHG9: Fixed-point grain blending process for film grain characteristics SEI message" was presented by Vijayakumar GR, Kaustubh Patankar, Shireesh Kadaramandalgi and Ajayshyam via teleconference from July 7 to 16, 2021. Moreover, adding film grain is not a mandatory process in VVC even if film grain SEI is present in the bitstream (which can be simply discarded by the decoder).

[0107] Typically, two subsets of syntax elements are used in film grain metadata. The first subset consists of syntax elements that are not directly related to model parameters. They are considered to be additional syntax used to support the transmission of model parameters. It can be named general film grain metadata.

[0108] The second subset consists of model specific film grain parameters.

[0109] Examples of these two sets of VVC are given in Table 1 below. Note that the model specific parameters are the same for frequency filtering and autoregressive model variant-2. However, the interpretation of the syntax is different based on the model.

[0110] Table 1: Generic film grain parameters and model specific film grain parameters for VVC (autoregressive model variant-2 and frequency filter model)

[0111]

[0112] For simplicity, it is assumed here that fg_separate_colour_description_present_flag is set to 0. The given flag is needed in a specific use case when the decoded picture is not in the same colour representation domain as the simulated film grain. Here, it can be assumed that the film grain space and the video colour space are the same.

[0113] The film grain metadata for AV1 is different. It is given in Table 2 below.

[0114] Table 2: Generic film grain parameters and model specific film grain parameters for AV1 (autoregressive model variant-1)

[0115]

[0116]

[0117] Based on the semantics of certain film grain syntax elements, mappings between them can be defined. For generic film grain metadata, simple relationships can be defined. For model specific parameters, the conversion is more complex and should be implemented in conjunction with requirements in terms of film grain quality and in conjunction with preservation of the original film grain appearance, if this is one of the requirements. Additional requirements related to complexity or memory requirements can be imposed. For example, if computing resources are limited, the conversion process can only convert generic film grain metadata and use default values ​​for model specific parameters.

[0118] Some embodiments for converting the film grain metadata given in Tables 1 and 2 are described below. Figure 4 In , different conversions depending on the input and output formats are indicated. The embodiments provided herein can be implemented in different ways, with different requirements in terms of memory, complexity and / or final quality of the synthesized film grain (after metadata conversion).

[0119] According to the embodiment, different conversion modes are provided, each of which corresponds to a variation of the principle of the present invention. They are summarized as follows:

[0120] 1. Discard - According to this variant, the conversion is performed in a way that provides a minimal subset of parameters so that the decoder can process the bitstream syntax, but it does not synthesize film grain. This is an alternative way that does not render film grain and can produce output that is significantly different from what is expected. However, it is a less demanding process in terms of computation and memory.

[0121] 2. Random—According to this variant, the generic film grain metadata is converted, but for specific model parameters, random values ​​are used.

[0122] 3. Default - According to this variant, generic film grain metadata is converted, but for model specific parameters, default values ​​defined a priori for each model type are used. A total of three sets of default parameters can be provided, one for each model (frequency filtering, autoregressive variant-1, and autoregressive variant-2). In addition, for each model, several different configurations can be defined a priori, reflecting different film grain appearances.

[0123] 4. Mapping - According to this variant, the complete film grain metadata is converted with the highest possible accuracy. The goal is to achieve a visual match between the original film grain pattern after conversion and before conversion.

[0124] The different modes reflect different complexity, memory and quality requirements. In any case, the original video decoded stream remains unchanged, only the metadata is transformed or discarded / ignored.

[0125] According to this embodiment, the device that realizes conversion can select conversion mode based on need.Can select among the above-mentioned conversion modes or in the subset of these modes.According to another embodiment, this selection can be made by the user.

[0126] According to another embodiment, each of the variants described herein can be implemented independently of each other. In this case, the device does not need to select a conversion mode because the device can only implement one of the variants.

[0127] Variant: Discard Mode

[0128] In a variation called discard mode, the process simply ignores existing film grain metadata and no film grain compositing is performed. Instead, it removes / ignores the input film grain metadata or sets appropriate flags so that no film grain compositing is performed afterwards. There are multiple ways to achieve this, depending on the direction of the conversion.

[0129] In a first variant, if VVC film grain metadata (the model type is not important in this particular mode) needs to be converted to AV1 metadata, the film_grain_params_present or apply_grain flag should be set to 0. Since film_grain_params_present or apply_grain is 0, it does not require any other syntax elements.

[0130] In another variant, if AV1 film grain metadata needs to be converted to VVC film grain metadata in discard mode, one option is to just ignore all metadata and not embed it in the VVC bitstream. This is possible because film grain metadata is not a mandatory part of the VVC bitstream and there is no strict requirement to create and encode film grain SEI in the bitstream. Another option is to create SEI and embed it in the bitstream, but fg_characteristics_cancel_flag should be set to 1. Since fg_characteristics_cancel_flag is 1, no other syntax elements are required.

[0131] In another variant, when the conversion between VVC models is performed in discarding mode (VVC autoregressive variant-2 to VVC frequency filtering or vice versa), the same as the previous exemplary application. Just discard the film grain SEI from the bitstream or set fg_characteristics_cancel_flag to 1, and the decoder can process the bitstream without converting other syntax elements. However, it will not produce output with film grain.

[0132] Discarding mode is a simple solution that does not require any significant processing power or memory; however, it may produce undesirable results when synthetic film grain is expected and added to decoded video.

[0133] Variant: Random Mode

[0134] Fig.10 An example of a method for converting film grain metadata from a first format to a second format according to another variation referred to herein as random mode is illustrated. In this variation, only generic film grain metadata is converted, while model-specific film grain parameters are set in a random manner. At 1000, generic film grain metadata is converted by mapping one or more syntax elements of a first format to one or more syntax elements of a second format. At 1001, random values ​​are assigned to model-specific parameters. In this case, the model-specific parameters may be restricted by a supported range, and the parameters are initialized by taking random values ​​from the range.

[0135] VVC to VVC

[0136] In one example, the simplest conversion is between VVC models. All generic film grain metadata remains the same except for the fg_model_id flag. It should be changed in a way that the new value is represented by ~fg_model_id, where ~ is the negation of the flag. The model specific film grain parameters are then set in a random manner relative to the supported value range.

[0137] VVC to AV1 and vice versa

[0138] The conversion between VVC and AV1 or vice versa is more complex. It requires a mapping of generic film grain metadata. The model specific film grain parameters are set in a random way, the same as in the previous example, and relative to the supported value range. In such an example, the conversion is established as follows. There are two phases, the mapping phase and the expansion phase. The mapping phase produces relations between existing parameters and relations between parameters for which logical and / or mathematical relations can be observed. The expansion phase adds additional parameters that were not mapped in the first phase.

[0139] VVC to AV1

[0140] In another example, in the mapping stage, fg_characteristics_cancel_flag is mapped to film_grain_params_present and apply_grain. If fg_characteristics_cancel_flag is set to 1, film_grain_params_present and apply_grain are set to 0. After that, no other syntax elements need to be sent, so they do not need to be converted. If fg_characteristics_cancel_flag is set to 0, film_grain_params_present and apply_grain are set to 1.

[0141] There is no need to convert the fg_model_id flag, because there is only one model available in AV1. However, the VVC model needs to be converted to the AV1 model, which is converted in the same way. Therefore, when the frequency filter model and the autoregressive model variant-2 need to be converted to the autoregressive model variant-1, the conversion is applied to both the frequency filter model and the autoregressive model variant-2.

[0142] Furthermore, there is no need to convert fg_blending_mode_id, since only additive blending is defined in AV1. Even if fg_blending_mode_id indicates that multiplicative blending is used within VVC film grain modeling, it should not affect the result of this random mode conversion, since random parameters will be used to generate film grain anyway.

[0143] If fg_characteristics_persistence_flag is 0 (meaning that film grain applies only to the current decoded picture), it indicates that update_grain should be 1 (meaning that new film grain parameters need to be transmitted).

[0144] If fg_characteristics_persistence_flag is 1, then for the current frame update_grain remains 1. However, it has an effect on future frames. For future frames, update_grain is set to 0 and film_grain_params_ref_idx is set to the index of the current frame until one of the following conditions is met (as defined in the VSEI's Film Grain specification):

[0145] - New CLVS starts for the current layer

[0146] - End of bitstream

[0147] - Output the picture in the current layer in the access unit associated with the film grain characteristics SEI message, which follows the current picture in output order.

[0148] Therefore, after one of the given conditions is met, a new film grain SEI may be transmitted within the VVC bitstream, and the process is repeated. Essentially, this means that the film grain metadata is reused for all future frames until one of the given conditions applies. Therefore, for future frames that do not carry a film grain SEI, film grain is still applied until one of the conditions is met. The given condition basically resets the "scope" of the previous film grain SEI message. In AV1, this is equivalent to setting update_grain to 0, which means using the parameters from one of the previous frames. In contrast to VVC, in AV1, all frames need to be set until a new set of parameters is embedded in the bitstream (when update_grain is set to 1, indicating a new set of parameters).

[0149] During the mapping phase, for some syntax elements, it is not possible to provide a direct (or in some cases an indirect) mapping. In this case, a priori defined "best effort" values ​​can be provided during the expansion phase. In other words, in some specific implementations of the conversion process, predefined values ​​for those parameters for which no mapping exists can be used. Those parameters are then selected in a way that produces the best possible film grain quality.

[0150] The missing parameters are overlap_flag and grain_seed. overlap_flag defines whether deblocking is performed between film grain blocks. For example, in Sean McCarthy, Peng Yin, Walt Husak, Fangjun Pu, Taoran Lu, Tao Chen, Edouard In the JVET-W0095 report "AHG9: Fixed-Point Grain Blending Process for Film Grain Characterization SEI Messages", presented by Vijayakumar GR, KaustubhPataankar, Shireesh Kadaramandalgi and Ajayshyam via teleconference on July 7-16, 2021, a possible implementation design for VVC film grain synthesis is presented, using deblocking by default without the option to disable it. In this case, it makes sense to set overlap_flag to 1. However, it uses additional resources, so this can be a design choice. grain_seed is used in the pseudo number generator during the blending process. Since both synthesis algorithms (VVC's and AV1's) need to initialize the seed for each frame in order to ensure bit-accurate reproduction, it can be reused between the algorithms. The only difference is that VVC does not provide the seed as a parameter within the SEI, but it is an implementation-oriented choice, while AV1 still incorporates the seed into the film grain metadata structure.

[0151] Additionally, clip_to_restricted_range is equivalent to fg_full_range_flag, and it should be set based on the input video range.

[0152] AV1 to VVC

[0153] In another example, in the mapping stage, if film_grain_params_present and apply_grain are both set to 1, fg_characteristics_cancel_flag is set to 0. If one of the film_grain_params_present or apply_grain flags is 0, fg_characteristics_cancel_flag is set to 1 and no further mapping is needed since no other parameters are sent in this case.

[0154] grain_seed does not need to be mapped, as there is no seed parameter in the VVC syntax. Depending on the implementation, synthesis may still need to use a seed to ensure bit-accurate film grain generation, however it is implementation dependent and not part of the film grain metadata.

[0155] In order to map update_grain and film_grain_params_ref_idx to fg_characteristics_persistence_flag, it is useful to observe not only the current frame but also other future frames. If update_grain is set to 1, it means that a new set of film grain parameters is transmitted. For such frames, a new film grain SEI with the appropriately converted parameters needs to be created. However, some future frames may have update_grain set to 0 and film_grain_params_ref_idx pointing to the current frame. In this case, fg_characteristics_persistence_flag should be set to 1, meaning that the current film grain SEI applies to the current decoded frame, but also to future frames in output order.

[0156] For example, in Fig.11 In the given scenario, performing the metadata conversion is very simple. For the current (first) frame with update_grain equal to 1, the mapping and conversion flag fg_characteristics_persistence_flag needs to be set to 1. After that, for all subsequent frames with update_grain equal to 0, no associated film grain SEI needs to be created. If fg_characteristics_persistence_flag is set to 1 and subsequent frames do not have a film grain SEI embedded in their structure, they will reuse the last known set of parameters, as was the case before conversion from AV1. Next time a new SEI is created for the next frame with update_grain equal to 1. The process is repeated afterwards.

[0157] Fig.12 Another variant is illustrated. In this case, the above reference cannot be applied Fig.11 The problem is caused by the connection indicated by the dotted line. AV1 allows a frame to refer to film grain metadata from any previous frame by setting the appropriate film_grain_params_ref_idx. However, in VVC, it is only possible to reuse the latest available SEI. Fig.12In the illustrated case, the two frames connected by a dotted line can no longer use the indicated metadata, because in VVC they will be overwritten by new SEI messages. One solution could be to create a SEI at the "switch time" of each frame, and embed the SEI of each frame by switching the parameters pointed to by film_grain_params_ref_idx. In VVC, it is possible and even desirable to have one SEI per frame, because it ensures bit-accurate reproduction in tricky modes. In this case, when it is necessary to create one SEI per frame, set fg_characteristics_persistence_flag to 0 for each of those SEI messages.

[0158] In short, if it happens Fig.11 In the illustrated case, the current frame fg_characteristics_persistence_flag can be set to 1, and the conversion parameters and the creation of SEI messages for any frames with update_grain equal to 0 can be skipped. When the next frame with update_grain equal to 1 appears, a new SEI message is created in the same way. Fig.12 In the illustrated case, fg_characteristics_persistence_flag can be set to 0 for the current frame and all other frames, and then SEI messages can be created for all frames even if update_grain is 0. In this case, fg_characteristics_persistence_flag needs to be set to 0 and the other parameters converted as before. Note that the parameters to be converted are taken from the frame indicated by film_grain_params_ref_idx. Then, using random mode, the other model-specific parameters are set in a random manner as explained above.

[0159] Note that this is just one possible instantiation of a given problem. There may be other situations where similar logic can be applied.

[0160] Next, clip_to_restricted_range is equal to fg_full_range_flag, and it should be set based on the input video range. In addition, overlap_flag indicates whether deblocking will be performed. In the VVC SEI, there is no such parameter, so it is left to the implementer to choose whether to apply deblocking of film grain blocks during synthesis.

[0161] Afterwards, in the expansion phase fg_blending_mode_id is set to 0, indicating additive blending. In some embodiments, it can be set to 1, indicating multiplicative mode, but in this case the model-specific parameters need to be adjusted to this blending mode. fg_model_id is set to 0 or 1, depending on the target model after the conversion. Since random mode is used, the model-specific parameters will be set in a random manner and according to the selected model and blending mode.

[0162] Variant: Default Mode

[0163] In another variation, called default mode, the process is essentially similar to random mode, except that the model-specific parameters are defined in a way that uses some predefined set of parameters. Fig.13 An example of a method of converting film grain metadata from a first format to a second format according to a default mode variant is illustrated. At 1300, for a random mode, generic film grain metadata is mapped from a first format to a second format. At 1301, default values ​​for model-specific parameters are obtained. Several sets (different predefined sets of film grain-specific model parameters) may be used, and an index of the set may be input to the conversion process. Thus, according to this variant, obtaining a value at 1301 may include determining a set of specific model parameters and / or selecting a set of specific model parameters among the predefined sets. At 1302, the selected value is assigned to the specific model parameter of the second format.

[0164] In this way, the most appropriate set of parameters can be selected more flexibly. Compared to the random mode, the default mode can provide more meaningful transformations, because model-specific parameters can be manually selected and stored for further use. Compared to the random mode, a more controlled and meaningful transformation is achieved in this mode, but it requires the storage of predetermined parameters.

[0165] Choose from different predefined metadata for different models and blending modes. Choose based on different complexity or memory requirements combined with the final film grain look.

[0166] Generic film grain metadata is converted into a random pattern.

[0167] Variant: First Mapping Mode

[0168] Another variation referred to as the first mapping mode is provided below. Note that the names given to all variations for film grain metadata conversion are for purposes of illustration and naming, and the names given to any of the variations are not limiting and do not induce any particular characteristics of the variation.

[0169] The first mapping mode variant is the mode that requires the highest complexity. However, its purpose is to produce the best possible conversion in terms of maintaining the appearance of the original film grains. The process aims to establish the best mapping between all film grain grammar elements.

[0170] Fig.14 An example of a method for converting film grain metadata from a first format to a second format according to the first mapping mode variant is illustrated. At 1400, for the random mode and the default mode, the general film grain metadata is mapped from the first format to the second format.

[0171] Then, the conversion of the model-specific film grain parameters is performed.

[0172] According to the variant, at 1401, a first set of film grain patterns is determined from different parameter sets of the first film grain model, and a second set of film grain patterns is determined from different parameter sets of the second film grain model. The aim is to match the film grain patterns obtained from each film grain model. At 1402, a mapping between the first set of film grain patterns and the second set of film grain patterns is obtained. This can be performed in different ways.

[0173] According to one example, all possible film grain patterns are calculated offline for different models. In the offline phase, after the film grain patterns are calculated, experts can visually compare different patterns from different models to match similar patterns. For example, Fig.15 illustrates the possible visual matching and relationships of different patterns of two models. Note that there is a limited number of different patterns that can be represented by one model. In Fig.15 the left part, N patterns are mapped to M patterns, where N < M. Therefore, each pattern N can establish a unique relationship with a pattern from the other model. In Fig.15 the right part, M patterns of one model need to establish relationships with N patterns of the other model, where M > N. Therefore, several patterns of one model are mapped to the same pattern of the other model. Even if there is no unique mapping, this situation cannot be ignored in some cases, and a compromise must be made. For example, compared with VVC, AV1 supports a larger number of modelable patterns. Therefore, several AV1 film grain patterns are connected to one VVC pattern.

[0174] Since the model-specific parameters for each pattern are known, visual matching between the patterns results in a parameter conversion between the models. Thus, at 1403, a pattern from the film grain model of the second format is selected as the pattern that will be obtained from the film grain model parameters of the first format. Due to this match, each set of model parameters can be represented by another set of another model simply by replacing the model parameters of the input film grain metadata with the parameters of its visual equivalent from the other model. Since a given model is deterministic, visual matching is performed only once and stored for further use.

[0175] This method can be used for any of the conversion directions mentioned above.

[0176] In another variation, a numerical technique is proposed to find relationships rather than visual matches of patterns. For example, at 1402, the visual match is replaced by the minimization (or maximization) of some objective criteria. Divergence measures between two empirical probability distributions can be used to quantify the similarity between film grain patterns. One possibility is to use the Jensen-Shannon divergence (J. Lin, "Divergence measures based on the Shannonentropy", IEEE Transactions on Information Theory, Vol. 37, No. 1, pp. 145-151, January 1991), which provides a method for measuring the similarity between two distributions. The more similar the patterns are, the smaller the JSD. Therefore, for a pattern, the match is associated with the minimum JSD between the pattern and other patterns from different models. For a particular pattern i, the associated match j representing the most similar pattern from another model is given by the following relationship:

[0177] where i∈{0,N-1}

[0178] Other divergence measures can also be used instead of JSD. As mentioned before, this method can be used for any of the transformation directions mentioned above.

[0179] Variant: Secondary Mapping Mode

[0180] As an alternative example, a second mapping mode is proposed to match a given number of values ​​of model-specific film grain parameters by finding an analytical mapping between sets of parameter values ​​that give similar outputs (subject to a minimization criterion) due to the neural network.

[0181] In this variation, the goal is to determine the parameters of the neural network. The neural network does not necessarily need to be deep, and different types of network architectures can be used. It requires a training phase for determining the parameters of the neural network. However, once the neural network model is trained, it can be used immediately (at 1403) to convert model specific film grain parameters.

[0182] For the training phase, predetermined film grain patterns (e.g., as determined in 1401 above) and associated model-specific parameters are used. The input attributes, including the parameters of the film grain model, are typically given in the form of a vector. The output attributes, including the parameters of the mapped film grain, are also vectors that are not necessarily of the same size as the input vectors. The cost function (i.e., optimization criterion) of the neural network may be any objective measure that quantifies the similarity between two patterns obtained from the parameters of the film grain model input to the neural network and the similarity between the parameters of the film grain model obtained from the output of the neural network.

[0183] Once the neural network is trained, the parameters of the film grain model of the second set of film grain metadata are obtained as an output of the neural network, which has been fed with the parameters of the film grain model of the first set of film grain metadata.

[0184] By adopting one of the variants provided above, the bitstream delivered after decoding can produce an output picture with film grain. If the mapping mode occurs, a similar visual appearance of film grain will be observed.

[0185] Fig.16 A block diagram of a system in which aspects of embodiments of the present invention may be implemented is illustrated according to another embodiment. Fig.16 One embodiment of an apparatus 1600 for converting a first set of film grain metadata in a first format into a second set of metadata in a second format is shown, according to any of the embodiments described herein.

[0186] The device includes a processor 1610 and may be interconnected to a memory 1620 through at least one port. Both the processor 1610 and the memory 1620 may also have one or more additional interconnects to external connections.

[0187] The processor 1610 is also configured to receive a first set of film grain metadata in a first format, convert the first set of film grain metadata in the first format into a second set of metadata in a second format, and send the second set of metadata in the second format using any of the embodiments described herein. For example, the processor 1610 is configured using a computer program product that includes code instructions that implement any of the embodiments described herein.

[0188] exist Fig.17 In the illustrated embodiment, in the context of transmission between two remote devices A and B via a communication network NET, device A comprises a processor associated with memory RAM and ROM, the memory RAM and ROM being configured to implement as described with respect to Figure 1 and Figure 2 The method for encoding a video as described, and the device B comprises a processor associated with a memory RAM and a ROM, the memory RAM and the ROM being configured to implement the method as described with respect to Figure 1 and Figure 3 According to an embodiment, device A and / or device B is further configured to convert film grain metadata sent together with the decoded video, the film grain metadata being as combined with Figures 4 to 15 In some embodiments, device B is further configured to synthesize film grain on the received video using the converted film grain metadata.

[0189] According to an example, the network is a broadcast network adapted to broadcast / transmit the encoded video and converted film grain metadata from device A to decoding devices including device B.

[0190] Fig.18 An example of the syntax of a signal sent via a packet-based transmission protocol is shown. Each transmitted packet P includes a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may include decoded video data encoded according to any of the embodiments described above and converted film grain metadata.

[0191] Various specific implementations involve decoding. As used in the present application, "decoding" may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by decoders of various specific implementations described in the present application, such as decoding resampling filter coefficients, resampling decoded pictures.

[0192] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding, and in another embodiment, "decoding" refers to the entire reconstructed picture process including entropy decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to refer broadly to a broader decoding process will be clear based on the context of the specific description and is considered to be well understood by those skilled in the art.

[0193] Various specific implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in this application may cover, for example, all or part of the process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy decoding. In various embodiments, such processes also or alternatively include processes performed by encoders of various specific implementations described in this application, such as determining resampling filter coefficients, resampling decoded pictures.

[0194] As a further example, in one embodiment, "encoding" refers only to entropy coding, in another embodiment, "encoding" refers only to differential coding, and in yet another embodiment, "encoding" refers to a combination of differential coding and entropy coding. Whether the phrase "encoding process" specifically refers to a subset of operations or broadly refers to a broader encoding process will be clear based on the context of the specific description, and is believed to be well understood by those skilled in the art.

[0195] Note that the syntax elements used herein are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0196] This disclosure has described various information that can be sent or stored, such as syntax, for example. This information can be packaged or arranged in a variety of ways, including, for example, ways that are common in video standards, such as putting the information in an SPS, PPS, NAL unit, header (e.g., NAL unit header or slice header), or SEI message. Other ways are also available, including, for example, common ways for system-level or application-level standards, such as putting the information in one or more of the following:

[0197] a. SDP (Session Description Protocol), which is a format for describing multimedia communication sessions for session announcement and session invitation, for example, as described in RFC and is compatible with RTP

[0198] (Real-time Transport Protocol) sending.

[0199] b. DASH MPD (Media Presentation Description) descriptor, eg as used in DASH and sent over HTTP, a descriptor is associated with a representation or a set of representations to provide additional characteristics to the content representation.

[0200] c. RTP header extension, for example, as used during RTP streaming.

[0201] d. ISO Base Media File Format, e.g. as used in OMAF and uses boxes, which are object-oriented building blocks defined by a unique type identifier and a length, also referred to as "atoms" in some specifications.

[0202] e. HLS (HTTP Live Streaming) manifest sent over HTTP. For example, the manifest may be associated with a version or set of versions of the content to provide characteristics of the version or set of versions.

[0203] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of corresponding devices. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of corresponding methods / processes.

[0204] Some embodiments refer to rate-distortion optimization. Specifically, during the encoding process, the balance or trade-off between rate and distortion is usually considered, which often takes into account the constraints of computational complexity. Rate-distortion optimization is usually expressed as minimizing the rate-distortion function, which is the weighted sum of rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options (including all considered modes or decoding parameter values), and fully evaluate their decoding costs and the related distortion of the reconstructed signal after decoding and decoding. Faster methods can also be used to reduce coding complexity, especially for the calculation of approximate distortion based on prediction or prediction residual signal rather than reconstructed residual signal. A mixture of these two methods can also be used, such as by using approximate distortion for only some of the possible coding options, and using complete distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many methods use any of various techniques to perform optimization, but optimization is not necessarily a complete evaluation of both decoding cost and related distortion.

[0205] The specific implementations and aspects described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of specific implementation (e.g., discussed only as a method), the specific implementation of the features discussed may also be implemented in other forms (e.g., a device or program). The device may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0206] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearance of the phrase "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.

[0207] Additionally, the present application may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0208] Furthermore, the present application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0209] Additionally, the present application may involve "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Moreover, "receiving" generally involves, in one way or another, during operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0210] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first listed option and the second listed option (A and B), or only the first listed option and the third listed option (A and C), or only the second listed option and the third listed option (B and C), or all three options (A and B and C). This can be extended to as many items as listed, as will be apparent to one of ordinary skill in this and related arts.

[0211] In addition, as used herein, the term "signaling" refers to (among other things) indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a specific one of a plurality of resampled filter coefficients. In this way, in an embodiment, the same parameters are used on both the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding sending any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, information is signaled to the corresponding decoder using one or more syntax elements, flags, etc. Although the verb form "signaling" is mentioned above, the term can also be used as a noun "signal" herein.

[0212] It will be apparent to one of ordinary skill in the art that a specific implementation may generate various signals formatted to carry, for example, storable or transmittable information. The information may include, for example, instructions for executing a method or data generated by one of the described specific implementations. For example, a signal may be formatted to carry a bit stream of the described embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, the signal may be sent over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0213] A number of embodiments have been described above. The features of these embodiments may be provided individually or in any combination in the various claim categories and types.

Claims

1. A method, comprising: - obtaining first film grain metadata, said first film grain metadata conforming to a first film grain syntax and representing a first film grain model; - converting at least a portion of the first film grain metadata into at least a portion of second film grain metadata, the second film grain metadata conforming to a second film grain syntax and representing a second film grain model; - wherein the second film grain model is different from the first film grain model, and wherein syntax elements representing parameters of the film grain model are assigned based on a match of film grain patterns obtained for the first film grain model and the second film grain model, respectively.

2. The method of claim 1, wherein at least one of the first film grain syntax or the second film grain syntax complies with a video compression standard.

3. The method of claim 1 , wherein converting at least a portion of the first film grain metadata into at least a portion of the second film grain metadata comprises: - identifying a film grain pattern from a set of film grain patterns obtained from different parameters of said second film grain model that matches a film grain pattern obtained from parameters of said first film grain model of said first film grain metadata; - assigning to a syntax element of said second film grain model representing said second film grain metadata film grain model parameters providing said identified film grain pattern.

4. The method of claim 3, wherein converting at least a portion of the first film grain metadata into at least a portion of the second film grain metadata further comprises: - determining a first set of film grain patterns from different sets of parameters of the first film grain model and a second set of film grain patterns from different sets of parameters of the second film grain model, respectively; - obtaining a mapping between film grain patterns of said first set of film grain patterns and film grain patterns of said second set of film grain patterns.

5. The method of claim 4, wherein the mapping between the film grain patterns of the first film grain pattern set and the film grain patterns of the second film grain pattern set is obtained using a similarity measure determined between the film grain patterns of the respective first and second film grain pattern sets.

6. The method according to claim 1, further comprising: The converted second film grain metadata is sent together with the decoded video.

7. A computer-readable storage medium having stored thereon instructions for causing one or more processors to execute the method according to claim 1.

8. An apparatus comprising one or more processors, wherein the one or more processors are operable to: - obtaining first film grain metadata, said first film grain metadata conforming to a first film grain syntax and representing a first film grain model; - converting at least a portion of the first film grain metadata into at least a portion of a second film grain metadata set, the second film grain metadata set conforming to a second film grain syntax and representing a second film grain model; - wherein the second film grain model is different from the first film grain model, and wherein syntax elements representing parameters of the film grain model are assigned based on a match of film grain patterns obtained for the first film grain model and the second film grain model, respectively.

9. The device of claim 8, wherein at least one of the first film grain syntax or the second film grain syntax conforms to a video compression standard.

10. The apparatus of claim 8, wherein converting at least a portion of the first film grain metadata into at least a portion of the second film grain metadata comprises: - identifying a film grain pattern from a set of film grain patterns obtained from different parameters of said second film grain model that matches a film grain pattern obtained from parameters of said first film grain model of said first film grain metadata; - assigning to a syntax element of said second film grain model representing said second film grain metadata film grain model parameters providing said identified film grain pattern.

11. The apparatus of claim 10, wherein converting at least a portion of the first film grain metadata into at least a portion of the second film grain metadata further comprises: - determining a first set of film grain patterns from different sets of parameters of the first film grain model and a second set of film grain patterns from different sets of parameters of the second film grain model, respectively; - obtaining a mapping between film grain patterns of said first set of film grain patterns and film grain patterns of said second set of film grain patterns.

12. The apparatus of claim 11, wherein the mapping between the film grain patterns of the first film grain pattern set and the film grain patterns of the second film grain pattern set is obtained using a similarity measure determined between the film grain patterns of the corresponding first film grain pattern set and the second film grain pattern set.

13. The apparatus of claim 8, wherein the one or more processors are further configured to send the converted second film grain metadata with the decoded video.

14. A device, comprising: - a device according to claim 8; and - at least one of: (i) an antenna configured to receive a signal comprising data representing a video and at least one of a first set of film grain metadata or a second set of film grain metadata; (ii) a band limiter configured to limit the received signal to a frequency band comprising the data representing the video; Or (iii) a display configured to display decoded video having film grain synthesized using said second set of film grain metadata.

15. The device of claim 14, the device comprising a TV, a cell phone, a tablet or a set-top box.