Video encoding and decoding based on region-dependent film grain metadata

Abstract

A method and apparatus allow defining image regions, a set of film grain characteristics, and provide means for associating image regions with film grain characteristics. An encoded video bitstream includes specific syntax allowing transmission of such information from an encoding apparatus to a decoding apparatus. In one use example of the application, determined film grain is synthesized and applied to images of a decoded video. This enables differentiation of film grain characteristics for different regions, for example to accommodate different types of video sources.

Description

Technical Field

[0001] At least one embodiment of the present invention relates in general to a method and apparatus for encoding and decoding video including regionally correlated film grain metadata. Background Technology

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transform to utilize spatial and temporal redundancy in the video content. Generally, intra-frame or inter-frame prediction is used to leverage intra-frame or inter-frame correlations, followed by transform, quantization, and entropy coding of the differences between the original and predicted image blocks (typically represented as prediction error or prediction residual). During encoding, the original image blocks are typically partitioned / divided into sub-blocks, for example, using quadtree partitioning. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to prediction, transform, quantization, and entropy coding. Summary of the Invention

[0003] According to a first aspect of at least one embodiment, a method includes: obtaining a bitstream including encoded video data and region-related film grain metadata; determining film grain characteristics of one or more regions of a frame of the video based on the region-related film grain metadata for the regions; and providing the determined film grain characteristics.

[0004] According to a second aspect of at least one embodiment, a method includes: obtaining a bitstream including encoded video data and region-related film grain metadata; determining film grain characteristics of one or more regions of a frame of the video based on the region-related film grain metadata for the regions; and providing the determined film grain characteristics. A variant embodiment of the second aspect further includes: decoding the frame based on the encoded video data; determining a region-specific film grain sample based on the determined film grain characteristics; adding the film grain sample to the decoded frame; and providing the modified decoded frame.

[0005] According to a third aspect of at least one embodiment, an apparatus includes one or more processors configured to: acquire video data; determine region-related film grain metadata representing film grain characteristics; insert the region-related film grain metadata into an encoded bitstream including the video data and the region-related film grain metadata; and provide the bitstream.

[0006] According to a fourth aspect of at least one embodiment, an apparatus includes one or more processors configured to: acquire a bitstream including encoded video data and region-related film grain metadata; determine film grain characteristics of one or more regions of a frame of the video based on the region-related film grain metadata; and provide the determined film grain characteristics. In a variation of the fourth aspect, the one or more processors are further configured to decode the frame based on the encoded video data, determine region-related film grain samples based on the determined film grain characteristics, add the film grain samples to the decoded frame, and provide the modified decoded frame.

[0007] One or more embodiments of the present invention also provide a computer-readable storage medium storing instructions according to at least a portion of any of the methods described above. One or more embodiments also provide a computer-readable storage medium storing a bitstream generated according to the encoding method described above. One or more embodiments also provide a computer program product including instructions for performing at least a portion of any of the methods described above. Attached Figure Description

[0008] Figure 1 A block diagram of an example system in which various aspects and embodiments are implemented is shown.

[0009] Figure 2 Examples of contexts in which the following embodiments can be implemented are described.

[0010] Figure 3 A block diagram of an example video encoder is shown.

[0011] Figure 4 A block diagram of an example video decoder is shown.

[0012] Figure 5 An example of the partitions that the images in the original video sequence have gone through is shown.

[0013] Figure 6 An example of film grain parameters, including frequency cutoff and gain, is shown.

[0014] Figure 7 An example of the reproduction of a film scan is shown.

[0015] Figure 8 An example of a region of the screen according to an embodiment is shown.

[0016] Figure 9A An exemplary process for encoding a bitstream including FGRC-SEI messages, according to an embodiment, is shown.

[0017] Figure 9B An exemplary process for decoding a bitstream including FGRC-SEI messages, according to an embodiment, is shown.

[0018] Figure 9C An exemplary process is shown, according to an embodiment, for decoding a bitstream including FGRC-SEI messages and applying the determined film grain characteristics to the decoded image. Detailed Implementation

[0019] Various embodiments involve embedding metadata describing region-related film grain metadata into an SEI message. On the encoding side, the SEI message, including the region-related film grain metadata, is inserted into the bitstream. On the decoding side, the SEI message is extracted from the bitstream, and the region-related film grain metadata can be used to generate synthetic film grain samples, for example, adding them to the decoded image for artistic effects.

[0020] Although the principles are described in the context of the VVC (Video Coding Universal) or HEVC (High-Efficiency Video Coding) specifications, aspects of the invention are not limited to these video coding standards and can be applied to, for example, other standards and recommendations, as well as any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application may be used alone or in combination.

[0021] Figure 1 This is a block diagram illustrating examples of systems in which the various aspects and examples described herein can be implemented. System 100 can be implemented as a device or apparatus including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as computers, smartphones, tablets, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, in-vehicle entertainment systems, vehicle control systems, drones, video surveillance cameras, and more broadly, data servers. The elements of system 1000 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0022] System 1000 includes at least one processor 1010 configured to execute instructions loaded thereon to implement various aspects described herein, for example. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 440, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash-based memory, disk drives, solid-state drives (SSDs), and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices (also known as cloud storage).

[0023] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device to perform the encoding and / or decoding functions described further below. It is well known that a device may include one or both of the encoding and decoding modules. Alternatively, the encoder / decoder module 1030 may be implemented as a separate element of system 1000, or it may be incorporated into processor 1010 as a combination of hardware and software known to those skilled in the art.

[0024] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute one or more of the aspects described in this document may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from equations, formulas, operations, and operational logic processing.

[0025] In some embodiments, the memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be the processor 1010 or encoder / decoder module 1030) is used for one or more of these functions. External memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, fast external volatile memory such as RAM is used as working memory for video encoding and decoding operations, for example for MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, where 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265), or VVC (Various Video Coding, also known as H.266).

[0026] Inputs to the components of system 1000 can be provided through various input devices, as indicated in box 1130. Such input devices include, but are not limited to, radio frequency (RF) sections that receive, for example, RF signals transmitted over the air by a broadcaster, component (COMP) input terminals (or a set of COMP input terminals), universal serial bus (USB) input terminals, and / or high-definition multimedia interface (HDMI) input terminals. Other examples include composite video.

[0027] In various embodiments, the input device of block 1130 has associated input processing elements known in the art. For example, the RF section may be associated with elements suitable for: selecting a desired frequency (also referred to as selecting a signal, or limiting a signal band to a band), down-converting the selected signal, further band-limiting it to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), demodulating the down-converted and band-limited signal, performing error correction, and demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners performing various functions among these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to the desired frequency band. Various embodiments rearrange the order of the aforementioned (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna. The RF section may conform to standards specifications such as those published by Digital Video Broadcasting (DVB), the Advanced Television Systems Committee (ATSC), the Radio Industry Association and Commercial Association (ARIB), or other organizations.

[0028] Furthermore, USB and / or HDMI terminals may include corresponding interface processors for connecting System 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within processor 1010 as needed. Similarly, aspects of USB or HDMI interface processing may be implemented, as needed, within a separate interface IC or within processor 1010. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements (including, for example, processor 1010 and encoder / decoder 1030), which operate in conjunction with memory and storage elements to process the data stream as needed for presentation on the output device.

[0029] Various components of system 1000 can be housed within an integrated housing. Within the integrated housing, the various components can be interconnected and transmit data between them using suitable connection devices 1140 (e.g., internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0030] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 460. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0031] In various embodiments, data is streamed to or otherwise provided to system 1000 using a wireless network, such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 1000, delivering data via an HDMI connection to input box 1130. Still other embodiments use an RF connection to input box 1130 to provide streaming data to system 1000. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth® networks.

[0032] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. Display 1100 can also be integrated with other components (e.g., as in a smartphone) or separate (e.g., an external monitor for a laptop computer). In various examples, other peripheral devices 1120 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, for both terms), an optical disc player, a stereo system, and / or a lighting system. Various embodiments of the embodiments use one or more peripheral devices 1120 to provide functionality based on the output of system 1000. For example, an optical disc player performs the function of playing the output of system 1000.

[0033] In various embodiments, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention are used to transmit control signals between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120. Output devices may be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 may be integrated into a single unit along with another component of system 1000 in an electronic device, such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller chip.

[0034] For example, if the RF input section 1130 is part of a separate set-top box, the display 1100 and speaker 1110 can alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal can be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0035] The embodiments can be executed by processor 1010 or by computer software implemented by hardware or a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 1010 can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0036] Figure 2Examples of contexts in which the following embodiments can be implemented are described. In this context 200, system 210 transmits a video stream to system 230 using communication channel 220. Examples of system 210 include cameras, storage devices, computers, drones, video surveillance cameras, servers, or any device capable of delivering video streams. The video stream may be encoded and transmitted by system 210, or it may be received and / or stored by system 210 and then transmitted. Communication channel 220 is a wired (e.g., Internet, Ethernet, cable network) or wireless (e.g., WiFi, 3G, 4G or 5G, satellite TV, terrestrial TV) network link. System 230 receives and decodes the video stream to generate a sequence of decoded images. An example of system 230 is a set-top box. The obtained sequence of decoded images is then transmitted to display system 250 using communication channel 240, which may be a wired or wireless network as described above. Display system 250 then displays the images. An example of display system 250 is a television or display monitor.

[0037] In one embodiment, system 230 and display system 250 are included in a single device, thereby combining the reception, decoding, and display of a video stream. Examples of such devices are televisions, computers, tablets, smartphones, head-mounted displays, in-vehicle entertainment systems, and medical devices.

[0038] Figure 3 A block diagram of an example video encoder is shown. Variations of this encoder 300 can be envisioned, but for clarity, encoder 300 is described below without describing all anticipated variations. Before being encoded, the video sequence may undergo pre-coding processing 301 (e.g., applying color transformations to the input color picture (e.g., a conversion from RGB 4:4:4 to YcbCr 4:2:0), or performing remapping of the input picture components) to obtain a signal distribution more suited to compression (e.g., histogram equalization using one of the color components). For standards that include such a mechanism, metadata may be associated with the pre-processing and appended to the bitstream, for example, in the form of Supplemental Enhancement Information (SEI) messages.

[0039] In an encoder, the image is encoded by encoder elements, as described below. For example, as... Figure 5The text further describes how the frame to be encoded is partitioned (302) and processed in units such as coding units. Each unit is encoded using, for example, intra-frame or inter-frame modes. When a unit is encoded in intra-frame mode, intra-frame prediction (360) is performed. In inter-frame mode, motion estimation (375) and compensation (370) are performed. The encoder determines (305) which of the intra-frame or inter-frame modes to use to encode the unit, and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (320) the prediction block from the original image block. The prediction residual is then transformed (325) and quantized (330). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy-coded (345) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass the transform and quantization, i.e., encode the residual directly without applying transform or quantization.

[0040] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual is combined with the prediction block (355) to reconstruct the image block. An in-loop filter (365) is applied to the reconstructed image to perform filtering such as deblocking / SAO (Sampling Adaptive Offset) and Adaptive Loop Filter (ALF) to reduce coding artifacts. The filtered image is stored in a reference image buffer (380) for further use.

[0041] Encoders typically also perform video decoding as part of the encoded video data.

[0042] Figure 4 A block diagram of an example video decoder is shown. In decoder 400, the bitstream is decoded by decoder elements as described below. Video decoder 400 typically performs a decoding process that is the inverse of the encoding process described in the previous figures. The input to the decoder includes a video bitstream, which can be generated by... Figure 3 The video encoder 300 generates the image. First, the bitstream is entropy decoded (430) to obtain transform coefficients, motion vectors, and other encoded information. The frame partitioning information indicates how the frame is partitioned. Therefore, the decoder can partition (435) the frame based on the decoded frame partitioning information. The transform coefficients are dequantized (440) and inverse transformed (450) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (455) to reconstruct the image blocks. The prediction blocks (470) can be obtained from intra-frame prediction (460) or motion-compensated prediction (i.e., inter-frame prediction) (475). An in-loop filter (465) is applied to the reconstructed image. The filtered image is stored in a reference frame buffer (480).

[0043] The decoded image can undergo further post-decoding processing (485), such as inverse color transformation (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs pre-encoding processing ( Figure 3 The reverse process of the remapping process performed in (301). Post-decoding processing can use metadata derived in the precoding process and signaled in the bitstream.

[0044] Figure 5 An example of partitioning the images of the original video sequence is shown. The original video sequence 500 includes multiple frames 510. The images consist of multiple pixels, which are typically arranged in a grid consisting of rows and columns. In this document, a pixel is considered to consist of three components: one luminance component and two chrominance components. However, it is also possible for other types of pixels to include fewer or more components, such as only a luminance component or with an additional depth component or an additional transparency component.

[0045] The image is divided into multiple coded entities. First, as indicated by reference numeral 530, the image is divided into a grid of blocks called Coded Tree Units (CTUs). A CTU consists of a luminance sample block and two corresponding chrominance sample blocks. The size of such a block is typically N×N, and N is usually a power of two, for example, having a maximum value of "128". Second, the image is divided into one or more groups of CTUs. For example, it can be divided into one or more tile rows and tile columns, where a tile is a sequence of CTUs covering a rectangular area of ​​the image. In some cases, a tile can be divided into one or more bricks, each brick consisting of at least one row of CTUs within the tile. Above the concepts of tiles and bricks, there is another coded entity called a slice, which can contain at least one tile of the image or at least one brick of a tile. In the example indicated by reference numeral 520, the image 21 is divided into three slices S1, S2, and S3 in a raster scan slice pattern, each slice comprising multiple tiles (not shown), and each tile comprising only one brick.

[0046] As indicated by reference numeral 540 in the accompanying figure, a CTU can be partitioned into a hierarchical tree of one or more sub-blocks called coding units (CUs). The CTU is the root (i.e., the parent node) of the hierarchical tree and can be divided into multiple CUs (i.e., child nodes). If each CU is not further partitioned into smaller CUs, it becomes a leaf of the hierarchical tree; otherwise, it becomes the parent node of smaller CUs (i.e., child nodes). During frame encoding, partitioning is adaptive, partitioning each CTU to optimize compression efficiency.

[0047] For example, first, use quadtree partitioning to divide CTU 540 into four square CUs. The top-left CU 541 is a leaf of the hierarchical tree because it is not further divided; that is, it is not the parent node of any other CU. Again, use quadtree partitioning to further partition the top-right CU into four smaller square CUs 551, 552, 553, and 554. Use ternary tree partitioning to vertically partition the bottom-left CU into three rectangular CUs 561, 562, and 563. Use binary tree partitioning to vertically partition the bottom-right CU into two rectangular CUs 571 and 572.

[0048] In HEVC, the concepts of Prediction Unit (PU) and Transform Unit (TU) emerge. In practice, in HEVC, the coding entities used for prediction (i.e., PU) and transformation (i.e., TU) can be subdivisions of a CU. For example, as shown in the figure, a CU of size 2N×2N can be divided into PUs of size N×2N or 2N×N (580). Furthermore, this coding unit can be divided into four TUs of size N×N (590), or into "16" TUs of size (N / 2) × (N / 2). Other video coding standards also use these concepts. In VVC, except in some specific cases, the boundaries of TUs and PUs are aligned with the boundaries of the CU. Therefore, a CU typically includes one TU and one PU.

[0049] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," "subpicture," "slice," and "frame." Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side. In this application, the term "block" or "picture block" can be used to refer to any of CTU, CU, PU, ​​and TU. Additionally, the term "block" or "picture block" can be used to refer to macroblocks, partitions, and subblocks as specified in H.264 / AVC or other video decoding standards, and more generally to sample arrays of many sizes.

[0050] Video coding standards such as AVC, HEVC, and VVC enable the embedding of metadata into the video bitstream via SEI (Supplemental Enhancement Information) messages. These messages are defined in core standards or supporting standards (such as VSEI (General Supplemental Enhancement Information)), and more messages are added in subsequent versions of these standards.

[0051] SEI messages are syntactic structures defined in various MPEG standards to allow the carrying of metadata. It is a specific type of NAL (Network Access Layer) unit, which is the basic block of the MPEG bitstream format. SEI syntax may vary slightly between different standards, but it typically includes at least the payload type, payload length, and the payload itself. Table 1 shows an example of the SEI syntax defined for VVC.

[0052] Table 1

[0053]

[0054] In addition, a specific syntax structure is usually defined for each payload type, and that specific syntax structure is instantiated according to each payload type through the sei_payload syntax structure.

[0055] Various video coding standards, including AVC, HEVC, and VVC / VSEI, allow the embedding of specific metadata that describes film grain characteristics as parameters of a statistical model (film grain characteristics are sometimes referred to as model parameters). This model typically defines grain intensity and spatial characteristics as functions of image sample values ​​or intensities. The parameters can be different for each color component. Spatial characteristics can be described as coefficients of a frequency cutoff or a 2D autoregressive filter. This model can be used to describe grain present in the source video, which may be lost in the encoded bitstream. The decoder can then use this model to add synthetic grain on top of the decoded video, thereby recovering a grain appearance as close as possible to the original.

[0056] Film grain is the random optical texture of photographic film introduced during the film development process. When used appropriately, film grain is an artistic effect that makes video feel more natural and believable, enhancing the viewer's experience. However, in today's digital world, because film is no longer present, digitally captured and digitally generated content no longer exhibits film grain. Furthermore, because natural film grain is a form of noise, it is often suppressed by certain features of video coding algorithms. By adding grain to digital images, the visual and artistic effects of film grain can be simulated. Therefore, in the digital video ecosystem, film grain can be introduced on the decoder side by adding it to digital images to simulate its visual and artistic effects.

[0057] In AVC, HEVC, and VVC / VSEI, film grain characteristics can be described using FGC (Film Grain Features) SEI messages; the model used is based on intensity intervals, where model parameters (intensity attributes and spatial attributes) are specified for each interval. The intervals and model parameters are unique to each color component. After extracting the film grain features, the decoder can generate corresponding film grain samples and add them to the decoded image to achieve the desired artistic effect. Table 2 shows the current metadata structure for film grain characteristics defined in VSEI.

[0058] Table 2

[0059] In the structure of Table 2, the syntax element `fg_characteristics_cancel_flag` is used to handle the persistence of any previous film grain characteristic SEI messages. The syntax element `fg_model_id` identifies the film grain simulation model, selected from the following: a value of 0 for the frequency_filtering model, and a value of 1 for the autoregressive model. The syntax element `fg_separate_colour_description_present_flag` equal to 1 indicates the existence of different color space descriptions for the film grain characteristics specified in the SEI message, and further specifies the colors. Color space definitions are not represented in Table 2 and are replaced by three dots between parentheses in the seventh row. The syntax element `fg_blending_mode_id` identifies the blending mode used to blend the simulated film grain with the decoded image, selected from the following: a value of 0 for the additive mode, and a value of 1 for the multiplicative mode. The syntax element `fg_log2_scale_factor` specifies the scaling factor used in the film grain characterization equation. The syntax element fg_comp_model_present_flag[c] indicates whether film grain is modeled on color components, such as Y / Cb / Cr components or RGB components. fg_comp_model_present_flag[c] equal to 1 indicates that the syntax element specifying film grain modeling on color component c exists in the SEI message. The syntax element fg_num_intensity_intervals_minus1[c] plus 1 specifies the number of intensity intervals for which a particular set of parameters is provided. The syntax element fg_num_model_values_minus1[c] plus 1 specifies the number of model values ​​present for each intensity interval in which film grain has been modeled. Its value should be in the range of 0 to 5, inclusive. The syntax element fg_intensity_interval_lower_bound[c][i] specifies the lower bound of the intensity level interval i to which the set of model values ​​is applied. The syntax element `fg_intensity_interval_upper_bound[c][i]` specifies the upper bound of the intensity level interval `i` to which this set of model values ​​is applied. The syntax element `fg_comp_model_value[c][i][j]` represents each of the model values ​​for the color component `c` and the intensity interval `i`, and this set of model values ​​has different meanings depending on the value of `fg_film_grain_model_id`.The `fg_film_grain_characteristics_persistence_flag` flag specifies the persistence of the film grain characteristic SEI message for the current layer. A value of 0 indicates that the film grain characteristic SEI message applies only to the currently decoded frame.

[0060] Figure 6 An example of film grain parameters, including frequency cutoff and gain, is shown. Figure 600 illustrates the relationship between film grain model parameters and intensity. The intensity used in this example is the luminance component (Y). Curve 610 represents grain intensity (also called (or "gain")), with the intensity interval boundaries indicated by vertical dashed lines. Curve 620 represents grain sharpness or size, here expressed in the form of a spatial frequency cutoff. In this example, the intensity interval covers the entire range without gaps or overlaps, but this is not necessarily the case: the FGC SEI message syntax allows for the definition of intervals with gaps, overlaps, and disorder.

[0061] Figure 7 An example of film scan reproduction is shown. It demonstrates a significant difference in film grain characteristics between shadows and highlights, thus proving the correctness of the different model parameters (including spatial properties) that depend on intensity.

[0062] The embodiments described below were designed in light of the foregoing, and these embodiments propose defining image regions, sets of film grain characteristics, and provide means for associating image regions with film grain characteristics. The encoded video bitstream includes a specific syntax that allows this information to be passed from the encoding device to the decoding device and allows the regeneration of the corresponding film grain of the image to be applied to the decoded video. Variations of this embodiment are described below.

[0063] At least one embodiment proposes defining a film grain region feature metadata structure (hereinafter abbreviated as FGRC-SEI) that takes the form of a specific SEI message to carry region-related film grain metadata. This FGRC-SEI is inserted into the video bitstream, enabling the decoder to generate appropriate synthetic film grain samples for different regions of the frame based on the film grain features (i.e., parameters) extracted from the metadata carried by the FGRC-SEI. The film grain samples are then added to the decoded frame.

[0064] Different techniques can be used to associate image regions with film grain characteristics. The first technique is to assign regions to intensity ranges. The second technique is to assign model parameters to regions. The third technique is to assign label identifiers to both model parameters and regions, thus using an indirect reference system for the film property set.

[0065] Different methods can be used to define image regions. Regions can be defined as a list of rectangular areas, each identified by its list index or a special identifier and characterized by its location and size. A single region (or area) can be determined by its top-left corner and size, or by the position of a pair of opposite corners, or by the horizontal and vertical boundaries of the region, or by the location of the region's center and its size. Regions can be defined by reusing a list of externally defined image regions, such as regions with identifiers from annotation area SEI messages, sub-pictures and their identifiers (e.g., defined for VVC bitstreams), tile patches and their identifiers (e.g., defined for VVC and HEVC bitstreams), rectangular or non-rectangular slices, or object masks defined by sample values ​​from auxiliary pictures (as in the proposed OMI SEI message). In some cases, the syntax describing the characteristics of the object mask (sample values, bounding boxes, or other parameters associated with each object mask) can be included in the film grain region characteristic metadata structure. In embodiments, image regions can be updated independently. In practice, it is not necessary to fully specify the list of regions embedded in the film grain region characteristic metadata structure in each message that is regularly inserted within the bitstream. For example, a portion (e.g., a subset of a region) can be updated by defining its new position, by translating its position by a specified motion vector, or by transforming its position and size (or two relative angles) using specified parametric transformations (such as affine transformations). Other parametric transformations, such as homography transformations, can also be considered. When defining a region transformation, the transformation can be specified to be repeatedly applied to subsequent frames without requiring new Film Grain Region Feature (SEI) messages until canceled by a subsequent FGRC-SEI message (or the sequence ends). In at least one embodiment, the regions do not overlap.

[0066] Figure 8 An example of a region of a frame according to an embodiment is shown. In this example, frame 800 is divided into three distinct regions, identified as A, B, and C in the diagram. These regions may have different film grain characteristics. This occurs, for example, when the frame is a composite of different sources (e.g., film scans, video from a digital camera, or synthetic content). Due to the specific film grain metadata for each region, the embodiments described herein allow for the generation of frames with consistent film grain across all pixels.

[0067] Film grain characteristics can be defined using different methods. They can be defined as a list of common parameter parts (e.g., color space selection, model selection, number of model parameters) and intensity intervals, where model parameters are specific to each interval. Furthermore, image regions can be defined and associated with intervals and interval-specific parameters. Intervals and model parameters for a region can be grouped in a specific structure and / or identified by identifiers, allowing for more efficient reuse of parameter sets among several image regions. Existing structures (e.g., film_grain_characteristics introduced in Table 2), as defined in the AVC, HEVC, and VSEI standards, can be used. In this case, some common parts may be restricted to having the same values ​​(e.g., model selection: frequency filtering or autoregression). Different parts of the film grain characteristics can be updated independently (e.g., for different regions). In practice, it is not necessary to fully specify the list of intervals (potentially associated with different image regions) embedded in the film grain region characteristic metadata structure in each message: for example, a portion can be updated by specifying which intervals (of which regions) are being updated.

[0068] All these techniques can be combined to design the most appropriate mechanism to associate a specific image region with a specific film grain characteristic.

[0069] The embodiments described herein can be applied to AVC, HEVC, VVC / VSEI, and other video coding systems based on the same principles and using similar data structures.

[0070] In at least one embodiment, a new FGRC-SEI message type is created to describe region-related film grain characteristics using a statistical model, wherein a set of model parameters is specified, each set being associated with a sample intensity interval and an image region. In such an embodiment, the syntax structure specifies a set of intensity intervals, the model parameters corresponding to each interval, and one or more regions corresponding to each interval.

[0071] In at least one embodiment, the new FGRC-SEI message defines a set of image regions and defines the film grain characteristics of each region. This may include intensity ranges and a list of range-specific model parameters, or model parameters of a parameterized function of intensity (e.g., a piecewise linear function for defining grain intensity).

[0072] In at least one embodiment, film grain characteristics are associated with image regions via label identifiers. In the implementation based on the aforementioned intervals, a label identifier is assigned to each intensity interval, and a set of label identifiers is assigned to each image region. Several intensity intervals may have the same label identifier. In the implementation based on the parameterized functions described above, a label identifier is assigned to each parameterized function, and a set of label identifiers is assigned to each image region.

[0073] In at least one embodiment, the syntax for the FGRC-SEI message, including a film grain region characteristic metadata structure, is defined as shown in Table 3. In this embodiment, common parameters, such as the selected model or color component, are specified. An interval label identifier named fgr_intensity_interval_tag is specified for each intensity interval. This interval label identifier can be the same for several intervals. The film grain metadata structure includes a list of regions. Each region is defined by a corresponding region represented using one of the different techniques described above (e.g., the top-left corner and size of the region). For each region, the list of interval label identifiers pertains to one of the intensity intervals, enabling the use of the corresponding film grain characteristics. When using this syntax, different regions can be defined, and each of these regions can use specific film grain parameters. The syntax in Table 3 does not include the aforementioned update mechanism.

[0074] Table 3

[0075] In the structure of Table 2, syntax elements with the same names as those in Table 2 have the same meaning. Furthermore, the syntax element `fgr_intensity_interval_tag[c][i]` specifies the identifier for the intensity interval `i` of color component `c`, and the syntax element `fgr_num_regions_minus1` plus 1 specifies the number of regions providing a specific set of parameters. The syntax element `fg_num_region_interval_tags_minus1[i]` plus 1 specifies the number of tag identifiers used for region `i`, and the syntax element `fgr_region_interval_tag[i][j]` specifies the identifier used for region `i`, which matches one of the identifiers in `fgr_intensity_interval_tag`: the film grain characteristics of region `i` include all intervals for which the same identifier is specified for all `j` in region `i`.

[0076] In the embodiments, the prefixes “fg” and “fgr” can be replaced by other prefixes, and the labeling of elements can vary when expressing the same function.

[0077] The syntax elements used for region definition can use any regular syntax elements used to describe regions. An example is given in Table 4 below, where a region is defined by its position using the syntax elements `fg_rect_region_position_top` and `fg_rect_region_position_left`, and by the size of the region using the syntax elements `fg_rect_region_width` and `fg_rect_region_height`.

[0078] Table 4

[0079] In a variant embodiment, instead of using interval label ids for each interval as shown in Table 3, each region is specified with an interval index range as shown in Table 5, and film grain characteristics corresponding to the specified interval are used for that region.

[0080] Table 5

[0081] In the variant embodiments described above, the region definition can be updated independently, as in the annotated region SEI message (using flags to indicate updates, specifying which region to update, and the new region definition or region modification, such as transformation vectors and / or size changes). In the variant embodiments described above, parameters describing the temporal evolution (transformation) of the region shape can also be inserted. In the variant embodiments described above, intensity intervals can be updated independently (using flags to indicate updates, specifying which intervals to update, and the new model parameters). Minor modifications to the syntax presented in Tables 3 and 4 will allow for the implementation of these variant embodiments.

[0082] The example described in this article is based on SEI messages to carry region-related film grain metadata, but any other means that provide the same functionality can be used.

[0083] Figure 9A An exemplary process for encoding a bitstream including an FGRC-SEI message, comprising region-related film grain metadata, is illustrated according to an embodiment. For example, by... Figure 1 The processor 1010 in device 1000 Figure 2 System 210 or Figure 3The encoder 300 implements process 910. In step 912, the processor acquires video data. In step 914, the processor determines the region-related film grain characteristics and region-related film grain metadata representing the region-related film grain characteristics. In step 916, the processor encapsulates the metadata into an FGRC-SEI message, for example, according to the syntax of Table 3 or 4. In step 918, the processor provides a bitstream including encoded video data and an FGRC-SEI message, the FGRC-SEI message including the region-related film grain characteristics.

[0084] Figure 9B An exemplary process for decoding a bitstream including an FGRC-SEI message, which includes region-related film grain metadata, is illustrated according to an embodiment. For example, by... Figure 1 Equipment 1000 Figure 2 System 230 or Figure 4 The decoder 400 implements process 930. In step 935, the processor obtains a bitstream including encoded video data and metadata representing region-related film grain characteristics, which is encapsulated in an FGRC-SEI message, for example, based on the syntax of Table 3 or 4. In step 950, the processor obtains the FGRC-SEI message and determines the metadata representing region-related film grain characteristics. In step 954, the processor determines film grain characteristics (e.g., parameters required for synthesizing film grain) for one or more regions based on the region-related film grain metadata and, for example, using intervals and region characteristics defined in the metadata. In step 990, the processor provides the determined film grain characteristics for further use.

[0085] Figure 9C An exemplary process is shown according to an embodiment for decoding a bitstream including FGRC-SEI messages and applying the determined film grain characteristics to the decoded image. Process 931 is established on... Figure 9B In process 930, and further proposed an application for the determined film grain characteristics. In this case, within the process ( Figure 9BStep 990) provides the determined characteristics and utilizes them directly in further steps. Steps 935, 950, and 954 are identical to the corresponding steps in process 930. In addition to these steps, in step 940, the processor decodes the image. In step 956, the processor generates a film grain sample for the region based on the extracted film grain characteristics. Steps 954 and 956 can be performed iteratively over multiple regions (e.g., on each region defined in the FGRC-SEI message) (arrow 958). In step 970, the processor adds the generated film grain sample to the decoded image according to the corresponding position and size of the region relative to the decoded image. In step 980, the processor provides an image including the film grain as specified in the FGRC-SEI message.

[0086] This application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as specific and are generally described in a manner that may sound restrictive, at least to illustrate individual characteristics. However, this is for the purpose of clarity and does not limit the application or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in prior documents.

[0087] The aspects described and contemplated in this application can be implemented in many different forms. The accompanying drawings provide some embodiments, but other embodiments are conceived, and the discussion of these drawings does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having a bitstream generated according to any of the described methods stored thereon.

[0088] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined.

[0089] The various methods and other aspects described in this application can be used to define modules, for example, as follows: Figure 3 and 4 The motion compensation and motion estimation modules of the video encoder 300 and decoder 400 shown are illustrated.

[0090] Various numerical values ​​are used in this application, such as 128 for a block size. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0091] Various implementations involve decoding. As used herein, "decoding" can include, for example, all or part of a process performed on a received encoded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also include, or alternatively include, processes performed by the decoders of the various implementations described herein, such as an adaptive lighting compensation process.

[0092] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. It will be clear from the context of the specific description whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to refer to a broader decoding process, and it is believed that those skilled in the art will understand this well.

[0093] Various implementations involve encoding. In a manner similar to the above discussion of “decoding,” “encoding,” as used herein, can include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also include, or alternatively include, processes performed by an encoder of the various implementations described herein.

[0094] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential and entropy encoding. It will be clear from the specific context of the description whether the phrase "encoding process" is intended to specifically refer to a subset of operations or to refer to a broader encoding process, and it is believed that those skilled in the art will understand this well.

[0095] Note that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0096] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / device.

[0097] Various embodiments involve rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, usually under constraints of computational complexity. Rate-distortion optimization is generally expressed as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. Different approaches exist to address the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, where a comprehensive evaluation of the encoding cost and the associated distortion of the reconstructed signal is performed after encoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on prediction or prediction of the residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion only for some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but the optimization is not necessarily a comprehensive evaluation of encoding cost and associated distortion.

[0098] This application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in considerable detail, and often in a manner that may sound restrictive, at least to illustrate the various characteristics. However, this is merely for clarity and does not constitute a limitation on these aspects or the scope of this application. In fact, all these different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in prior applications.

[0099] The implementations and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features in question can be implemented in other forms (e.g., devices or programs). Devices can be implemented, for example, with appropriate hardware, software, and firmware. Methods can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, tablets, smartphones, cellular phones, portable / personal digital assistants, and other devices that facilitate information communication between end users.

[0100] The references to "an embodiment" or "an embodiment" or "an implementation" or "an implementation," and their variations, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation," and any other variations appearing throughout this application, do not necessarily refer to the same embodiment.

[0101] Additionally, this application may involve "determining" various types of information. Determining information may include one or more of the following: for example, estimation information, calculation information, prediction information, and information retrieved from memory.

[0102] Furthermore, this application may involve "accessing" various types of information. Accessing information may include one or more of the following: for example, receiving information, retrieving information (e.g., retrieving information from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, and estimating information.

[0103] Additionally, this application may relate to "receiving" various types of information. Like "accessing," receiving is a broad term. Receiving information may include one or more of the following: for example, accessing information and retrieving information (e.g., retrieving information from memory). Furthermore, "receiving" is generally referred to in one or more ways during operation, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0104] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," "frame," "slice," and "tile." Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0105] It should be understood that the use of any of the following “ / ”, “and / or”, and “…at least one of” (e.g., in the cases of “A / B”, “A and / or B”, and “at least one of A and B”) is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As yet another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, this wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to a large number of listed items.

[0106] Furthermore, as used herein, the term "signaling" specifically refers to instructing the corresponding decoder to provide certain information. For example, in some embodiments, the encoder signals a specific one of the illumination compensation parameters, etc. In this way, the same parameter is used on both the encoder and decoder sides in the embodiments. Thus, for example, the encoder can send (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the word "signal" was mentioned above, the word "signal" can also be used as a noun herein.

[0107] It will be apparent to those skilled in the art that implementations can generate various signals that are formatted to carry, for example, information that can be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0108] We have described numerous embodiments. Features of these embodiments may be provided individually or in any combination across various claim classes and types.

Claims

1. A method comprising: Obtain video data; Determine the region-specific film grain metadata that represents the characteristics of film grain; The region-related film grain metadata is inserted into an encoded bitstream that includes the video data and the region-related film grain metadata; as well as Provide the bit stream.

2. A method comprising: Obtain a bitstream including encoded video data and region-related film grain metadata; Based on the region-related film grain metadata for the region, determine the film grain characteristics of one or more regions of the video frame; as well as Provides the determined film grain characteristics.

3. The method according to claim 2, further comprising: Decode the image based on the encoded video data; Based on the determined characteristics of the film grain, film grain samples for the specific region are determined. Add the film grain sample to the decoded image; as well as Provides the modified, decoded screen.

4. The method according to any one of claims 1 to 3, wherein the region-related film grain metadata is carried by a supplemental enhancement information message.

5. The method according to any one of claims 1 to 4, wherein the region-related film grain metadata includes information related to a list of regions, wherein for each region in the list of regions, the region-related film grain metadata includes information related to the location and size of the region.

6. The method according to any one of claims 1 to 5, wherein the region-related film grain metadata includes information related to the interval list, and wherein the region is associated with one or more intervals.

7. The method of claim 6, wherein the association is performed by associating intervals with characteristic identifiers and associating one or more regions with one or more characteristic identifiers.

8. An apparatus comprising one or more processors, said one or more processors being configured to: Obtain video data; Determine the region-specific film grain metadata that represents the characteristics of film grain; Insert the region-related film grain metadata into an encoded bitstream that includes the video data and the region-related film grain metadata; and Provide the bit stream.

9. An apparatus comprising one or more processors, said one or more processors being configured to: Obtain a bitstream including encoded video data and region-related film grain metadata; Based on the region-specific film grain metadata, determine the film grain characteristics of one or more regions of the video frame; and Provides the determined film grain characteristics.

10. The apparatus of claim 9, wherein the one or more processors are further configured to: Decode the image based on the encoded video data; Based on the determined characteristics of the film grain, film grain samples for the specific region are determined. Add the film grain sample to the decoded image; as well as Provides the modified, decoded screen.

11. The apparatus according to any one of claims 8 to 10, wherein the region-related film grain metadata is carried by a supplemental enhancement information message.

12. The apparatus of any one of claims 8 to 11, wherein the region-related film grain metadata includes information related to a list of regions, wherein for each region in the list of regions, the region-related film grain metadata includes information related to the location and size of the region.

13. The apparatus of any one of claims 8 to 12, wherein the region-related film grain metadata includes information related to a list of intervals, and wherein a region is associated with one or more intervals.

14. A non-transitory computer-readable medium comprising data content generated by the method according to any one of claims 1 to 7.

15. A computer program product comprising instructions for performing the method of any one of claims 1 to 7 when executed by one or more processors.

Images