SEI messages for film particle composite mask

Abstract

A method and apparatus including computer code configured to cause one or more processors to obtain video data including at least one encoded picture; reconstructing an encoded picture in the video data associated with a film particle characteristic SEI message, the first sample, the second sample, the third alpha-mapped sample, and the fourth alpha-mapped sample, where the first sample and the third alpha-mapped sample and the second sample and the fourth sample are spatially co-located after the reconstruction; applying film particle synthesis to the first sample based on the first value of the third sample; and determining that film particle synthesis is not applied to the second sample based on a second value of the fourth sample, where the first value and the second value are different from each other.

Description

Cross - Reference to Related Applications

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 540,559, filed on September 26, 2023, and U.S. Application No. 18 / 895,486, filed on September 25, 2024, the disclosures of which are hereby incorporated by reference in their entireties. Background Art 1. Technical Field

[0002] The disclosed subject matter relates to video encoding and decoding, and more particularly, to the application of film grain or similar noise to a portion of a picture controlled by metadata including an alpha map or a depth map. 2. Description of Related Art

[0003] Video encoding and decoding using inter - picture prediction with motion compensation has been known for decades. Uncompressed digital video can include a sequence of pictures, each picture having spatial dimensions such as 1920×1080 luminance samples and associated chrominance samples. The sequence of pictures can have a fixed or variable picture rate (also informally referred to as frame rate), e.g., 60 pictures per second or 60 Hz. Uncompressed video has significant bit - rate requirements. For example, 1080p60 4:2:0 video (1920×1080 luminance sample resolution at 60 Hz frame rate) at 8 bits per sample requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 gigabytes (GByte) of storage space.

[0004] One goal of video encoding and decoding can be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage - space requirements, in some cases by two orders of magnitude or more. Both lossless compression and lossy compression can be employed, as well as combinations thereof. Lossless compression refers to techniques by which an exact copy of the original signal can be reconstructed from the compressed signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough such that the reconstructed signal is useful for the intended application. In the case of video, lossy compression is widely employed. The amount of distortion tolerated depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that higher allowable / tolerable distortion can result in a higher compression ratio.

[0005] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transformation, quantization, and entropy coding, some of which will be described below.

[0006] Film Grain Synthesis (also known as FGS) is a tool aimed at preserving the impression of original film grain, mainly for content shot using chemical film (compared to digital imaging devices) in a digital compression video environment. The digitized input material can be pre-filtered, which can remove the film grain but helps subsequent encoding steps achieve better compression efficiency compared to input material including noise (such as film grain). Artificially approximated film grain can be re-inserted after reconstruction. The amount and characteristics of the noise can be part of the video bitstream in the form of metadata. For example, ITU-T Recommendation H.274 includes Film Grain Characteristics SEI messages.

[0007] ITU H.274 Recommendation also includes an Alpha Channel information SEI message, which can be used to define the application of alpha channel information that can exist in, for example, an H.266 bitstream. Summary of the Invention

[0008] Includes a method and an apparatus, the apparatus comprising: a memory configured to store computer program code; and one or more processors configured to access the computer program code and operate in accordance with what the computer program code indicates. The computer program is configured to cause the processor to implement: obtaining code configured to cause at least one processor to obtain video data including at least one encoded picture; reconstruction code configured to cause at least one processor to reconstruct the encoded picture in the video data associated with the Film Grain Characteristics SEI message, the first sample, the second sample, the third alpha mapping sample, and the fourth alpha mapping sample, wherein, after reconstruction, the first sample and the third alpha mapping sample and the second sample and the first sample are spatially co-located; application code configured to cause at least one processor to apply Film Grain Synthesis to the first sample based on a first value of the third sample; and determination code configured to cause at least one processor to determine not to apply Film Grain Synthesis to the second sample based on a second value of the fourth sample, wherein the first value and the second value are different from each other.

[0009] The encoded picture can also be associated in the video data with an Alpha Channel information SEI message indicating the third alpha mapping sample and the fourth alpha mapping sample.

[0010] The Alpha Channel information SEI message can indicate the alpha channel sample values encoded in an auxiliary picture of the AUX_ALPHA type and the post-processing applied to decode the alpha plane and one or more associated primary pictures.

[0011] The alpha channel information SEI message can have a predetermined value that indicates that the analog film grain value at the same location and color component sample is to be multiplied by the decoded auxiliary picture's decoded sample value before calculating the film grain value.

[0012] This value can be the integer 3.

[0013] The film grain characteristic SEI message can identify the blending mode used to blend the analog film grain with the input image.

[0014] The grain characteristic SEI message can also identify the weight applied to the film grain synthesis of the first sample. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which:

[0016] Figure 1 is a schematic illustration of a computer environment according to an embodiment.

[0017] Figure 2 is a simplified block diagram of media processing according to an embodiment;

[0018] Figure 3 is a simplified illustration of decoding according to an embodiment;

[0019] Figure 4 is a simplified illustration of encoding according to an embodiment;

[0020] Figure 5 is a simplified illustration of an NAL unit and an SEI header according to an embodiment;

[0021] Figure 6 is a schematic illustration of a video scene according to an embodiment, the video scene including movie content that requires film grain synthesis and computer-generated content that does not require film grain synthesis;

[0022] Figure 7 is a schematic illustration of a modified film grain characteristic SEI message according to an embodiment;

[0023] Figure 8 is a schematic illustration of a modified alpha channel information SEI message according to an embodiment; and

[0024] Figure 9 is a schematic illustration of a computer system according to an embodiment. DETAILED DESCRIPTION

[0025] Techniques for applying film grain synthesis on a per-sample basis using SEI messages are disclosed.

[0026] The proposed features discussed below can be used alone or in any combination in any order. Additionally, embodiments can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0027] Today's video signals typically consist of multiple sources. For example, a movie with narration can consist of the movie content itself and surrounding frames with narration. The movie may have been shot using chemical film technology and may thus include film grain after digitization. To achieve reasonable compression, the film grain can be removed by pre-filtering. On the other hand, the surrounding frames are created digitally and do not include film grain. After encoding, transmission, and reconstruction, the resulting reconstructed picture does not include the noise associated with the film grain. This is preferable for the surrounding frames but not for the movie, as the film grain here is removed by pre-filtering. Therefore, a technique is needed in which the film grain can be selectively inserted into various parts of the reconstructed picture while leaving the unselected parts of the picture free of film grain. More complex scenarios may involve including multiple movies with different film grain characteristics in a synthetic picture. In such scenarios, different film grain reinsertion parameters can be advantageously applied to different parts of the reconstructed picture.

[0028] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. The communication system 100 can include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional transmission of data, a first terminal 103 can encode video data at a local location for transmission via the network 105 to another terminal 102. The second terminal 102 can receive the encoded video data of the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in media service applications and the like.

[0029] Figure 1 A second pair of terminals 101 and 104 is shown, which are arranged to support two-way transmission of encoded video that may occur, for example, during a video conference. For two-way transmission of data, each terminal 101 and 104 can encode video data captured at a local location for transmission via the network 105 to the other terminal. Each terminal 101 and 104 can also receive the encoded video data sent by the other terminal, can decode the encoded data, and can display the recovered video data at a local display device.

[0030] In Figure 1Among them, the terminals 101, 102, 103, and 104 may be shown as servers, personal computers, and smart phones, but the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. The network 105 represents any number of networks that transmit encoded video data between the terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. The communication network 105 may exchange data in circuit-switched channels and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, unless otherwise stated below, the architecture and topology of the network 105 may be immaterial to the operation of the present disclosure. The network 105 may include a Media Aware Network Element (MANE), which may be included, for example, in the transmission path between the terminal 101 and the terminal 104. The purpose of the MANE may be to selectively forward portions of the media data in response to network congestion, media switching, media mixing, archiving, and similar tasks typically performed by service providers rather than end users. Such a MANE may be able to parse and respond to a limited portion of the media transmitted through the network (e.g., syntax elements related to video coding techniques or the network abstraction layer of a standard).

[0031] Figure 2 The placement of a video encoder and a video decoder in a streaming environment is shown as an example of an application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0032] A streaming system may include a capture subsystem 203, which may include a video source 201, such as a digital camera device, that creates a video sample stream 213, such as an uncompressed video sample stream. The sample stream 213 may be characterized as having a high data volume when compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the video source 201, which may be, for example, a camera device as discussed above. The encoder 202 may include hardware, software, or a combination thereof to implement or enforce aspects of the disclosed subject matter described in more detail below. The encoded video bitstream 204 may be characterized as having a lower data volume when compared to the sample stream and may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video bitstream 204. The client 212 may include a video decoder 211 that decodes an incoming copy of the encoded video bitstream 208 and creates an outgoing video sample stream 210 that may be rendered on a display 209 or other rendering device (not shown). In some streaming systems, the video bitstreams 204, 206, and 208 may be encoded according to certain video coding / compression standards. Examples of such standards were mentioned above and are further described herein. Examples of such standards include ITU-T Recommendations H.265 and H.266. The disclosed subject matter may be used in the context of VVC.

[0033] Figure 3 May be a functional block diagram of a video decoder 300 according to an embodiment of the present invention.

[0034] The receiver 302 may receive one or more codec video sequences to be decoded by the decoder 300; in the same or another embodiment, one encoded video sequence is received at a time, where the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequences may be received from the channel 301, which may be a hardware / software link to a storage device storing the encoded video data. The receiver 302 may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective using entities (not depicted). The receiver 302 may separate the encoded video sequences from the other data. To prevent network jitter, a buffer memory 303 may be coupled between the receiver 302 and the entropy decoder / parser 304 (hereinafter referred to as "parser"). When the receiver 302 is receiving data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer 303 may not be required or the buffer 303 may be small. For use on a best-effort packet network such as the Internet, the buffer 303 may be required, and the buffer 303 may be relatively large and may advantageously have an adaptive size.

[0035] Video decoder 300 may include a parser 304 to reconstruct symbols 313 from an entropy-coded video sequence. The categories of these symbols include: information for managing the operation of decoder 300; and information potentially for controlling a rendering device such as display 312, which is not part of the decoder but may be coupled to the decoder. The control information for the rendering device may be in the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set segments (not depicted). Parser 304 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be according to a video coding technology or standard and may follow principles well-known to those skilled in the art, including: variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 304 may extract subgroup parameter sets for at least one subgroup of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to a group. The subgroups may include Group of Picture (GOP), picture, tile, slice, macroblock, Coding Unit (CU), block, Transform Unit (TU), Prediction Unit (PU), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0036] Parser 304 may perform entropy decoding / parsing operations on the video sequence received from buffer 303 to create symbols 313. Parser 304 may receive the encoded data and selectively decode specific symbols 313. Additionally, parser 304 may determine whether to provide a specific symbol 313 to motion compensation prediction unit 306, scaler / inverse transform unit 305, intra prediction unit 307, or loop filter 311.

[0037] Depending on the type of the encoded video picture or its part (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of symbols 313 may involve multiple different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by parser 304 from the encoded video sequence. For the sake of brevity, this subgroup control information flow between parser 304 and the multiple units below is not depicted.

[0038] In addition to the functional blocks already mentioned, decoder 300 can conceptually be subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the following functional units.

[0039] The first unit is scaler / inverse transform unit 305. Scaler / inverse transform unit 305 receives, from parser 304, quantized transform coefficients as symbols 313 and control information, including which transform to use, block size, quantization factor, quantization scaling matrix, etc. Scaler / inverse transform unit 305 can output a block including sample values, which can be input into aggregator 310.

[0040] In some cases, the output samples of scaler / inverse transform 305 can belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but can use predictive information from a previously reconstructed portion of the current picture. Such predictive information can be provided by intra picture prediction unit 307. In some cases, intra picture prediction unit 307 uses the surrounding reconstructed information extracted from the current (partially reconstructed) picture 309 to generate a block of the same size and shape as the block being reconstructed. In some cases, aggregator 310 adds, on a per-sample basis, the predictive information already generated by intra prediction unit 307 to the output sample information as provided by scaler / inverse transform unit 305.

[0041] In other cases, the output samples of scaler / inverse transform unit 305 can belong to an inter-coded and possibly motion-compensated block. In such a case, motion compensation prediction unit 306 can access reference picture memory 308 to extract samples for prediction. After motion compensating the extracted samples according to the symbols 313 belonging to the block, these samples can be added by aggregator 310 to the output of scaler / inverse transform unit (referred to as residual samples or residual signal in this case) to generate output sample information. The address within the reference picture memory from which the motion compensation unit extracts the prediction samples can be controlled by a motion vector, which is available to the motion compensation unit in the form of symbols 313, which can have, for example, an X component, a Y component, and a reference picture component. Motion compensation can also include interpolation of sample values extracted from the reference picture memory when using sub-sampled accurate motion vectors, motion vector prediction mechanisms, etc.

[0042] The output samples of aggregator 310 can undergo various loop filtering techniques in loop filter unit 311. Video compression techniques can include in-loop filter techniques that are controlled by parameters included in the encoded video bitstream and available to loop filter unit 311 as symbols 313 from parser 304, but video compression techniques can also respond to meta-information obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0043] The output of loop filter unit 311 can be a sample stream that can be output to rendering device 312 and stored in reference picture memory 557 for future inter-picture prediction.

[0044] Once fully reconstructed, some encoded pictures can be used as reference pictures for future prediction. Once an encoded picture has been fully reconstructed and the encoded picture has been identified as a reference picture (e.g., by parser 304), the current reference picture 309 can become part of reference picture buffer 308, and a new current picture memory can be reallocated before starting to reconstruct subsequent encoded pictures.

[0045] Video decoder 300 can perform decoding operations according to predetermined video compression techniques that can be documented in standards such as ITU-T Recommendation H.266. In the sense that the encoded video sequence follows the syntax of the video compression technique or standard, the encoded video sequence can conform to the syntax specified by the video compression technique or standard being used, as specified in the video compression technique document or standard and explicitly in the profile document therein. For compliance, it is also required that the complexity of the encoded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (measured, for example, in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level can be further restricted by the Hypothetical Reference Decoder (HRD) specification and the metadata for HRD buffer management signaled in the encoded video sequence.

[0046] In an embodiment, the receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the (one or more) encoded video sequences. The additional data may be used by the video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

[0047] Figure 4 may be a functional block diagram of a video encoder 400 according to an embodiment of the present disclosure.

[0048] The encoder 400 may receive video samples from a video source 401 (which is not part of the encoder), and the video source 401 may capture video images to be encoded by the encoder 400.

[0049] The video source 401 may provide a source video sequence in the form of a stream of digital video samples to be encoded by the encoder (303), and the stream of digital video samples may have any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits,...), any color space (e.g., BT.601 Y CrCb, RGB,...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source 401 may be a storage device storing previously prepared videos. In a video conferencing system, the video source 401 may be a camera device that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that are given motion when viewed in sequence. The pictures themselves may be organized as a spatial pixel array, where each pixel may include one or more samples depending on the sampling structure, color space, etc. used. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0050] According to an embodiment, the encoder 400 may encode and compress pictures of the source video sequence into an encoded video sequence 410 in real time or according to any other time constraint required by the application. Implementing an appropriate encoding speed is a function of the controller 402. The controller controls the other functional units as described below and is functionally coupled to these units. For clarity, the couplings are not depicted. The parameters set by the controller may include: rate control related parameters (picture skip, quantizer, λ value of rate distortion optimization techniques,...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller 402 because these functions may belong to the video encoder 400 optimized for a specific system design.

[0051] Some video encoders operate in a manner that is readily recognizable to those skilled in the art as an "encoding loop". As a gross simplification, the encoding loop can include: an encoding portion of encoder 400 (hereinafter "source encoder") (responsible for creating symbols based on input pictures and reference pictures to be encoded); and an (local) decoder 406 embedded in encoder 400, where the (local) decoder 406 reconstructs the symbols to create sample data that the (remote) decoder will also create (since in the video compression techniques contemplated in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). This reconstructed sample stream is input to reference picture memory 405. Since decoding the symbol stream results in a bit-exact result independent of the decoder location (local or remote), the reference picture buffer content is also bit-exact between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same as the sample values that the decoder will "see" when using prediction during decoding. The basic principle of this reference picture synchronization (and the resulting drift if synchronization cannot be maintained, for example due to channel errors) is well known to those skilled in the art.

[0052] The operation of the "local" decoder 406 can be the same as that of the "remote" decoder 300 already described in detail above. However, also briefly referring to Figure 3 When symbols are available and the entropy encoder 408 and parser 304 can encode / decode the symbols losslessly into an encoded video sequence, the entropy decoding portion of decoder 300 including channel 301, receiver 302, buffer 303, and parser 304 may not be fully implemented in local decoder 406. Figure 4

[0053] It can be observed at this point that any decoder technology other than the parsing / entropy decoding present in the decoder must also necessarily exist in the corresponding encoder in substantially the same functional form. Since the encoder technology is reciprocal to the decoder technology that has been fully described, the description of the encoder technology can be simplified. More detailed descriptions will only be provided and are needed in certain places below.

[0054] As part of its operation, source encoder 403 can perform motion-compensated predictive coding that predicts an input frame by referring to one or more previously encoded frames in the video sequence designated as "reference frames". In this way, encoding engine 407 encodes the difference between a pixel block of the input frame and a pixel block of the reference frame, where the reference frame can be selected as the prediction reference for the input frame.

[0055] ​The local video decoder 406 may decode the encoded video data of the frames that may be designated as reference frames based on the symbols created by the source encoder 403. The operation of the encoding engine 407 may advantageously be lossy processing. When the encoded video data may be decoded at the video decoder ( Figure 4 not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder 406 replicates the decoding process that may be performed by the video decoder on the reference frames and may cause the reconstructed reference frames to be stored in the reference picture memory 405 that may be, for example, a cache memory. In this way, the encoder 400 may locally store a copy of the reconstructed reference frames, which has common content (in the absence of transmission errors) with the reconstructed reference frames to be obtained by the remote video decoder.

[0056] The predictor 404 may perform a prediction search for the encoding engine 407. That is, for a new frame to be encoded, the predictor 404 may search the reference picture memory 405 for sample data (as candidate reference pixel blocks) or some metadata such as reference picture motion vectors, block shapes, etc. that may be used as an appropriate prediction reference for the new picture. The predictor 404 may operate on a per-pixel block basis of the sample blocks to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor 404, the input picture may have prediction references extracted from multiple reference pictures stored in the reference picture memory 405.

[0057] The controller 402 may manage the encoding operations of the source encoder 403 that may be, for example, a video encoder, including, for example, setting parameters and subgroup parameters for encoding the video data.

[0058] The outputs of all the above-mentioned functional units may undergo entropy encoding in the entropy encoder 408. The entropy encoder converts these symbols into an encoded video sequence by losslessly compressing the symbols generated by the various functional units according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0059] The transmitter 409 may buffer the encoded video sequence(s) created by the entropy encoder 408 in preparation for transmission via the communication channel 411, which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter 409 may merge the encoded video data from the source encoder 403 with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0060] The controller 402 may manage the operation of the encoder 400. During encoding, the controller 402 may assign a certain type of encoded picture to each encoded picture, which may affect the encoding techniques that can be applied to the corresponding picture. For example, pictures can generally be assigned to one of the following frame types:

[0061] Intra pictures (I pictures), which may be pictures that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including for example independently decoded refresh pictures. Those skilled in the art are aware of these variations of I pictures and their corresponding applications and characteristics.

[0062] Predictive pictures (P pictures), which may be pictures that can be encoded and decoded using intra prediction or inter prediction that can use at most one motion vector and a reference index to predict the sample values of each block.

[0063] Bi - predictive pictures (B pictures), which may be pictures that can be encoded and decoded using intra prediction or inter prediction that can use at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0064] Source pictures can generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples respectively), and encoded on a block - by - block basis. These blocks can be predictively encoded with reference to other (encoded) blocks, which are determined by the encoding assignment applied to the corresponding picture of the block. For example, blocks of an I picture can be non - predictively encoded, or can be predictively encoded (spatial prediction or intra prediction) with reference to encoded blocks of the same picture. Pixel blocks of a P picture can be non - predictively encoded with reference to one previously encoded reference picture via spatial prediction or via temporal prediction. Blocks of a B picture can be non - predictively encoded with reference to one or two previously encoded reference pictures via spatial prediction or via temporal prediction.

[0065] The encoder 400, which may be for example a video encoder, may perform encoding operations according to a predetermined video coding technique or standard such as the ITU - T H.266 recommendation. In the operation of the encoder 400, the encoder 400 may perform various compression operations, including predictive coding operations that utilize temporal redundancy and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video coding technique or standard used.

[0066] In an embodiment, the transmitter 409 may transmit additional data along with the encoded video. The source encoder 403 may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplementary enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, and the like.

[0067] The compressed video may be enhanced in the video bitstream by supplementary enhancement information, for example, in the form of supplementary enhancement information (SEI) messages or video usability information (VUI). The video coding standard may include a normative part for SEI and VUI. The SEI and VUI information may also be specified in a separate specification that may be referenced by the video coding specification.

[0068] Referring to Figure 5 Example 500 of, shows an exemplary layout of a coded video sequence (CVS) according to H.266. The coded video sequence is subdivided into network abstraction layer units (NAL units). An exemplary NAL unit 501 may include a NAL unit header 502, which in turn includes the following 16 bits: The forbidden_zero_bit 503 and the nuh_reserved_zero_bit 504 may not be used by H.266 and may be zero in the NAL unit. Conforming to H.266. The three bits of the nuh_layer_id 505 may indicate the layer (spatial, SNR, or multi-view enhancement) to which the NAL unit belongs. The five bits of the nuh_nal_unit_type define the type of the NAL unit. In H.266, 22 NAL unit type values are defined for the NAL unit types defined in H.266, 6 NAL unit types are reserved, and 4 NAL unit type values are unspecified and may be used by specifications other than H.266. Finally, the three bits nuh_temporal_id_plus1 506 of the NAL unit header indicate the temporal layer to which the NAL unit belongs.

[0069] The coded picture may contain one or more Video Coding Layer (VCL) NAL units and zero or more non-VCL NAL units. The VCL NAL units may contain coded data that conceptually belongs to the video coding layer as described above. The non-VCL NAL units may contain data that conceptually does not belong to the video coding layer. Using H.266 as an example, as shown below, they can be classified into (1) parameter sets, (2) picture headers (PH_NUT), (3) NAL units, (4) prefix and suffix SEI NAL unit types (PREFIX_SEI_NUT and SUFFIX_SEI_NUT), (5) filler data NAL unit type FD_NUT, and (6) reserved and undefined NAL unit types.

[0070] (1) Parameter sets, which include information that may be necessary for the decoding process and can be applied to more than one coded picture. Parameter sets and conceptually similar NAL units may be NAL unit types such as DCI_NUT (Decoding Capability Information (DCI)), VPS_NUT (Video Parameter Set (VPS), establishing layer relationships, etc.), SPS_NUT (Sequence Parameter Set (SPS), establishing parameters used and kept constant in the entire Coded Video Sequence (CVS), etc.), PPS_NUT (Picture Parameter Set (PPS), establishing parameters used and kept constant within the coded picture, etc.), and PREFIX_APS_NUT and SUFFIX_APS_NUT (prefix and suffix adaptation parameter sets). The parameter sets may include information required by the decoder to decode the VCL NAL units and are thus referred to as "normative" NAL units here.

[0071] (2) Picture headers (PH_NUT), which are also "normative" NAL units.

[0072] (3) NAL units that mark certain positions in the NAL unit stream. These include NAL units with NAL unit types AUD_NUT (Access Unit Delimiter), EOS_NUT (End of Sequence), and EOB_NUT (End of Bitstream). These are non-normative and also called informative in the sense that a compliant decoder does not require them for its decoding process, although it needs to be able to receive them in the NAL unit stream.

[0073] (4) PREFIX_SEI_NUT and SUFFIX_SEI_NUT, which indicate NAL units containing prefix and suffix supplementary enhancement information. In H.266, these NAL units are informative as they are not required for the decoding process.

[0074] (5) The FD_NUT of the filler data NAL unit type indicates filler data; it can be random and can be used to "waste" bits in the NAL unit stream or bitstream, which may be necessary for transmission over certain synchronous transmission environments.

[0075] (6) Reserved and undefined NAL unit types.

[0076] Still referring to Figure 5 , the layout of the NAL unit stream of decoding order 510 is shown. The NAL unit stream contains the encoded picture 511, and the encoded picture 511 contains some of the previously introduced types of NAL units. Somewhere early in the NAL unit stream, the DCI 512, VPS 513, and SPS 514 can jointly establish the parameters that the decoder can use to decode the encoded pictures (including the encoded picture 511 of the NAL unit stream) of the encoded video sequence (CVS).

[0077] The encoded picture 511 can contain, in the depicted order or any other order compliant with the video coding technology or standard in use (here: H.266): prefix APS 516, Picture Header (PH) 517, prefix SEI 518, one or more VCL NAL units 519, and suffix SEI 520.

[0078] The prefix and suffix SEI NAL units 518 and 520 are triggered during standard development because for some SEI messages, the content of the message will be known before the start of encoding of a given picture, while other content will only be known after the picture has been encoded. Allowing certain SEI messages to appear earlier or later in the NAL unit stream of the encoded picture via the prefix and suffix SEIs allows buffering to be avoided. As an example, in an encoder, the sampling time of the picture to be encoded is known before the picture is encoded, and thus the picture timing SEI message can be a prefix SEI message 516. On the other hand, the decoded picture hash SEI message (which contains the hash of the sample values of the decoded picture and can be used for, e.g., debugging the encoder implementation) is a suffix SEI message 518 because the encoder cannot compute the hash of the reconstructed samples before the picture has been encoded. The positions of the prefix and suffix SEI NAL units are not limited to their positioning in the NAL unit stream. The phrases "prefix" and "suffix" can imply what encoded picture or NAL unit the prefix / suffix SEI message can belong to, and details of such applicability can be specified, e.g., in the semantic description of a given SEI message.

[0079] Still referring to Figure 5 , a simplified syntax diagram of a NAL unit containing a prefix or suffix SEI message 520 is shown. This syntax is a container format for multiple SEI messages that can be carried in one NAL unit. For clarity, the details of the emulation prevention syntax specified in H.266 are omitted here. As with other NAL units, the SEI NAL unit starts with a NAL unit header 521. After the header are one or more SEI messages; two messages 530, 531 are depicted and described below. Each SEI message within the SEI NAL unit includes: an 8-bit payload_type_byte 522 that specifies one of 256 different SEI types; an 8-bit payload_size_byte 523 that specifies the number of bytes of the SEI payload; and the number of bytes of the payload_size-byte payload 524. This structure can be repeated until the payload_type_byte is observed to be equal to 0xff, which indicates the end of the NAL unit. The syntax of the payload 524 depends on the SEI message and can be any length between 0 bytes and 255 bytes.

[0080] Referring to Figure 6Example 600 shows a still image cropped from a black-and-white movie shot on chemical film 601. The light conditions in that particular shot were unfavorable (night shot), so many film grains could be observed when viewing the original movie content, and today, it is desirable to reproduce that film grain because it is directly associated with the original content and might even be part of the original artistic expression. In contrast, a computer screenshot 602 relies on clear, noise-free information to represent, for example, small letters on the right sidebar. In this use case, it may be acceptable that the potential noisy mountain view in the background is not faithfully represented.

[0081] Now consider the mixing of two contents in the same screen layout 603 and possibly in the same video stream. Some parts of the content benefit from film grain, while other parts (such as the letters on the right sidebar) do not.

[0082] This is shown in screenshot 604 without distracting details. The movie image 605 benefits from the reapplication of film grain, while the background 606 and especially areas of high-spatial computer-generated details such as string 607 do not benefit from film grain. Note that the screen layout may also include a window labeled "Alarm" 608 that partially obscures the displayed movie 605. The window is depicted as a rectangle, but it can be any shape in modern systems. It is shown here to emphasize that indicating the movie area 605 by a rectangle encoded as metadata (e.g., by its corner coordinates) may not be sufficient to identify the screen areas where film grain processing is desired. Those skilled in the art can easily observe other cases where it is necessary to identify non-rectangular (or non-uniform) areas that may require film grain processing. For example, consider a scenario that shows a TV playing a movie in a room, where the TV is at an angle to the viewer's parallax. A sample-by-sample solution for indicating film grain is desired.

[0083] Using a per-sample mask to identify film grain can also be used for, for example: ● For artistic intent (focusing on a person's face), to invite the viewer to pay more attention to a specific area in the picture (producers often use a narrow depth of field to blur the background) ● To restore texture, such as the grass on a football field, which is especially useful when frame interpolation is used in a TV (film grain will maintain the perception of details) ● To provide a sense of realism only on the face / skin in a video conferencing scenario (if grain is applied across the whole image, it will look like an old-style video: this can also be a problem if there is screen content in it) ● In movie production, synthetic objects from special effects may have too soft textures. In such cases, this film grain mask will be used as a post - production tool, such as a plugin for Apple Final Cut Pro or DaVinci Resolve. ● Video games have cinematographic animations that would benefit from the local application of film grain. ● Control exposure and contrast by applying higher film grain to shadows and highlights. ● Limit the perception of visual artifacts in pictures.

[0084] Brief reference Figure 2 Consider an application scenario where in the sending system 203, the content source 201 is not an actual imaging device but an image synthesizer that puts together a video sequence 202 covering a scene equivalent to the synthetic scene 603 shown. In other words, the signal presented to the encoder 202 is a synthetic scene where some parts benefit from denoising and film grain identification and encoding in the form of metadata such as SEI messages, while other parts do not. Such an encoded stream can be transmitted 204, 208 potentially through a server 205 to the receiving system 212 and the decoder 211 located therein. The receiving system can decode the metadata identifying the samples that require film grain processing and apply it to the reconstructed bitstream 210 through a post - processing step.

[0085] Summarizing the above use - case discussion, what is needed is: 1. A mechanism for identifying one or more samples of a reconstructed encoded picture to which film grain processing should be applied to obtain an optimal user experience. Such samples can be represented by rectangular regions but can also have more complex shapes. Additionally, a) it may make sense to apply different film grain properties to different samples, and / or b) apply different intensities of film grain to different samples. 2. A mechanism for describing film grain characteristics. 3. A mechanism for combining the two mechanisms described above, where the sample selection / intensity mechanism is applicable to film grain.

[0086] Re 1: Sample selection mechanisms are readily available in video coding standards and related metadata standards. As an example, an "annotation area" SEI may be available, which may allow an area of a picture identified by its four corners to be annotated with metadata that can be processed by a renderer. Another option may be to use an alpha map. An alpha map may be a reconstructed picture with a single plane, which in the current case may be used as a boolean or integer value for each corresponding sample of the reconstructed picture. The alpha planes may have the same spatial dimensions as the reconstructed picture, or they may have different sizes, in which case they may be enlarged / shrunk to map each sample of the reconstructed picture to a sample (or filtered composite of samples) of the alpha plane.

[0087] Re 2: The VSEI specifies a film grain SEI message, and similar information may be available in the VUI of the video coding specification.

[0088] Re 3: Currently, no such binding is available in video coding or metadata specifications.

[0089] In an embodiment, the alpha map is used to indicate whether film grain processing should be applied to spatially corresponding samples of the reconstructed picture. Assuming that the sample values of the alpha plane are 10-bit integers with a value range of 0 to 1023 (as is the case when using H.266), alpha plane values below 512 may indicate that film grain synthesis should not be applied, while values above 511 may indicate that film grain synthesis should be applied. For a scene such as a scene associated with picture layout 608, such an alpha map can be very well compressed (possibly compressed to a few bytes).

[0090] In an embodiment, the alpha map may follow similar design principles as described above, but may include integer values instead of integer-encoded boolean values. For each sample, the value of the integer may indicate the intensity of the application of film grain.

[0091] Referring to Figure 7 Example 700 of, to identify the alpha map as a mechanism indicating the use or intensity of film grain per sample, the H.274 existing Alpha Channel Information (ACI) SEI message may be modified as follows: 1. The syntax remains unchanged 2. In the semantics of the film grain synthesis information SEI, insert a paragraph modifying FGS mixing based on alpha channel information as Figure 7 shown.. 3. In the semantics of the alpha channel information SEI, insert as Figure 8The paragraph shown is to introduce the previously reserved value "3" and associate it with the use of the alpha map for film grain synthesis.

[0092] Referring to Figure 7 , an excerpt of the semantic definition of the film grain characteristic SEI message modified according to an embodiment is shown. The modification is shown by the underlined text. For clarity, some parts of the semantics are deleted and are represented by "[...]" 702. For most parts, the semantics remain unchanged 703. However, if the current picture unit (PU) contains an alpha channel control indication with the (newly defined) value "3" 704, then the blending formula changes such that the alpha channel sample value Iaux[][][] 705 is considered in (either addition or multiplication 706, 707).

[0093] In the same or another embodiment, referring to Figure 8 Example 800 of, an alpha channel control SEI 801 is shown, with some parts omitted for clarity 802. The underlined text indicates the content added / modified with respect to the published H.264 specification.

[0094] A new value "3" is introduced for alpha_channel_use_idc. Its semantics are that when found, the sample values in the reconstructed auxiliary picture (in this case the standard term for the alpha map) are used for film grain synthesis. Specifically, when a value equal to 3 is found, this can indicate that the analog film grain value at the same location and color component sample should be multiplied by the interpreted sample value of the decoded auxiliary picture before calculating the film grain value.

[0095] In an embodiment, instead of extending the previously existing alpha channel information to add a new mode, a new type of SEI message can be specifically generated for the purpose of masking the regions in the picture where the applicable film grain is present. The new type of SEI message can include a syntax element (e.g., fgs_aux_weighting_idc) that indicates whether the analog film grain value at the same location and color component sample should be weighted by a weighting factor as determined based on the associated auxiliary picture. For example, when the value of fgs_aux_weighting_idc is equal to 0, the analog film grain value at the same location and color component sample is not weighted by the weighting factor determined based on the associated auxiliary picture. On the other hand, if the value of fgs_aux_weighting_idc is equal to 1, the analog film grain value at the same location and color component sample is not weighted by the weighting factor determined based on the associated auxiliary picture. Specifically, before calculating the film grain value, the analog film grain value at the same location and color component sample in the picture can be multiplied by the interpreted sample value of the decoded auxiliary picture.

[0096] In one embodiment, if the current PU contains a new type of SEI message, the simulated film grain value will be weighted by the auxiliary data value as follows: - If fg_blending_mode_id is equal to 0, the blending mode is additive, as specified by the following equation: - Otherwise (fg_blending_mode_id is equal to 1), the blending mode is multiplicative, as specified by the following equation: where Iaux[c][x][y] represents the sample value at coordinates x, y of color component c of the decoded auxiliary picture.

[0097] Film grain synthesis (aka FGS) is a tool that aims to preserve the original film grain while maintaining coding efficiency due to its randomness and complexity. Although used to preserve artistic intent or mitigate visual artifacts, film grain is not relevant to the entire sequence or even within an image. This aspect of the contribution is addressed by using auxiliary data as a mask for locally applying film grain synthesis on a per-image basis.

[0098] Film grain is a well-known characteristic of small grains originating from silver halide crystals on film, which has become easily recognizable as a feature of cinematic art. The film grain effect also gives games a cinematic feel by simulating the grain visual effect that appears in some movies. If used softly and locally where it is valuable to the viewer, it creates a sense of realism. In such cases, those artificial defects make the synthetic content more realistic. Finally, film grain can also be used to soften visual artifacts generated during the capture process or during the compression of video signals.

[0099] In many cases, film grain provides a certain degree of realism only in certain parts of the picture (e.g., on skin textures), while keeping the background soft.

[0100] In its simplest approach, the spatial adaptation of film grain synthesis combines the calculated film grain value with a binary mask transmitted in the auxiliary data. In this case, each film grain value is applied as is in the image or not applied at all.

[0101] Another method is to use a mask to define a weighting factor that follows the CTU structure of the main picture, and apply more film grain in regions with lower frequencies.

[0102] An alternative method that still relies on the same process is to use a mask that has a weighting factor for the film grain values. Such a mask can be a depth map sent as auxiliary data for 3D rendering purposes. In such a scenario, the film grain is applied to the foreground object while the background remains unchanged. This configuration is typically used to increase depth perception.

[0103] A film grain characteristic SEI message has been defined to synthesize the grain of the content without encoding it. Today, it lacks spatial adaptability to the image.

[0104] The VSEI specification also stipulates an alpha channel information (ACI) SEI message, which defines how to combine the auxiliary data with the output image for alpha blending.

[0105] The ACI SEI message has 2 defined modes in alpha_channel_use_idc: Mode 0: Multiplication; and Mode 1: Non-multiplication. A third mode (Mode 3, since Mode 2 is set to "unspecified") can be defined as the multiplication use of the auxiliary data with the corresponding simulated film grain defined by the film grain characteristic SEI message.

[0106] When interpreting the film grain characteristic SEI message, if there is also an ACI SEI message for the same picture where alpha_channel_use_idc is equal to 3, the associated auxiliary picture is used as the weighting factor for the film grain values.

[0107] The following clauses depict the affected changes on the VSEI specification.

[0108] As specified in Table 6, fg_blending_mode_id identifies the blending mode used to blend the simulated film grain with the input image. fg_blending_mode_id shall be in the range of 0 to 1 (including the endpoints). The values 2 and 3 of fg_blending_mode_id are reserved for future use by ITU-T|ISO / IEC and shall not appear in the bitstream of this version of the specification that conforms to this specification. The decoder shall ignore the film grain characteristic SEI messages where fg_blending_mode_id is equal to 2 or 3.

[0109] According to the value of fg_blending_mode_id, the blending mode is specified as follows: - If fg_blending_mode_id is equal to 0, the blending mode is additive, as specified by the following formula: - Otherwise (fg_blending_mode_id is equal to 1), the blending mode is multiplicative, as specified by the following formula: in, Represents the input image The sample value at coordinates x, y of color component c of , G[c][x][y] is the simulated film grain value at the same position and color component, and fgBitDepth[c] is the ... grain [c][x][y], and the number of bits used for each sample in the fixed-length unsigned binary representation of G[c][x][y], where c = 0..2, x = 0..PicWidthInLumaSamples-1, and y = 0..PicHeightInLumaSamples-1.

[0110] If the current PU contains an ACI SEI message (as defined in clause 8.23.2) with alpha_channel_use_idc equal to 3, the simulated film grain value shall be weighted by the ancillary data value as follows: - If fg_blending_mode_id is equal to 0, the blending mode is additive, as specified by the following formula: Otherwise (fg_blending_mode_id is equal to 1), the blending mode is multiplicative, as specified by the following formula: Among them, I aux [c][x][y] represents the sample value at coordinates x, y of the color component c of the decoded auxiliary picture.

[0111] The Alpha Channel Information (ACI) SEI message provides information about alpha channel sample values ​​encoded in auxiliary pictures of type AUX_ALPHA and the post-processing applied to decode the alpha plane and one or more associated primary pictures.

[0112] When the CVS does not contain an SDI SEI message with at least one value sdi_aux_id[i] equal to 1 for i, no picture in the CVS shall be associated with an ACI SEI message.

[0113] When an AU contains both an SDI SEI message and an ACI SEI message with at least one value sdi_aux_id[i] equal to 1 for i, the SDI SEI message shall precede the ACI SEI message in decoding order.

[0114] When the access unit contains the auxiliary picture picA in the layer with nuh_layer_id equal to nuhLayerIdA which is indicated as the alpha auxiliary layer by the SDISEI message, the alpha channel sample values of picA continue in output order until one or more of the following conditions become true: - The next picture with nuh_layer_id equal to nuhLayerIdA is output in output order. - The CLVS containing the auxiliary picture picA ends. - The bitstream ends. - The CLVS of any associated primary layer of the auxiliary picture layer with nuh_layer_id equal to nuhLayerIdA ends.

[0115] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id values to which the ACISEI message is applied.

[0116] alpha_channel_cancel_flag equal to 1 indicates that the SEI message cancels the persistence of any previous ACI SEI message in the output order applied to the current layer. alpha_channel_cancel_flag equal to 0 indicates ACI follow - through.

[0117] Let currPic be the picture associated with the ACI SEI message. The semantics of the ACI SEI message continue in output order for the current layer until one or more of the following conditions become true: - A new CLVS for the current layer starts. - The bitstream ends. - The picture associated with the ACISEI message in the current layer in the output AU that follows the current picture in output order.

[0118] alpha_channel_use_idc equal to 0 indicates that for the purpose of alpha blending, the decoded samples of the associated primary picture shall be multiplied by the interpreted sample values of the decoded auxiliary picture during the display process after output from the decoding process. alpha_channel_use_idc equal to 1 indicates that for the purpose of alpha blending, the decoded samples of the associated primary picture shall not be multiplied by the interpreted sample values of the decoded auxiliary picture during the display process after output from the decoding process. alpha_channel_use_idc equal to 2 indicates that the use of the auxiliary picture is not specified. alpha_channel_use_idc equal to 3 indicates that before calculating the film grain value, the analog film grain value at the same position and color component sample shall be multiplied by the interpreted sample values of the decoded auxiliary picture. For alpha_channel_use_idc, values greater than 3 are reserved for future use by ITU-T|ISO / IEC. When absent, the value of alpha_channel_use_idc is inferred to be equal to 2. The decoder shall ignore the alpha channel information SEI message in which alpha_channel_use_idc is greater than 3.

[0119] Referring briefly again to Figure 2 , consider an application scenario where in the transmission system 203, the content source 201 is not an actual imaging device, but an image synthesizer that puts together the video sequence 202, which covers a scene equivalent to the scene shown as the synthetic scene 603. In other words, the signal presented to the encoder 202 is a synthetic scene, where some parts benefit from denoising and film grain recognition and encoding in the form of metadata such as SEI messages, while other parts do not. Such an encoded stream can be transmitted 204, 208 potentially through the server 205 to the receiving system 212 and the decoder 211 located therein. The receiving system can decode the metadata that identifies the samples that require film grain processing and apply it to the reconstructed bitstream 210 through a post-processing step.

[0120] Summarizing the above use case discussion, what is needed and provided by the embodiments is: 1. A mechanism for identifying one or more regions of a reconstructed encoded picture to which film grain processing should be applied to obtain an optimal user experience. Such regions can be rectangular regions, but can overlap. Additionally, it may make sense to apply different film grain characteristics to different regions. 2. A mechanism for describing film grain characteristics. 3. A mechanism that combines the two mechanisms described above, where for regions that require film grain, appropriate film grain characteristics (including possibly no film grain characteristics) can be applied; that is, film grain can be synthesized in a post-processing step.

[0121] Re 1: The "annotation region" SEI (AR) is available in the VSEI specification and, without using the video encoding technology of VSEI - or in such cases where VSEI is available but insufficient for the application - a mechanism equivalent to AR can be designed by those skilled in the art. Note that AR can, in particular, specify multiple regions in an ordered list and associate each region with a string selected by the encoder.

[0122] Re 2: VSEI specifies the film grain SEI message (FGS), and similar information can be obtained in the VUI of the video encoding specification. Multiple FGS messages can be in a picture unit (PU), and their order of appearance in the bitstream can be used by the decoder / receiver (211) to associate a given one of the multiple FGSs with one of the regions defined in AR.

[0123] Re 3: A new mechanism is needed to create an association between one or more FGSs and at one or more ARs to associate the film grain characteristics conveyed by a given FGS with the regions defined in AR.

[0124] A process that can be performed in the decoder can involve the following steps: The process will be and, according to the embodiment, will be the following: Step 1: Parameter initialization Step 2: Scan the tags and look for "FGS-E" Step 3: Scan the objects and look for at least one annotation region marked with FGS. Step 4: If at least one region (object) with FGS is found, then apply the simulated film grain value only to the regions associated with the FGS-E and FGS-D tags, and keep the image unchanged in the remaining regions. Otherwise, by default, the film grain synthesis will be applied to the entire picture.

[0125] Summarizing the above use case discussion, what is needed and provided by the embodiment is: 1. A mechanism for identifying one or more samples of reconstructed coded pictures to which film grain processing should be applied to obtain an optimal user experience. Such samples may be represented by rectangular regions, but may also have more complex shapes. Additionally, it may be meaningful to a) apply different film grain properties to different samples, and / or b) apply different intensities of film grain to different samples. 2. A mechanism for describing film grain characteristics. 3. A mechanism for combining the two mechanisms described above, where the sample selection / intensity mechanism is applicable to film grain.

[0126] Furthermore, for the 3.38.2 film grain region characteristic SEI message semantics according to an exemplary embodiment, the SEI message provides a parameterized model for the film grain synthesis process to the decoder. The film grain synthesis process should be applied to the decoded picture before the decoded picture is displayed. Using this SEI message requires the definition of the following variables:

[0127] - The picture width and picture height in units of luma samples, represented herein by PicWidthInLumaSamples and PicHeightInLumaSamples, respectively.

[0128] - When the syntax element rdf_separate_colour_description_present_flag of the film grain region characteristic SEI message is equal to 0, the following additional variables are required:

[0129] - The chroma format indicator, represented herein by ChromaFormatIdc, as described in Clause 7.3.

[0130] - The bit depth of the samples of the luma component, represented herein by BitDepthY; and when ChromaFormatIdc is not equal to 0, the bit depth of the samples of the two associated chroma components, represented herein by BitDepthC.

[0131] The film grain model specified in the SEI message for film grain region characteristics is expressed for application to decoded pictures with a 4:4:4 color format, where the luma and chroma bit depths correspond to the luma and chroma bit depths of the film grain model, and the same color representation domain as the identified film grain model is used. When the color format of the decoded video is not 4:4:4 or the decoded video uses luma or chroma bit depths different from those of the film grain model, or uses a color representation domain different from that of the identified film grain model, it is expected that an unspecified conversion process will be applied to convert the decoded picture into a form expressed for application of the film grain model. Since no specific method is required to perform the film grain generation function used by the display process, if necessary, the decoder can down-convert the model information for chroma to simulate film grain for other chroma formats (4:2:0 or 4:2:2), rather than up-converting the decoded video (using a method not specified in this specification) before performing film grain generation.

[0132] And according to the embodiment, fgr_region_top[i], fgr_region_left[i], fgr_region_width[i], and fgr_region_height[i] respectively specify the coordinates of the upper left corner and the width and height of the bounding box of the i-th region in the picture.

[0133] The value of fgr_region_left[i] shall be in the range from 0 to PicWidthInLumaSamples - 1 (including the endpoints).

[0134] The value of fgr_region_top[i] shall be in the range from 0 to PicHeightInLumaSamples - 1 (including the endpoints).

[0135] The value of fgr_region_width[i] shall be in the range from 0 to PicWidthInLumaSamples - fgr_region_left[i] (including the endpoints).

[0136] The value of fgr_region_height[i] shall be in the range from 0 to PicHeightInLumaSamples - fgr_region_top[i] (including the endpoints).

[0137] The techniques described above (such as the SEI messages for the film grain synthesis mask described above) can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media or implemented by one or more hardware processors specially configured. For example, Figure 9 FIG. 900 shows a computer system suitable for implementing certain embodiments of the disclosed subject matter.

[0138] The computer software can be encoded using any suitable machine code or computer language, which can be subject to mechanisms such as assembly, compilation, linking, etc. to create code including instructions that can be directly executed by a computer central processing unit (CPU), a graphics processing unit (GPU), etc. or executed through interpretation, microcode execution, etc.

[0139] The instructions can be executed on various types of computers or their components including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0140] Figure 9 The components shown in FIG. 900 are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should also not be construed as having any dependency or requirement related to any one component or combination of components shown in the exemplary embodiments of the computer system 900.

[0141] The computer system 900 may include certain human-machine interface input devices. Such human-machine interface input devices can respond to inputs made by one or more human users through, for example, tactile inputs (e.g., keystrokes, swipes, data glove movements), audio inputs (e.g., voice, taps), visual inputs (e.g., gestures), olfactory inputs (not depicted). The human-machine interface devices can also be used to capture certain media not necessarily directly related to inputs made by human consciousness, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera device), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0142] The input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard 901, mouse 902, touchpad 903, touch screen 910, joystick 905, microphone 906, scanner 908, camera device 907.

[0143] The computer system 900 may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, haptic output, sound, light, and smell / taste. Such human-machine interface output devices may include: haptic output devices (e.g., haptic feedback through the touch screen 910, or the joystick 905, but there may also be haptic feedback devices that do not serve as input devices); audio output devices (e.g., speakers 909, headphones (not depicted)); visual output devices (e.g., the screen 910, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without haptic feedback capabilities - some of which may be able to output two-dimensional visual output or more than three-dimensional output through means such as stereoscopic picture output; virtual reality glasses (not depicted); holographic displays and smoke cans (not depicted)); and printers (not depicted).

[0144] The computer system 900 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 920 with media such as CD / DVD 911, thumb drives 922, removable hard disk drives or solid state drives 923, traditional magnetic media such as tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.

[0145] Those skilled in the art should also understand that the term "computer-readable medium" used in connection with the currently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0146] The computer system 900 may also include an interface 999 to one or more communication networks 998. The network 998 may be, for example, wireless, wired, optical. The network 998 may also be local, wide area, metropolitan area, vehicle and industrial, real-time, delay-tolerant, etc. Examples of the network 998 include: local area networks such as Ethernet; wireless LAN; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicle and industrial networks including CANBus, etc. Some networks 998 typically require an external network interface adapter attached to certain common data ports or peripheral buses (950 and 951) (such as, for example, the USB port of the computer system 900); other networks are typically integrated into the core of the computer system 900 by attaching to the system bus described below (for example, to the Ethernet interface in a PC computer system or to the cellular network interface in a smart phone computer system). Using any of these networks 998, the computer system 900 can communicate with other entities. Such communication can be unidirectional receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus to certain CANbus devices), or bidirectional such as to other computer systems using local digital networks or wide area digital networks. Certain protocols and protocol stacks can be used on each of these networks and network interfaces described above.

[0147] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces can be attached to the core 940 of the computer system 900.

[0148] The core 940 may include one or more central processing units (CPUs) 941, a graphics processing unit (GPU) 942, a graphics adapter 917, a dedicated programmable processing unit in the form of a field programmable gate area (FPGA) 943, a hardware accelerator 944 for certain tasks, etc. These devices, together with a read-only memory (ROM) 945, a random access memory 946, and an internal mass storage device 947 such as an internal non-user-accessible hard disk drive, SSD, etc., can be connected via a system bus 948. In some computer systems, the system bus 948 may be accessible in the form of one or more physical plugs to allow expansion by attaching additional CPUs, GPUs, etc. Peripheral devices can be attached directly or via a peripheral bus 949 to the system bus 948 of the core. The architecture of the peripheral bus includes PCI, USB, etc.

[0149] The CPU 941, GPU 942, FPGA 943, and accelerator 944 can execute certain instructions, which, when combined, can constitute the computer code mentioned above. This computer code can be stored in the ROM 945 or RAM 946. Transient data can also be stored in the RAM 946, while permanent data can be stored in, for example, the internal mass storage device 947. Fast storage and retrieval of any storage device in the memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 941, GPUs 942, mass storage devices 947, ROM 945, RAM 946, etc.

[0150] A computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and the computer code can be media and computer code specially designed and constructed for the purposes of the present disclosure, or the medium and the computer code can be of the types well known and available to those skilled in the field of computer software.

[0151] By way of example and without limitation, a computer system having the architecture 900 and particularly the core 940 can provide functionality due to a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with a user-accessible mass storage device as introduced above, and certain storage devices of the core 940 having a non-transitory nature, such as the core internal mass storage device 947 or ROM 945. The software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core 940. Depending on specific needs, the computer-readable medium can include one or more memory devices or chips. The software can cause the core 940 and particularly the processors therein (including the CPU, GPU, FPGA, etc.) to perform the specific processing or specific parts of the specific processing described herein, including defining data structures stored in the RAM 946 and modifying such data structures according to the processing defined by the software. Additionally or alternatively, the computer system can provide functionality due to being logically hardwired or otherwise implemented in a circuit (e.g., accelerator 944), which can operate in place of or in conjunction with the software to perform the specific processing or specific parts of the specific processing described herein. In appropriate cases, a reference to software can include logic, and a reference to logic can also include software. In appropriate cases, a reference to a computer-readable medium can include a circuit (e.g., an integrated circuit (IC)) storing software for execution, a circuit implementing logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0152] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various alternative equivalents that fall within the scope of the present disclosure. Accordingly, it will be recognized that those skilled in the art will be able to conceive of many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are thus within the spirit and scope of the present disclosure.

[0153] The foregoing disclosure also encompasses the features mentioned below. The features may be combined in various ways and are not limited to the combinations mentioned below.

[0154] (1) A method for video decoding, the method comprising: obtaining video data including at least one encoded picture; reconstructing the encoded picture in the video data associated with a film grain characteristic SEI message, a first sample, a second sample, a third alpha map sample, and a fourth alpha map sample, wherein, after reconstruction, the first sample and the third alpha map sample and the second sample and the fourth sample are spatially co-located; applying film grain synthesis to the first sample based on a first value of the third sample; and determining not to apply film grain synthesis to the second sample based on a second value of the fourth sample, wherein the first value and the second value are different from each other.

[0155] (2) The method according to feature (1), wherein the encoded picture in the video data is further associated with an alpha channel information SEI message indicating the third alpha map sample and the fourth alpha map sample.

[0156] (3) The method according to any one of features (1) to (2), wherein the alpha channel information SEI message indicates alpha channel sample values encoded in an auxiliary picture of AUX_ALPHA type and post-processing applied to decode the alpha plane and one or more associated primary pictures.

[0157] (4) The method according to any one of features (1) to (3), wherein the alpha channel information SEI message having a predetermined value indicates that, before calculating the film grain value, the simulated film grain value at the same position and color component sample is to be multiplied by the interpreted sample value of the decoded auxiliary picture.

[0158] (5) The method according to any one of features (1) to (4), wherein the value is the integer 3.

[0159] (6) The method according to any one of features (1) to (5), wherein the film grain characteristic SEI message identifies a blending mode for blending simulated film grain with the input image.

[0160] (7) The method according to any one of features (1) to (6), wherein the granularity characteristic SEI message further identifies the weight applied to the film grain synthesis of the first sample.

[0161] (8) A method for video coding, the method comprising: obtaining video data including at least one picture; and encoding the video data such that reconstructing the at least one picture includes reconstructing the at least one picture in the video data associated with a film grain characteristic SEI message, a first sample, a second sample, a third alpha map sample, and a fourth alpha map sample, wherein, after reconstruction, the first sample and the third alpha map sample, and the second sample and the fourth sample are spatially co-located, wherein reconstructing the at least one picture further includes applying film grain synthesis to the first sample based on a first value of the third sample, and determining not to apply film grain synthesis to the second sample based on a second value of the fourth sample, wherein the first value and the second value are different from each other.

[0162] (9) The method according to feature (8), wherein encoding the video data includes associating pictures in the video data with an alpha channel information SEI message indicating the third alpha map sample and the fourth alpha map sample.

[0163] (10) The method according to any one of features (8) to (9), wherein the alpha channel information SEI message indicates alpha channel sample values encoded in an auxiliary picture of AUX_ALPHA type, post-processing applied to decode the alpha plane, and one or more associated primary pictures.

[0164] (11) The method according to any one of features (8) to (10), wherein the alpha channel information SEI message having a predetermined value indicates that, before calculating the film grain value, the analog film grain value at the same position and color component sample is to be multiplied by the interpreted sample value of the decoded auxiliary picture.

[0165] (12) The method according to any one of features (8) to (11), wherein the value is the integer 3.

[0166] (13) The method according to any one of features (8) to (12), wherein the film grain characteristic SEI message identifies a blending mode for blending the analog film grain with the input image.

[0167] (14) The method according to any one of features (8) to (13), wherein the granularity characteristic SEI message further identifies the weight applied to the film grain synthesis of the first sample.

[0168] (15) A method for processing visual media data, the method comprising: performing a conversion between a visual media file and a bitstream of visual media data according to format rules, the format rules indicating an encoded picture in reconstructed video data associated with a film grain characteristic SEI message, a first sample, a second sample, a third alpha mapping sample, and a fourth alpha mapping sample, wherein, after reconstruction, the first sample and the third alpha mapping sample, and the second sample and the fourth sample are spatially co-located; applying a film grain synthesis to the first sample based on a first value of the third sample; and determining not to apply the film grain synthesis to the second sample based on a second value of the fourth sample, wherein the first value and the second value are different from each other.

[0169] (16) The method according to feature (15), wherein the encoded picture is further associated in the video data with an alpha channel information SEI message indicating the third alpha mapping sample and the fourth alpha mapping sample.

[0170] (17) The method according to any one of features (15) to (16), wherein the alpha channel information SEI message indicates alpha channel sample values encoded in an auxiliary picture of AUX_ALPHA type and post-processing applied to the decoded alpha plane and one or more associated primary pictures.

[0171] (18) The method according to any one of features (15) to (17), wherein the alpha channel information SEI message with a predetermined value indicates that, before calculating the film grain value, the analog film grain value at the same position and color component sample is to be multiplied by the interpreted sample value of the decoded auxiliary picture.

[0172] (19) The method according to any one of features (15) to (18), wherein the value is the integer 3.

[0173] (20) The method according to any one of features (15) to (19), wherein the film grain characteristic SEI message identifies a mixing mode for mixing the analog film grain with the input image.

[0174] (21) A device for video decoding, comprising processing circuitry configured to perform the method according to any one of features (1) to (7).

[0175] (22) A device for video encoding, comprising processing circuitry configured to perform the method according to any one of features (8) to (15).

[0176] (23) A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method according to any one of features (1) to (20).

Claims

1. A method for video decoding, the method comprising: Obtaining video data comprising at least one encoded picture; reconstructing a coded picture associated with a film grain characteristics SEI message, a first sample, a second sample, a third alpha map sample, and a fourth alpha map sample in the video data, wherein after reconstruction, the first sample and the third alpha map sample and the second sample and the fourth sample are spatially co-located; applying film grain synthesis to the first sample based on a first value of the third sample; and A determination is made based on a second value of the fourth sample not to apply film grain synthesis to the second sample, wherein the first value and the second value are different from each other.

2. The method according to claim 1, in, The encoded picture is also associated in the video data with an alpha channel information SEI message indicating the third alpha map sample and the fourth alpha map sample.

3. The method according to claim 2, in, The alpha channel information SEI message indicates alpha channel sample values ​​encoded in auxiliary pictures of type AUX_ALPHA and the post-processing applied to decode the alpha plane and one or more associated primary pictures.

4. The method according to claim 2, in, The alpha channel information SEI message having a predetermined value indicates that the simulated film grain value at the same position and color component sample is to be multiplied by the interpreted sample value of the decoded auxiliary picture before calculating the film grain value.

5. The method according to claim 4, in, The value is the integer 3.

6. The method according to claim 1, in, The film grain characteristics SEI message identifies a blending mode for blending simulated film grain with an input image.

7. The method according to claim 6, in, The grain characteristics SEI message also identifies a weight to be applied to the film grain synthesis of the first sample.

8. A method for video encoding, the method comprising: Obtaining video data including at least one picture; as well as encoding the video data such that reconstructing the at least one picture comprises reconstructing the at least one picture in the video data associated with a film grain characteristics SEI message, a first sample, a second sample, a third alpha map sample, and a fourth alpha map sample, wherein after reconstruction, the first sample and the third alpha map sample and the second sample and the fourth sample are spatially co-located, Wherein reconstructing the at least one picture further comprises applying film grain synthesis to the first sample based on a first value of the third sample, and determining not to apply film grain synthesis to the second sample based on a second value of the fourth sample, wherein the first value and the second value are different from each other.

9. The method according to claim 8, in, Encoding the video data includes associating a picture in the video data with an alpha channel information SEI message indicating the third alpha map sample and the fourth alpha map sample.

10. The method according to claim 9, in, The alpha channel information SEI message indicates the alpha channel sample values ​​encoded in an auxiliary picture of type AUX_ALPHA and the post-processing applied to decode the alpha plane and one or more associated primary pictures.

11. The method according to claim 9, in, The alpha channel information SEI message having a predetermined value indicates that the simulated film grain value at the same position and color component sample is to be multiplied by the interpreted sample value of the decoded auxiliary picture before calculating the film grain value.

12. The method according to claim 11, in, The value is the integer 3.

13. The method according to claim 8, in, The film grain characteristics SEI message identifies a blending mode for blending simulated film grain with an input image.

14. The method according to claim 13, in, The grain characteristics SEI message also identifies a weight to be applied to the film grain synthesis of the first sample.

15. A method of processing visual media data, the method comprising: Conversion is performed between the visual media file and the bitstream of visual media data according to format rules, the format rules indicating: reconstructing a coded picture in the video data associated with the film grain characteristics SEI message, a first sample, a second sample, a third alpha map sample, and a fourth alpha map sample, wherein after reconstruction, the first sample and the third alpha map sample and the second sample and the fourth sample are spatially co-located; applying film grain synthesis to the first sample based on a first value of the third sample; and A determination is made not to apply film grain synthesis to the second sample based on a second value of the fourth sample, wherein the first value and the second value are different from each other.

16. The method according to claim 15, in, The encoded picture is also associated in the video data with an alpha channel information SEI message indicating the third alpha map sample and the fourth alpha map sample.

17. The method according to claim 16, in, The alpha channel information SEI message indicates alpha channel sample values ​​encoded in auxiliary pictures of type AUX_ALPHA and the post-processing applied to decode the alpha plane and one or more associated primary pictures.

18. The method according to claim 16, in, The alpha channel information SEI message having a predetermined value indicates that the simulated film grain value at the same position and color component sample is to be multiplied by the interpreted sample value of the decoded auxiliary picture before calculating the film grain value.

19. The method according to claim 18, in, The value is the integer 3.

20. The method according to claim 15, in, The film grain characteristics SEI message identifies a blending mode for blending simulated film grain with an input image.

Images