Frame rate scalable video coding

Through frame rate scalability technology, the compatibility problem of video high dynamic range content on different devices is solved, and the cross-device compatibility and encoding performance of video content are improved.

CN114845111BActive Publication Date: 2024-11-26DOLBY LABORATORIES LICENSING CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210473165.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-09-24
Filing Date
2020-03-11
Publication Date
2024-11-26
Estimated Expiration
2040-03-11

AI Technical Summary

Technical Problem

Existing video high dynamic range content is limited by device performance when distributed and played back, especially frame rate and resolution, resulting in content producers having to make compatible versions for different devices, adding complexity and cost.

Method used

Through frame rate scalability technology, the video encoder allows the video encoder to generate scalable bitstreams when encoding, supporting video content at different frame rates and shutter angles. The decoder can dynamically adjust the frame rate and shutter angle according to the device's capabilities for compatibility .

Benefits of technology

It realizes cross-device compatibility of video content, simplifies content production and distribution processes, and improves the flexibility and performance of video encoding, especially in high resolution and high dynamic range scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114845111B_ABST
    Figure CN114845111B_ABST
Patent Text Reader

Abstract

Methods and systems for frame rate scalability are described. Support is provided for input and output video sequences with variable frame rates and variable shutter angles across scenes, or for input video sequences with fixed input frame rates and input shutter angles, but allowing the decoder to generate video output at output frame rates and shutter angles that differ from the corresponding input values. Techniques are also presented that allow the decoder to more computationally efficiently decode certain backward-compatible target frame rates and shutter angles among those allowed.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the Chinese invention patent application with application number 202080002086.5, filed on March 11, 2020, which entered the Chinese national phase on September 24, 2020, and invention name “Frame rate scalable video coding”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 816,521, filed on March 11, 2019, U.S. Provisional Application No. 62 / 850,985, filed on May 21, 2019, U.S. Provisional Application No. 62 / 883,195, filed on August 6, 2019, and U.S. Provisional Application No. 62 / 904,744, filed on September 24, 2019, each of which is incorporated by reference in its entirety. Technical Field

[0004] This document relates generally to images. More particularly, embodiments of the invention relate to frame-rate scalable video coding. Background Art

[0005] As used herein, the term "dynamic range (DR)" may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, such as from the darkest gray (black) to the brightest white (highlight). In this sense, DR is related to "scene-referred" intensities. DR may also relate to the ability of a display device to fully or approximately render a range of intensities of a particular breadth. In this sense, DR is related to "display-referred" intensities. Unless a particular sense is expressly specified at any point in the description herein to have a particular meaning, it should be inferred that the terms may be used in either sense, such as interchangeably.

[0006] As used herein, the term "high dynamic range (HDR)" refers to a DR width that spans 14 to 15 orders of magnitude across the human visual system (HVS). In practice, with respect to HDR, the DR width over a wide range of intensities that humans can simultaneously perceive may be slightly truncated.

[0007] In practice, an image includes one or more color components (e.g., luma Y and chroma Cb and Cr), where each color component is represented by n bits of precision per pixel (e.g., n=8). Using linear luminance encoding, images where n≤8 (e.g., color 24-bit JPEG images) are considered standard dynamic range (SDR) images, while images where n>8 can be considered enhanced dynamic range images. HDR images can also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

[0008] Currently, the distribution of video high dynamic range content (such as Dolby Laboratories' Dolby Vision or HDR10 in Blu-ray) is limited by the performance of many playback devices to 4K resolution (e.g., 4096×2160 or 3840×2160, etc.) and 60 frames per second (fps). In future versions, it is expected that content with up to 8K resolution (e.g., 7680×4320) and 120fps will be available for distribution and playback. It is expected that future content types will be compatible with existing playback devices to simplify the HDR playback content ecosystem, such as Dolby Vision. Ideally, content producers should be able to adopt and distribute future HDR technologies without having to also obtain and distribute special versions of content that are compatible with existing HDR devices (such as HDR10 or Dolby Vision). As understood by the inventors herein, improved techniques for scalable distribution of video content, especially HDR content, are desired.

[0009] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art simply by virtue of their inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to have been recognized in any prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0011] Figure 1 Describes an example process of a video delivery pipeline;

[0012] Figure 2 depicts an example process for combining consecutive raw frames to render a target frame rate at a target shutter angle in accordance with an embodiment of the present invention;

[0013] Figure 3depicts an example representation of an input sequence with a variable input frame rate and a variable shutter angle in a container with a fixed frame rate in accordance with an embodiment of the present invention; and

[0014] Figure 4 Depicted are example representations of temporal scalability at various frame rates and shutter angles with backward compatibility in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0015] Example embodiments related to frame rate scalability for video coding are described herein. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. However, it is clear that various embodiments of the present invention can be practiced without these specific details. In other cases, in order to avoid unnecessarily obscuring, blurring or confusing embodiments of the present invention, well-known structures and devices are not described in detail.

[0016] Overview

[0017] Example embodiments described herein relate to frame rate scalability in video coding. In an embodiment, a system with a processor receives a coded bitstream, the coded bitstream including coded video frames, wherein one or more coded frames are encoded at a first frame rate and a first shutter angle. The processor receives a first flag indicating that there is a group of coded frames to be decoded at a second frame rate and a second shutter angle, the processor accesses values ​​of the second frame rate and the second shutter angle for the group of coded frames from the coded bitstream, and generates decoded frames of the second frame rate and the second shutter angle based on the group of coded frames, the first frame rate, the first shutter angle, the second frame rate, and the second shutter angle.

[0018] In a second embodiment, a decoder having a processor:

[0019] Receiving a coded bitstream, the coded bitstream comprising a plurality of groups of coded video frames, wherein all coded video frames in the coded bitstream are encoded at a first frame rate;

[0020] The number of received combined frames N;

[0021] Receives the value of the baseline frame rate;

[0022] Accessing a set of N consecutive coded frames, wherein an i-th coded frame in the set of N consecutive coded frames represents an average of at most i input video frames encoded at the baseline frame rate and the i-th shutter angle based on a first shutter angle and the first frame rate in the encoder, wherein i=1, 2, ...N;

[0023] accessing values ​​of a second frame rate and a second shutter angle from the encoded bitstream or from user input for use in decoding the set of N consecutive encoded frames at the second frame rate and the second shutter angle; and

[0024] Based on the set of N consecutive encoded frames, the first frame rate, the first shutter angle, the second frame rate, and the second shutter angle, a decoded frame of the second frame rate and the second shutter angle is generated.

[0025] In a third embodiment, a coded video stream structure includes:

[0026] a coded picture portion, the coded picture portion comprising a coding of a sequence of video pictures; and

[0027] A signaling part, the signaling part comprising the encoding of:

[0028] a shutter interval time scale parameter indicating the number of time units that elapse in one second;

[0029] a shutter interval clock period parameter indicating the number of time units of a clock operating at the frequency of the shutter interval time scale parameter,

[0030] wherein the shutter interval clock period parameter divided by the shutter interval time scale parameter indicates an exposure duration value;

[0031] a shutter interval duration flag indicating whether exposure duration information is fixed for all temporal sub-layers in the coded picture portion; and

[0032] If the shutter interval duration flag indicates that the exposure duration information is fixed, decoding a decoded version of the video picture sequence for all the temporal sub-layers in the coded picture portion by calculating the exposure duration value based on the shutter interval time scale parameter and the shutter interval clock period parameter, otherwise

[0033] The signaling portion includes one or more sub-layer parameter arrays, wherein values ​​in the one or more sub-layer parameter arrays combined with the shutter interval time scale parameter are used to calculate, for each sub-layer, a corresponding sub-layer exposure duration value for a decoded version of the temporal sub-layer for displaying the video picture sequence.

[0034] Example Video Transmission Processing Pipeline

[0035] Figure 1 An example process of a conventional video transmission pipeline (100) is depicted, which shows various stages from video capture to video content display. An image generation block (105) is used to capture or generate a sequence of video frames (102). The video frames (102) can be digitally captured (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide video data (107). Alternatively, the video frames (102) can be captured on film by a film camera. The film is converted to a digital format to provide video data (107). In the production stage (110), the video data (107) is edited to provide a video production stream (112).

[0036] The video data of the production stream (112) is then provided to a processor at block (115) for post-production editing. Block (115) post-production editing may include adjusting or modifying the color or brightness in specific areas of the image to enhance the image quality or achieve a specific look of the image according to the creative intent of the video creator. This is sometimes referred to as "color timing" or "color grading". Other edits (e.g., scene selection and sequencing, image cropping, adding computer-generated visual effects, jitter or blur control, frame rate control, etc.) may be performed at block (115) to produce a final version (117) of the work for distribution. During post-production editing (115), the video images are viewed on a reference display (125). After post-production (115), the video data of the final work (117) may be transmitted to an encoding block (120) for downstream transmission to a decoding and playback device, such as a television, set-top box, movie theater, etc. In some embodiments, the encoding block (120) may include an audio encoder and a video encoder, such as those defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats, to generate an encoded bitstream (122). In the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a decoded signal (decoded signal) (132) representing an identical or nearly similar version of the signal (117). The receiver may be attached to a target display (140), which may have completely different characteristics from the reference display (125). In this case, a display management block (135) may be used to map the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a display-mapped signal (display-mapped signal) (137).

[0037] Scalable Coding

[0038] Scalable coding is already part of many video coding standards such as MPEG-2, AVC and HEVC. In an embodiment of the present invention, scalable coding is extended to improve performance and flexibility, especially when scalable coding involves very high resolution HDR content.

[0039] As used herein, the term "shutter angle" refers to an adjustable shutter setting that controls the proportion of time that the film is exposed during each frame interval.

[0040]

[0041] The term comes from the traditional mechanical rotating shutter; however, modern digital cameras can also adjust their shutters electronically. Cinematographers can use the shutter angle to control the amount of motion blur or jitter recorded in each frame. It should be noted that alternative terms such as "exposure duration", "shutter interval" and "shutter speed" can also be used instead of "exposure time". Similarly, the term "frame duration" can be used instead of "frame interval". Alternatively, "frame interval" can be replaced by "1 / frame rate". The value of the exposure time is usually less than or equal to the duration of the frame. For example, a shutter angle of 180 degrees indicates that the exposure time is half the frame duration. In some cases, the exposure time can be greater than the frame duration of the encoded video, for example, when the encoding frame rate is 120fps and the frame rate of the related video content before encoding and display is 60fps.

[0042] Without limitation, consider an embodiment where the original content is captured (or generated) at a 360-degree shutter angle at a native frame rate (e.g., 120 fps). Then, in a receiving device, video output can be rendered at various frame rates equal to or lower than the native frame rate by fair combination of the original frames (e.g., by averaging or other operations known in the art).

[0043] The combining process can be performed with non-linearly encoded signals (e.g., using gamma, PQ, or HLG), but the best image quality is obtained by combining frames in the linear light domain by first converting the non-linearly encoded signal to a linear light representation, then combining the converted frames, and finally re-encoding the output with a non-linear transfer function. This process provides a more accurate simulation of physical camera exposure than combining in the non-linear domain.

[0044] In general, the process of combining frames can be expressed in terms of the original frame rate, the target frame rate, the target shutter angle, and the number of frames to be combined, as follows:

[0045] n_frames=(target_shutter_angle / 360)*(original_frame_rate / target_frame_rate), (2)

[0046] It is equivalent to

[0047] target_shutter_angle=360*n_frames*(target_frame_rate / original_frame_rate), (3)

[0048] Where n_frames is the number of combined frames, original_frame_rate is the frame rate of the original content, target_frame_rate is the frame rate to be rendered (where target_frame_rate≤original_frame_rate), and target_shutter_angle indicates the amount of motion blur desired. In this example, the maximum value of target_shutter_angle is 360 degrees and corresponds to maximum motion blur. The minimum value of target_shutter_angle can be expressed as 360*(target_frame_rate / original_frame_rate) and corresponds to minimum motion blur. The maximum value of n_frames can be expressed as (original_frame_rate / target_frame_rate). The values ​​of target_frame_rate and target_shutter_angle should be selected so that the value of n_frames is a non-zero integer.

[0049] In the special case where the native frame rate is 120fps, equation (2) can be rewritten as

[0050] n_frames=target_shutter_angle / (3*target_frame_rate), (4)

[0051] It is equivalent to

[0052] target_shutter_angle=3*n_frames*target_frame_rate. (5)

[0053] For the case where original_frame_rate=120fps, the relationship between the values ​​of target_frame_rate, n_frames, and target_shutter_angle is shown in Table 1. In Table 1, "NA" indicates that the corresponding combination of the target frame rate and the number of combined frames is not allowed.

[0054] Table 1: Relationship between target frame rate, number of combined frames and target shutter angle for a native frame rate of 120 fps.

[0055]

[0056] Figure 2An example process of combining consecutive raw frames to render a target frame rate at a target shutter angle according to an embodiment is depicted. Considering an input sequence (205) at 120fps and a shutter angle of 360 degrees, the process generates an output video sequence (210) at 24fps and a shutter angle of 216 degrees by combining three input frames (e.g., the first three consecutive frames) in a set of five consecutive frames and discarding the other two frames. It should be noted that in some embodiments, the output frame -01 of (210) can be generated by combining alternative input frames (205) (such as frames 1, 3 and 5 or frames 2, 4 and 5, etc.); however, it is expected that combining consecutive frames will produce a better quality video output.

[0057] It is desirable to support raw content with variable frame rates, for example, for managing artistic and stylistic effects. It is also desirable to encapsulate the variable input frame rates of raw content in a "container" with a fixed frame rate to simplify content production, exchange, and distribution. As examples, three embodiments of how to represent variable frame rate video data in a fixed frame rate container are presented. For purposes of clarity, but not limitation, the following description uses a fixed 120fps container, but the method can be easily extended to alternative frame rate containers.

[0058] First embodiment (Variable frame rate)

[0059] The first embodiment is an explicit description of raw content with a variable (non-constant) frame rate encapsulated in a container with a constant frame rate. For example, raw content with different frame rates (e.g., 24, 30, 40, 60, or 120fps) for different scenes can be encapsulated in a container with a constant frame rate of 120fps. For this example, each input frame can be replicated 5x, 4x, 3x, 2x, or 1x times to encapsulate it into a common 120fps container.

[0060] Figure 3 An example of an input video sequence A with a variable frame rate and a variable shutter angle represented in an encoded bitstream B with a fixed frame rate is depicted. Then, in the decoder, the decoder reconstructs an output video sequence C at a desired frame rate and shutter angle, which may vary from scene to scene. For example, Figure 3 As depicted, to construct sequence B, some of the input frames are duplicated, some are encoded as is (without duplication), and some are copied four times. Then, to construct sequence C, any one frame is selected from each set of duplicate frames to generate an output frame, thereby matching the original frame rate and shutter angle.

[0061] In this embodiment, metadata is inserted into the bitstream to indicate the original (base) frame rate and shutter angle. The metadata can be signaled using high-level syntax such as sequence parameter set (SPS), picture parameter set (PPS), slice or tile group header, etc. The presence of metadata enables encoders and decoders to perform beneficial functions such as:

[0062] a) The encoder can ignore the duplicated frame, thereby increasing encoding speed and simplifying processing. For example, all coding tree units (CTUs) in the duplicated frame can be encoded using SKIP mode and reference index 0 in LIST 0 of the reference frame, which refers to the decoded frame from which the duplicated frame was copied.

[0063] b) The decoder may bypass decoding of duplicate frames, thereby simplifying processing. For example, metadata in the bitstream may indicate that a frame is a copy of a previously decoded frame that the decoder may reproduce by copying and without decoding a new frame.

[0064] c) The playback device can optimize downstream processing (e.g., by adjusting frame rate conversion or noise reduction algorithms) by indicating the base frame rate.

[0065] This embodiment enables the end user to view rendered content at the frame rate intended by the content creator. This embodiment does not provide backward compatibility with devices that do not support the container's frame rate (e.g., 120fps).

[0066] Tables 2 and 3 depict example syntaxes of the raw byte sequence payload (RBSB) for sequence parameter sets and tile group headers, where the proposed new syntax elements are depicted in highlighted font. The rest of the syntax follows the syntax in the proposed specification of the Versatile Video Codec (VVC) (reference [2]).

[0067] As an example, in the SPS (see Table 2), a flag may be added to enable variable frame rate.

[0068] sps_vfr_enabled_flag equal to 1 specifies that the coded video sequence (CVS) may contain variable frame rate content. sps_vfr_enabled_flag equal to 0 specifies that the CVS contains fixed frame rate content.

[0069] In the tile_group header() (see Table 3),

[0070] tile_group_vrf_info_present_flag equal to 1 specifies that the syntax elements tile_group_true_fr and tile_group_shutterangle are present in the syntax. tile_group_vrf_info_present_flag equal to 0 specifies that the syntax elements tile_group_true_fr and tile_group_shutterangle are not present in the syntax. When tile_group_vrf_info_present_flag is not present, it is inferred to be 0.

[0071] tile_group_true_fr indicates the true frame rate of the video data carried in this bitstream.

[0072] tile_group_shutterangle indicates the shutter angle corresponding to the actual frame rate of the video data carried in this bitstream.

[0073] tile_group_skip_flag equal to 1 specifies that the current tile group is copied from another tile group. tile_group_skip_flag equal to 0 specifies that the current tile group is not copied from another tile group.

[0074] tile_group_copy_pic_order_cnt_lsb specifies the picture order count of the previously decoded picture that the current picture is copied from modulo MaxPicOrderCntLsb when tile_group_skip_flag is set to 1.

[0075] Table 2: Example parameter set RBSP syntax for content with variable frame rate

[0076]

[0077] Table 3: Example of tile group header syntax to support content with variable frame rate

[0078]

[0079] Second Example - Fixed Frame Rate Container

[0080] The second embodiment implements a use case where original content with a fixed frame rate and shutter angle can be rendered by a decoder at an alternative frame rate and a variable simulated shutter angle, such as Figure 2As shown. For example, when the frame rate of the original content is 120fps and the shutter angle is 360 degrees (meaning that the shutter is open for 1 / 120 second), the decoder can render multiple frame rates less than or equal to 120fps. For example, as described in Table 1, in order to decode 24fps with a simulated shutter angle of 216 degrees, the decoder can combine three decoded frames and display them at 24fps. Table 4 expands Table 1 and illustrates how different numbers of encoded frames are combined to render at an output target frame rate and a desired target shutter angle. The combination of frames can be performed by simple pixel averaging, by weighted pixel averaging (where pixels from a frame can be weighted more than pixels from other frames, and the sum of all weights is one), or by other filtering interpolation schemes known in the art. In Table 4, function Ce(a,b) represents a combination of encoded frames a to b, where the combination can be performed by averaging, weighted averaging, filtering, etc.

[0081] Table 4: Example of combining input frames at 120fps to generate output frames at target fps and shutter angle values

[0082]

[0083] When the value of the target shutter angle is less than 360 degrees, the decoder can combine different sets of decoded frames. For example, according to Table 1, considering the original stream of 120fps@360 degrees, in order to generate a stream at 40fps and a shutter angle of 240 degrees, the decoder needs to combine two frames out of three possible frames. Therefore, the decoder can combine the first frame and the second frame or the second frame and the third frame. The choice of which frames to combine can be described by the "decoding phase" expressed as the following equation:

[0084] decode_phase=decode_phase_idx*(360 / n_frames), (6)

[0085] where decode_phase_idx represents an offset index within the set of sequential frames with index values ​​in [0, n_frames_max-1], where n_frames is given by equation (2), and

[0086] n_frames_max=orig_frame_rate / target_frame_rate. (7)

[0087] The range of decode_phase_idx is generally [0, n_frames_max-n_frames]. For example, for an original sequence at 120fps and a shutter angle of 360 degrees, for a target frame rate of 40fps at a shutter angle of 240 degrees, n_frames_max=120 / 40=3. According to equation (2), n_frames=2, so the range of decode_phase_idx is [0, 1]. Therefore, decode_phase_idx=0 indicates that frames with indices 0 and 1 are selected, and decode_phase_idx=1 indicates that frames with indices 1 and 2 are selected.

[0088] In this embodiment, the variable frame rate for rendering intended by the content creator can be signaled as metadata, such as a supplemental enhancement information (SEI) message or video availability information (VUI). Optionally, the rendered frame rate can be controlled by a receiver or user. An example of frame rate conversion SEI messaging that specifies the content creator's preferred frame rate and shutter angle is shown in Table 5. The SEI message can also indicate whether the combined frame is performed in the coded signal domain (e.g., γ (gamma), PQ, etc.) or in the linear light domain. It should be noted that in addition to the decoder picture buffer (DPB), post-processing also requires a frame buffer. The SEI message can indicate how many additional frame buffers are required or indicate some alternative method for combining frames. For example, to reduce complexity, frames can be reassembled at a reduced spatial resolution.

[0089] As depicted in Table 4, at certain combinations of frame rate and shutter angle (e.g., at 30fps and 360 degrees or at 24fps and 288 degrees or 360 degrees), the decoder may need to combine more than three decoded frames, which increases the amount of buffer space required by the decoder. To alleviate the burden of additional buffer space in the decoder, in some embodiments, certain combinations of frame rate and shutter angle may not use the set of allowed decoding parameters (e.g., by setting the appropriate encoding profile and level).

[0090] Considering again the case of playback at 24fps as an example, the decoder may decide to display the same frame five times to be displayed at a 120fps output frame rate. This is exactly the same as showing the frame once at a 24fps output frame rate. The advantage of maintaining a constant output frame rate is that the display can run at a constant clock speed, which makes all the hardware simpler. If the display can dynamically change the clock speed, it may make more sense to show the frame only once (for 1 / 24 second) rather than repeating the same frame five times (each time for 1 / 120 second). The former approach may produce slightly higher picture quality, better optical efficiency, or better power efficiency. Similar considerations apply to other frame rates.

[0091] Table 5 depicts an example of frame rate conversion SEI messaging syntax according to an embodiment.

[0092] Table 5: Example of SEI message syntax to allow frame rate conversion

[0093]

[0094] framerate_conversion_cancel_flag equal to 1 indicates that the SEI message cancels the preservation of any previous frame rate conversion SEI message in the output order. framerate_conversion_cancel_flag equal to 0 indicates that the frame rate conversion message follows.

[0095] base_frame_rate specifies the desired frame rate.

[0096] base_shutter_angle specifies the desired shutter angle.

[0097] decode_phase_idx_present_flag equal to 1 specifies that decoding phase information is present. decode_phase_idx_present_flag equal to 0 specifies that decoding phase information is not present.

[0098] decode_phase_idx indicates the offset index within the set of sequence frames with index values ​​0..(n_frames_max-1), where n_frames_max=120 / base_frame_rate. The value of decode_phase_idx shall be in the range of 0..(n_frames_max-n_frames), where n_frames=base_shutter_angle / (3*base_frame_rate). When decode_phase_idx is not present, it is inferred to be 0.

[0099] conversion_domain_idc equal to 0 specifies that frame combination is performed in the linear domain. conversion_domain_idc equal to 1 specifies that frame combination is performed in the non-linear domain.

[0100] num_frame_buffers specifies the number of additional frame buffers (DPB not counted).

[0101] framerate_conversion_persistence_flag specifies the persistence of frame rate conversion SEI messages for the current layer. framerate_conversion_persistence_flag equal to 0 specifies that the frame rate conversion SEI messages apply only to the current decoded picture. Let picA be the current picture. framerate_conversion_persistence_flag equal to 1 specifies that frame rate conversion SEI messages are persisted for the current layer in output order until one or more of the following conditions are true:

[0102] - A new coded layered video sequence (CLVS) for the current layer starts.

[0103] - End of bitstream.

[0104] - A picture picB in the current layer in the output access unit, wherein the picture picB contains a frame rate conversion SEI message applicable to the current layer, and for the picture picB, PicOrderCnt(picB) is greater than PicOrderCnt(picA), where PicOrderCnt(picB) and PicOrderCnt(picA) are the PicOrderCntVal values ​​of picB and picA, respectively, immediately after the decoding process for the picture order count of picB is called.

[0105] Third embodiment - Input encoded with multiple shutter angles

[0106] A third embodiment is a coding scheme that allows sub-frame rates to be extracted from the bitstream, thereby supporting backward compatibility. In HEVC, this is achieved through temporal scalability. Temporal layer scalability is achieved by assigning different values ​​to the temporal_id syntax element of the decoded frames. The bitstream can thus be simply extracted based on the temporal_id value. However, the HEVC-style approach to temporal scalability does not enable rendering output frame rates with different shutter angles. For example, a 60fps base frame rate extracted from a 120fps native frame rate will always have a shutter angle of 180 degrees.

[0107] In ATSC 3.0, an alternative approach is described in which frames at 60fps with a 360-degree shutter angle are simulated as a weighted average of two 120fps frames. The simulated 60fps frames are assigned a temporal_id value of 0 and combined with alternating original 120fps frames assigned a temporal_id value of 1. When 60fps is required, the decoder only needs to decode frames with a temporal_id of 0. When 120fps is required, the decoder can subtract each frame with temporal_id=1 (i.e., 120fps frame) from a scaled version of each corresponding frame with temporal_id=0 (i.e., simulated 60fps frame) to recover the corresponding original 120fps frame that was not explicitly transmitted, thereby reconstructing all original 120fps frames.

[0108] In an embodiment of the present invention, a new algorithm is described that supports multiple target frame rates and target shutter angles in a backward compatible (BC) manner. It is proposed to pre-process the original 120fps content at a base frame rate at several shutter angles. Then, at the decoder, other frame rates at various other shutter angles can be simply obtained. The ATSC 3.0 method can be considered as a special case of the proposed scheme, where frames with temporal_id=0 carry frames at 60fps@360 shutter angles, and frames with temporal_id=1 carry frames at 60fps@180 shutter angles.

[0109] As a first example, Figure 4 As depicted, consider an input sequence at 120fps and a shutter angle of 360, which is used to encode a sequence at a base layer frame rate of 40fps and shutter angles of 120, 240, and 360 degrees. In this scheme, the encoder calculates new frames by combining up to three frames from the original input frames. For example, encoded frame 2 (En-2) representing the input at 40fps and 240 degrees is generated by combining input frames 1 and 2, and encoded frame 3 (En-3) representing the input at 40fps and 360 degrees is generated by combining frame En-2 to input frame 3. In the decoder, in order to reconstruct the input sequence, decoded frame 2 (Dec-2) is generated by subtracting frame En-1 from frame En-2, and decoded frame 3 (Dec-3) is generated by subtracting frame En-2 from frame En-3. These three decoded frames represent the output at a base frame rate of 120fps and a shutter angle of 360 degrees. As depicted in Table 6, decoded frames can be used to extrapolate additional frame rates and shutter angles. In Table 6, function Cs(a,b) represents the combination of input frames a to b, where the combination can be performed by averaging, weighted averaging, filtering, etc.

[0110] Table 6: Examples of frame combinations with a baseline of 40fps

[0111]

[0112] An advantage of this approach is that all 40fps versions can be decoded without any further processing, as depicted in Table 6. Another advantage is that other frame rates can be obtained at various shutter angles. For example, a decoder that decodes at 30fps and a shutter angle of 360 is considered. According to Table 4, the output corresponds to a sequence of frames generated by Ce(1,4)=Cs(1,4), Cs(5,8), Cs(9,12), etc., which also matches the decoded sequence depicted in Table 6; however, in Table 6, Cs(5,8)=e6-e4+e8. In an embodiment, a lookup table (LUT) can be used to define how the decoded frames need to be combined to generate an output sequence at a specified output frame rate and simulated shutter angle.

[0113] In another example, it is proposed to combine up to five frames in the encoder to simplify the extraction of the 24fps base layer at shutter angles of 72, 144, 216, 288 and 360 degrees, as shown below. This is desirable for movie content that is best presented at 24fps on traditional televisions.

[0114] Table 7: Examples of frame combinations with a baseline of 24fps

[0115]

[0116] As depicted in Table 7, if the decoded frame rate matches the baseline frame rate (24fps), then in each group of five frames (e.g., e1 to e5), the decoder can select only one frame at the desired shutter angle (e.g., e2 for a shutter angle of 144 degrees). To decode at a different frame rate and a specific shutter angle, the decoder will need to determine how to appropriately combine (e.g., by addition or subtraction) the decoded frames. For example, to decode at 30fps and a shutter angle of 180 degrees, the following steps can be followed:

[0117] a) The decoder may consider the hypothetical encoder to transmit at 120fps and 360 degrees without any consideration for backward compatibility, then, according to Table 1, the decoder needs to combine 2 of the 4 frames to generate an output sequence at the desired frame rate and shutter angle. For example, as depicted in Table 4, the sequence includes Ce(1,2)=Avg(s1,s2), Ce(5,6)=Avg(s5,s6), etc., where Avg(s1,s2) may represent the average of frames s1 and s2.

[0118] b) Considering that according to the definition, the coded frame can be expressed as e1=s1, e2=Avg(s1,s2), e3=Avg(s1,s3), etc., it can be easily obtained that the sequence of frames in step a) can also be expressed as:

[0119] ●Ce(1,2)=Avg(s1,s2)=e2

[0120] ●Ce(5,6)=Avg(s5,s6)=Avg(s1,s5)-Avg(s1,s4)+s6=e5-e4+e6

[0121] ●And so on.

[0122] As before, the appropriate combination of decoded frames may be pre-computed and used as a LUT.

[0123] An advantage of the proposed method is that it provides options for both content creators and users; that is, it enables director / editor choice and user choice. For example, pre-processing content in the encoder allows the creation of a base frame rate with various shutter angles. Each shutter angle can be assigned a temporal_id value in the range [0, (n_frames-1)], where the value of n_frames is equal to 120 divided by the base frame rate. (For example, for a base frame rate of 24fps, the temporal_id is in the range [0, 4].) The choice can be made to optimize compression efficiency, or for aesthetic reasons. In some use cases, for example, for over the top streaming, multiple bitstreams with different base layers can be encoded and stored and provided to the user for selection.

[0124] In a second example of the disclosed method, multiple backward compatible frame rates can be supported. Ideally, it may be desirable to be able to decode at 24 frames per second to obtain a 24fps base layer, decode at 30 frames per second to obtain a 30fps sequence, decode at 60 frames per second to obtain a 60fps sequence, etc. If a target shutter angle is not specified, it is recommended that the default target shutter angle among those shutter angles allowed by the source frame rate and the target frame rate be as close to 180 degrees as possible. For example, for the values ​​depicted in Table 7, the preferred target shutter angles for 120, 60, 40, 30, and 24 fps are 360 ​​degrees, 180 degrees, 120 degrees, 180 degrees, and 216 degrees.

[0125] From the above examples, it can be observed that the choice of how to encode the content affects the complexity of decoding a specific base layer frame rate. One embodiment of the present invention is to adaptively select the encoding scheme based on the desired base layer frame rate. For example, for movie content, it may be 24fps, while for sports content, it may be 60fps.

[0126] Example syntax for the BC embodiment of the present invention is shown below and in Tables 8 and 9.

[0127] In the SPS (Table 8), two syntax elements are added: sps_hfr_BC_enabled_flag and sps_base_framerate (if sps_hfr_BC_enabled_flag is set equal to 1).

[0128] sps_hfr_BC_enabled_flag equal to 1 specifies that high frame rate with backward compatibility is enabled in coded video sequence (CVS). sps_hfr_BC_enabled_flag equal to 0 specifies that high frame rate with backward compatibility is not enabled in CVS.

[0129] sps_base_framerate specifies the base frame rate of the current CVS.

[0130] In the tile group header, if sps_hfr_BC_enabled_flag is set to 1, the syntax number_avg_frames is sent in the bitstream.

[0131] number_avg_frames specifies the number of frames at the highest frame rate (eg, 120 fps) that are combined to generate the current picture at the base frame rate.

[0132] Table 8: Example RBSP syntax for input at various shutter angles

[0133]

[0134] Table 9: Example picture parameter set RBSB syntax for input at various shutter angles

[0135]

[0136] Variation of the Second Embodiment (Fixed Frame Rate)

[0137] The HEVC (H.265) coding standard (reference [1]) and the developing general video coding standard (commonly referred to as VVC, see reference [2]) define a syntax element pic_struct that indicates whether a picture should be displayed as a frame or one or more fields and whether a decoded picture should be repeated. For ease of reference, a copy of Table D.2 "pic_struct interpretation" from HEVC is provided in the Appendix.

[0138] It is important to note that, as the inventors appreciate, when a fixed frame rate coding container is used, existing pic_struct syntax elements may only support a specific subset of content frame rates. For example, when a 60fps fixed frame rate container is used, when fixed_pic_rate_within_cvs_flag is equal to 1, the existing pic_struct syntax may support 30fps by using frame doubling, and support 24fps by using frame doubling and frame tripling in an alternating combination of every other frame. However, when a 120fps fixed frame rate container is used, the current pic_struct syntax cannot support frame rates of 24fps or 30fps. To alleviate this problem, two new approaches are proposed: one approach is an extension of the HEVC version, and the other is not.

[0139] Method 1: pic_struct without backward compatibility

[0140] VVC is still under development, so the syntax can be designed with maximum freedom. In an embodiment, in pic_struct, it is proposed to remove the options of frame doubling and frame tripling, use a specific value of pic_struct to indicate arbitrary frame repetition, and add a new syntax element num_frame_repetition_minus2 that specifies the number of frames to be repeated. An example of the proposed syntax is described in the following table, where Table 10 represents the changes relative to Table D.2.3 in HEVC, and Table 11 represents the changes to Table D.2 shown in the Appendix.

[0141] Table 10: Example picture timing SEI message syntax, method 1

[0142]

[0143] num_frame_repetition_minus2 plus 2 indicates that when fixed_pic_rate_within_cvs_flag is equal to 1, the frame should be displayed consecutively on the display num_frame_repetition_minus2 plus 2 times, where the frame refresh interval is equal to DpbOutputElementalInterval[n], as given in Equation E-73.

[0144] Table 11: Example of revised pic_struct according to method 1

[0145]

[0146] Method 2: Extended version of HEVC version of pic_struct

[0147] AVC and HEVC decoders are already deployed, so it may be desirable to simply extend the existing pic_struct syntax without removing the old options. In an embodiment, a new "frame repetition extension" value pic_struct=13 and a new syntax element num_frame_repetition_minus4 are added. Examples of the proposed syntax are described in Tables 12 and 13. For pic_struct values ​​0 to 12, the proposed syntax is the same as that in Table D.2 (as shown in the Appendix), so these values ​​are omitted for brevity.

[0148] Table 12: Example picture timing SEI message syntax, method 2

[0149]

[0150] num_frame_repetition_minus4 plus 4 indicates that when fixed_pic_rate_within_cvs_flag is equal to 1, the frame should be displayed consecutively on the display num_frame_repetition_minus4 plus 4 times, where the frame refresh interval is equal to DpbOutputElementalInterval[n], as given in Equation E-73.

[0151] Table 13: Example of revised pic_struct, method 2

[0152]

[0153] In HEVC, the parameter frame_field_info_present_flag is present in the Video Usability Information (VUI), but the syntax elements pic_struct, source_scan_type and duplicate_flag are in the pic_timing() SEI message. In an embodiment, it is proposed to move all relevant syntax elements to the VUI together with frame_field_info_present_flag. An example of the proposed syntax is depicted in Table 14.

[0154] Table 14: Example VUI parameter syntax supporting revised pic_struct syntax elements

[0155]

[0156] Alternative signaling of shutter angle information

[0157] When dealing with variable frame rates, it is desirable to determine both the desired frame rate and the desired shutter angle. In existing video coding standards, "Video Usability Information" (VUI) provides information essential for proper display of video content, such as aspect ratio, colour primaries, chroma subsampling, etc. If the fixed pic rate is set to 1, the VUI can also provide frame rate information; however, shutter angle information is not supported. An embodiment allows different shutter angles to be used for different temporal layers, and the decoder can use the shutter angle information to improve the final appearance on the display.

[0158] For example, HEVC supports temporal sub-layers that essentially go from a higher frame rate to a lower frame rate using frame dropping techniques. The main problem is that the effective shutter angle decreases with each frame dropped. As an example, 60fps can be obtained from a 120fps video by dropping every other frame; 30fps can be obtained by dropping 3 out of 4 frames; and 24fps can be obtained by dropping 4 out of 5 frames. Assuming a full 360-degree shutter at 120Hz, with simple frame dropping, the shutter angles for 60fps, 30fps, and 24fps are 180 degrees, 90 degrees, and 72 degrees, respectively [3]. Experience has shown that shutter angles below 180 degrees are generally unacceptable, especially at frame rates below 50Hz. By providing shutter angle information, for example, if the display is expected to produce a cinematic effect from a 120Hz video with a reduced shutter angle for each temporal layer, intelligent techniques can be applied to improve the final appearance.

[0159] In another example, one may want to support different temporal layers at the same shutter angle (e.g., a 60fps sub-bitstream within a 120fps bitstream). The main problem then is that when a 120fps video is displayed at 120Hz, the even / odd frames have different effective shutter angles. If the display has the relevant information, smart techniques can be applied to improve the final appearance. An example of the proposed syntax is shown in Table 15, where the E.2.1 VUI parameter syntax table of HEVC (reference [1]) is modified to support the mentioned shutter angle information. It should be noted that in another embodiment, the shutter_angle syntax can alternatively be expressed as a ratio of frame rate to shutter speed (see equation (1)) instead of in absolute degrees.

[0160] Table 15: Example VUI parameter syntax for supported shutter angles

[0161]

[0162] vui_shutter_angle_info_present_flag equal to 1 specifies that shutter angle information is present in the vui_parameters() syntax structure. vui_shutter_angle_info_present_flag equal to 0 specifies that shutter angle information is not present in the vui_parameters() syntax structure.

[0163] fixed_shutter_angle_within_cvs_flag equal to 1 specifies that the shutter angle information is the same for all temporal sub-layers in the CVS. fixed_shutter_angle_within_cvs_flag equal to 0 specifies that the shutter angle information may be different for all temporal sub-layers in the CVS.

[0164] fixed_shutter_angle specifies the shutter angle in degrees within the CVS. The value of fixed_shutter_angle should be in the range of 0 to 360.

[0165] sub_layer_shutter_angle[i] specifies the shutter angle in degrees when HighestTid is equal to i. The value of sub_layer_shutter_angle[i] should be in the range of 0 to 360.

[0166] Gradual frame rate update within a coded video sequence (CVS)

[0167] Experiments have shown that for HDR content displayed on an HDR display, in order to perceive the same motion judder as standard dynamic range (SDR) playback in a 100 nit display, the frame rate needs to be increased based on the brightness of the content. In most standards (AVC, HEVC, VVC, etc.), the video frame rate can be indicated in the VUI (contained in the SPS) using the vui_time_scale, vui_num_units_in_tick, and elemental_duration_in_tc_minus1[temporal_id_max] syntax elements, for example, as shown in Table 16 below (see Section E.2.1 in reference [1]).

[0168] Table 16: VUI syntax elements for indicating frame rate in HEVC

[0169]

[0170] As discussed in reference [1],

[0171] The variable ClockTick is obtained as follows and is called the clock cycle:

[0172] ClockTick=vui_num_units_in_tick÷vui_time_scale

[0173] picture_duration=ClockTick*(elemental_duration_in_tc_minus1[i]+1)

[0174] frame_rate=1 / pic_duration.

[0175] However, the frame rate can only be changed at certain times, for example, in HEVC, only at an intra random access point (IRAP) frame or at the start of a new CVS. For HDR playback, in the presence of fade-ins or fade-outs, the frame rate or picture duration of each picture may need to be changed because the brightness of the picture changes from frame to frame. In order to allow the frame rate or picture duration to be refreshed at any time (even on a frame-by-frame basis), in an embodiment, a new SEI message for "gradual refresh rate" is proposed, as shown in Table 17.

[0176] Table 17: Example syntax for supporting gradual refresh frame rates in SEI messaging

[0177]

[0178] The definition of the new syntax num_units_in_tick is the same as vui_num_units_in_tick, and the definition of time_scale is the same as vui_time_scale.

[0179] num_units_in_tick is the number of time units of a clock operating at a frequency time_scale Hz corresponding to one increment of the clock cycle counter (called a clock cycle). num_units_in_tick should be greater than 0. The clock cycle in seconds is equal to the quotient of num_units_in_tick divided by time_scale. For example, when the picture rate of the video signal is 25 Hz, time_scale can be equal to 27000000, and num_units_in_tick can be equal to 1080000, and thus the clock cycle can be equal to 0.04 seconds.

[0180] time_scale is the number of time units that pass in one second. For example, a time coordinate system that uses a 27MHz clock to measure time has a time_scale of 27000000. The value of time_scale should be greater than 0.

[0181] The picture duration for a picture using the gradual_refresh_rate SEI message is defined as:

[0182] picture_duration=num_units_in_tick÷time_scale.

[0183] Signaling of shutter angle information via SEI messaging

[0184] As previously discussed, Table 15 provides an example of VUI parameter syntax that supports shutter angle. As an example, but not limiting, Table 18 lists the same syntax elements, but now as part of the SEI message for shutter angle information. It should be noted that SEI messaging is used as an example only, and similar messaging can be constructed at other layers of high-level syntax, such as sequence parameter set (SPS), picture parameter set (PPS), slice or tile group header, etc.

[0185] Table 18: Example SEI message syntax for shutter angle information

[0186]

[0187] The shutter angle is usually expressed in degrees from 0 to 360 degrees. For example, a shutter angle of 180 degrees indicates that the exposure duration is 1 / 2 the frame duration. The shutter angle can be expressed as: shutter_angle = frame_rate*360*shutter_speed, where shutter_speed is the exposure duration and frame_rate is the inverse of the frame duration. The frame_rate for a given temporal sublayer Tid can be indicated by num_units_in_tick, time_scale, elemental_duration_in_tc_minus1[Tid]. For example, when fixed_pic_rate_within_cvs_flag[Tid] is equal to 1:

[0188] frame_rate=time_scale / (num_units_in_tick*(elemental_duration-in_tc_minusl[Tid]+1)).

[0189] In some embodiments, the value of the shutter angle (e.g., fixed_shutter_angle) may not be an integer, for example, it may be 135.75 degrees. To enable greater precision, in Table 21, u(9) (unsigned 9 bits) may be replaced by u(16) or some other suitable bit depth (e.g., 12 bits, 14 bits, or more than 16 bits).

[0190] In some embodiments, it may be beneficial to express shutter angle information in "clock cycles". In VVC, the variable ClockTick is obtained as follows:

[0191] ClockTick=num_units_in_tick÷time_scale.(8)

[0192] Both the frame duration and the exposure duration can then be expressed as multiples or fractions of the clock period:

[0193] exposure_duration=fN*ClockTick, (9)

[0194] frame_duration=fM*ClockTick, (10)

[0195] Wherein, fN and fM are floating point values, and fN ≤ fM.

[0196] Then

[0197]

[0198] where the numerator and denominator are integers that approximate the ratio fN / fM.

[0199] Table 19 shows an example of SEI messaging indicated by equation (11). In this example, for a camera in the real world, the shutter angle must be greater than 0.

[0200] Table 19: Example SEI messaging for clock cycle based shutter angle information

[0201]

[0202] As discussed previously, the use of u*16) (unsigned 16 bits) for shutter angle accuracy is depicted as an example and corresponds to the following accuracy: 360 / 2 16 = 0.0055. The precision can be adjusted based on the actual application. For example, using u(8), the precision is 360 / 2 8 =1.4063.

[0203] Note - Shutter angles are expressed in degrees greater than 0 but less than or equal to 360. For example, a shutter angle of 180 degrees indicates an exposure duration of 1 / 2 the frame duration.

[0204] fixed_shutter_angle_within_cvs_flag equal to 1 specifies that the shutter angle value is the same for all temporal sub-layers in the CVS. fixed_shutter_angle_within_cvs_flag equal to 0 specifies that the shutter angle value may be different for all temporal sub-layers in the CVS.

[0205] fixed_shutter_angle_numer_minus1 plus 1 specifies the numerator used to obtain the shutter angle value. The value of fixed_shutter_angle_numer_minus1 should be in the range of 0 to 65535 (inclusive).

[0206] fixed_shutter_angledemom_minus1 plus 1 specifies the denominator used to obtain the shutter angle value. The value of fixed_shutter_angledemom_minus1 should be in the range of 0 to 65535 (inclusive).

[0207] The value of fixed_shutter_angle_numer_minus1 should be less than or equal to the value of fixed_shutter_angledemom_minus1.

[0208] The degree of the variable shutterAngle is obtained as follows:

[0209] shutterAngle=360*(fixed_shutter_angle_numer_minus1+1)÷(fixed_shutter_angle_demom_minus1+1))

[0210] sub_layer_shutter_angle_numer_minus1[i] plus 1 specifies the numerator used to obtain the shutter angle value when HighestTid is equal to i. The value of sub_layer_shutter_angle_numer_minus1[i] should be in the range of 0 to 65535 (inclusive).

[0211] sub_layer_shutter_angle_demom_minus1[i] plus 1 specifies the denominator used to obtain the shutter angle value when HighestTid is equal to i. The value of sub_layer_shutter_angle_demom_minus1[i] should be in the range of 0 to 65535 (inclusive).

[0212] The value of sub_layer_shutter_angle_numer_minus1[i] should be less than or equal to the value of sub_layer_shutter_angle_denom_minus1[i].

[0213] The degree of the variable subLayerShutterAngle[i] is obtained as follows:

[0214] subLayerShutterAngle[i]=360*

[0215] (sub_layer_shutter_angle_numer_minus1[i]+1)÷(sub_layer_shutter_angle_demom_minus1[i]+1)

[0216] In another embodiment, the frame duration (e.g., frame_duration) may be specified by some other means. For example, in DVB / ATSC, when fixed_pic_rate_within_cvs_flag[Tid] is equal to 1:

[0217] frame_rate=time_scale / (num_units_in_tick*(elemental_duration_in_tc_minus1[Tid]+1)), frame_duration=1 / frame_rate.

[0218] The syntax in some of Table 19 and subsequent tables assumes that the shutter angle will always be greater than zero; however, shutter angle = 0 can be used to signal a creative intent in which the content should be displayed without any motion blur. This can be the case for moving graphics, animations, CGI textures, and matte screens, among others. Thus, for example, signaling shutter angle = 0 can be used for mode decisions in a transcoder (e.g., to select a transcoding mode that preserves edges) and in a display that receives shutter angle metadata over a CTA interface or a 3GPP interface. For example, shutter angle = 0 can be used to indicate a display that should not perform any motion processing (such as denoising, frame interpolation, etc.). In such an embodiment, the syntax elements fixed_shutter_angle_numer_minus1 and sub_layer_shutter_angle_numer_minus1[i] can be replaced by the syntax elements fixed_shutter_angle_numer and sub_layer_shutter_angle_numer[i], where,

[0219] fixed_shutter_angle_numer specifies the numerator used to obtain the shutter angle value. The value of fixed_shutter_angle_numer should be in the range of 0 to 65535 (inclusive).

[0220] sub_layer_shutter_angle_numer[i] specifies the numerator used to get the shutter angle value when HighestTid is equal to i. The value of sub_layer_shutter_angle_numer[i] should be in the range of 0 to 65535 (inclusive).

[0221] In another embodiment, fixed_shutter_angledenom_minus1 and sub_layer_shutter_angle_denom_minus1[i] may also be replaced by the syntax elements fixed_shutter_angle_denom and sub_layer_shutter_angle_denom[i].

[0222] In an embodiment, the num_units_in_tick and time_scale syntax defined in the SPS may be reused by setting the general_hrd_parameters_present_flag equal to 1 in the VVC as depicted in Table 20. In this case, the SEI message may be renamed as the exposure duration SEI message.

[0223] Table 20: Example SEI messaging for signaling exposure duration

[0224]

[0225] fixed_exposure_duration_within_cvs_flag equal to 1 specifies that the effective exposure duration value is the same for all temporal sub-layers in the CVS. fixed_exposure_duration_within_cvs_flag equal to 0 specifies that the effective exposure duration value may not be the same for all temporal sub-layers in the CVS.

[0226] fixed_exposure_duration_numer_minus1 plus 1 specifies the numerator used to get the exposure duration value. The value of fixed_exposure_duration_numer_minus1 should be in the range of 0 to 65535, inclusive.

[0227] fixed_exposure_duration_demom_minus1 plus 1 specifies the denominator used to obtain the exposure duration value. The value of fixed_exposure_duration_demom_minus1 should be in the range of 0 to 65535, inclusive.

[0228] The value of fixed_exposure_during_numer_minus1 should be less than or equal to the value of fixed_exposure_duration_demom_minus1.

[0229] The variable fixedExposureDuration is obtained as follows:

[0230] fixedExposureDuration=(fixed-exposure_duration-numer_minus1+1)÷(fixed_exposure-duration_demom_minus1+1)*ClockTicks

[0231] sub_layer_exposure_duration_numer_minus1[i] plus 1 specifies the numerator used to obtain the exposure duration value when HighestTid is equal to i. The value of sub_layer_exposure_duration_numer_minus1[i] should be in the range of 0 to 65535, inclusive.

[0232] sub_layer_exposure_duration_demom_minus1[i] plus 1 specifies the denominator used to obtain the exposure duration value when HighestTid is equal to i. The value of sub_layer_exposure_duration_demom_minus1[i] should be in the range of 0 to 65535, inclusive.

[0233] The value of sub_layer_exposure_duration_numer_minus1[i] shall be less than or equal to the value of sub_layer_exposure_duration_demom_minus1[i].

[0234] The variable subLayerExposureDuration[i] of HigestTid is equal to i as follows:

[0235] subLayerExposureDuration[i]=(sub_layer_exposure_duration_numer_minus1[i]+1)÷(sub_layer_exposure_duration_demom_minus1[i]+1)*ClockTicks.

[0236] In another embodiment, as shown in Table 21, clockTick can be explicitly defined by the syntax elements expo_num_units_in_tick and expo_time_scale. The advantage here is that it does not depend on whether general_hrd_parameters_present_flag is set to 1 in VVC as in the previous embodiment, so

[0237] clockTick=expo_num_units_in_tick÷expo_time_scale.(12)

[0238] Table 21: Example SEI messaging for exposure time signaling

[0239]

[0240] expo_num_units_in_tick is the number of time units of a clock operating at frequency time_scale Hz that corresponds to one increment of the clock tick counter (called a clock cycle). expo_num_units_in_tick should be greater than 0. The clock cycle in seconds defined by the variable clockTick is equal to the quotient of expo_num_units_in_tick divided by expo_time_scale.

[0241] expo_time_scale is the number of time units that elapse in one second.

[0242] clockTick=expo_num_units_in_tick÷expo_time_scale.

[0243] Note: The following two syntax elements: expo_num_units_in_tick and expo_time_scale are defined to measure exposure duration.

[0244] Bitstream conformance requires that when num_units_in_tick and time_scale are present, clockTick shall be less than or equal to ClockTick.

[0245] fixed_exposure_duration_within_cvs_flag equal to 1 specifies that the effective exposure duration value is the same for all temporal sub-layers in the CVS. fixed_exposure_duration_within_cvs_flag equal to 0 specifies that the effective exposure duration value may be different for all temporal sub-layers in the CVS. When fixed_exposure_duration_within_cvs_flag is equal to 1, the variable fixedExposureDuration is set equal to clockTick.

[0246] sub_layer_exposure_duration_numer_minus1[i] plus 1 specifies the numerator used to obtain the exposure duration value when HighestTid is equal to i. The value of sub_layer_exposure_duration_numer_minus1[i] should be in the range of 0 to 65535, inclusive.

[0247] sub_layer_exposure_duration_demom_minus1[i] plus 1 specifies the denominator used to obtain the exposure duration value when HighestTid is equal to i. The value of sub_layer_exposure_duration_demom_minus1[i] should be in the range of 0 to 65535, inclusive.

[0248] The value of sub_layer_exposure_duration_numer_minus1[i] shall be less than or equal to the value of sub_layer_exposure_duration_demom_minus1[i].

[0249] The variable subLayerExposureDuration[i] of HigestTid is equal to i as follows:

[0250] subLayerExposureDuration[i]=(sub_layer_exposure_duration_numer_minus1[i]+1)÷(sub_layer_exposure_duration_denom_minus1[i]+1)*clockTick.

[0251] As discussed before, the syntax parameters sub_layer_exposure_duration_numer_minus1[i] and sub_layer_exposure_duration_denom_minus1[i] may also be replaced by sub_layer_exposure_duration_numer[i] and sub_layer_exposure_duration_denom[i].

[0252] In another embodiment, as shown in Table 22, the parameter ShutterInterval (ie, exposure duration) may be defined by syntax elements sii_num_units_in_shutter_interval and sii_time_scale, where:

[0253] ShutterInterval=sii_num_units_in-shutter_interval÷sii_time_scale. (13)

[0254] Table 22: Example SEI messaging for exposure duration (shutter interval information) signaling

[0255]

[0256] Shutter Interval Information SEI Message Semantics

[0257] The shutter interval information SEI message indicates the shutter interval of the associated video content prior to encoding and display—eg, for content captured by a camera, the amount of time to expose an image sensor to produce a picture.

[0258] sii_num_units_in_shutter_interval specifies the number of time units of a clock operating at a frequency sii_time_scale Hz corresponding to one increment of the shutter clock period counter. The shutter interval in seconds defined by the variable ShutterInterval is equal to the quotient of sii_num_units_in_shutter_interval divided by sii_time_scale. For example, when ShutterInterval is equal to 0.04 seconds, sii_time_scale can be equal to 27000000 and sii_num_units_in_shutter_interval can be equal to 1 080 000.

[0259] sii_time_scale specifies the number of time units that pass in one second. For example, a time coordinate system that uses a 27MHz clock to measure time would have a sii_time_scale of 27000000.

[0260] When the value of sii_time_scale is greater than 0, the value of ShutterInterval is specified by the following equation:

[0261] ShutterInterval=sii_num_units_in_shutter_interval÷sii_time_scale

[0262] Otherwise (the value of sii_time_scale is equal to 0), ShutterInterval should be interpreted as unknown or unspecified.

[0263] NOTE 1—A value of ShutterInterval equal to 0 may indicate that the associated video content contains screen captured content, computer generated content, or other non-camera captured content.

[0264] NOTE 2 - A value of ShutterInterval greater than the inverse of the coded picture rate (i.e., the coded picture interval) may indicate that the coded picture rate is greater than the picture rate at which the associated video content was created - for example, when the coded picture rate is 120 Hz and the picture rate of the associated video content prior to encoding and display is 60 Hz. The coding interval for a given temporal sublayer Tid may be indicated by ClockTick and elemental_duration_in_tc_minus1[Tid]. For example, when fixed_pic_rate_within_cvs_flag[Tid] is equal to 1, the picture interval for a given temporal sublayer Tid defined by the variable PictureInterval[Tid] may be specified by the following equation:

[0265] PictureInterval[Tid]=ClockTick*(elemental_duration_in_tc_minus1[Tid]+1).

[0266] fixed_shutter_interval_within_cvs_flag equal to 1 specifies that the value of ShutterInterval is the same for all temporal sublayers in the CVS. fixed_shutter_interval_within_cvs_flag equal to 0 specifies that the value of ShutterInterval may be different for all temporal sublayers in the CVS.

[0267] sub_layer_shutter_interval_numer[i] specifies the numerator used to get the sub-layer shutter interval in seconds defined by the variable subLayerShutterInterval[i] when HighestTid is equal to i.

[0268] sub_layer_shutter_interval_denom[i] specifies the denominator used to obtain the sub-layer shutter interval in seconds defined by the variable subLayerShutterInterval[i] when HighestTid is equal to i.

[0269] The value of subLayerShutterInterval[i] equal to i for HighestTid is obtained as follows: When the value of fixed_shutter_interval_within_cvs_flag is equal to 0 and the value of sub_layer_shutter_interval_denom[i] is greater than 0:

[0270] subLayerShutterInterva1[i]=ShutterInterval*sub_layer_shutter_interval_numer[i]÷sub_layer_shutter_interval_denom[i]

[0271] Otherwise (the value of sub_layer_shutter_interval_denom[i] is equal to 0), subLayerShutterInterval[i] should be interpreted as unknown or unspecified. When the value of fixed_shutter_interval_within_Cvs_flag is not equal to 0,

[0272] subLayerShutterInterval[i]=ShutterInterval.

[0273] In an alternative embodiment, a single value is used instead of using a numerator and denominator for signaling the sub-layer shutter interval. An example of such a syntax is shown in Table 23.

[0274] Table 23: Example SEI messaging for shutter interval signaling

[0275]

[0276] Shutter Interval Information SEI Message Semantics

[0277] The shutter interval information SEI message indicates the shutter interval of the associated video content prior to encoding and display—eg, for content captured by a camera, the amount of time to expose an image sensor to produce a picture.

[0278] sii_num_units_in_shutter specifies the number of time units of a clock operating at a frequency sii_time_scale Hz corresponding to one increment of the shutter clock period counter. The shutter interval in seconds, defined by the variable ShutterInterval, is equal to the quotient of sii_num_units_in_shutter_interval divided by sii_time_scale. For example, when ShutterInterval is equal to 0.04 seconds, sii_time_scale can be equal to 27000000 and sii_num_units_in_shutter_interval can be equal to 1 080000.

[0279] sii_time_scale specifies the number of time units that pass in one second. For example, a time coordinate system that uses a 27MHz clock to measure time would have a sii_time_scale of 27000000.

[0280] When the value of sii_time_scale is greater than 0, the value of ShutterInterval is specified by the following equation:

[0281] ShutterInterval=sii_num_units_in_shutter_interval÷sii_time_scale

[0282] Otherwise (the value of sii_time_scale is equal to 0), ShutterInterval should be interpreted as unknown or unspecified.

[0283] NOTE 1 A value of ShutterInterval equal to 0 may indicate that the associated video content contains screen captured content, computer generated content, or other non-camera captured content.

[0284] NOTE 2 A value of ShutterInterval greater than the inverse of the coded picture rate (i.e., the coded picture interval) may indicate that the coded picture rate is greater than the picture rate at which the associated video content was created—for example, when the coded picture rate is 120 Hz and the picture rate of the associated video content prior to encoding and display is 60 Hz. The coded picture interval for a given temporal sublayer Tid may be indicated by ClockTick and elemental_duration_in_tc_minus1[Tid]. For example, when fixed_pic_rate_within_cvs_flag[Tid] is equal to 1, the picture interval for a given temporal sublayer Tid defined by the variable PictureInterval[Tid] may be specified by the following equation:

[0285] PictureInterval[Tid]=ClockTick*(elemental_duration_in_tc_minus1[Tid]+1).

[0286] fixed_shutter_interval_within_cvs_flag equal to 1 specifies that the value of ShutterInterval is the same for all temporal sublayers in the CVS. fixed_shutter_interval_within_cvs_flag equal to 0 specifies that the value of ShutterInterval may be different for all temporal sublayers in the CVS.

[0287] sub_layer_num_units_in_shutter_interval[i] specifies the number of time units of a clock operating at a frequency sii_time_scale Hz corresponding to one increment of the shutter clock period counter. When HighestTid is equal to i, the sublayer shutter interval in seconds defined by the variable subLayerShutterInterval[i] is equal to the quotient of sub_layer_num_units_in_shutter_interval[i] divided by sii_time_scale.

[0288] When the value of fixed_shutter_interval_within_cvs_flag is equal to 0 and the value of sii_time_scale is greater than 0, the value of subLayerShutterInterval[i] is specified by the following equation:

[0289] subLayerShutterInterval[i]=sub_layer_num_units_in_shutter_interval[i]÷sii_time_scale

[0290] Otherwise (the value of sii_time_scale is equal to θ), subLayerShutterInterval[i] should be interpreted as unknown or unspecified. When the value of fixed_shutter_interval_within_cvs_flag is not equal to 0,

[0291] subLayerShutterInterval[i]=ShutterInterval.

[0292] Table 24 provides an overview of the six methods discussed in Tables 18 to 23 for providing SEI information transfer related to shutter angle or exposure duration.

[0293] Table 24: Overview of SEI messaging method for signaling shutter angle information

[0294]

[0295] Variable frame rate signaling

[0296] As discussed in U.S. Provisional Application No. 62 / 883,195 filed on August 6, 2019, in many applications it is desirable for a decoder to support playback at variable frame rates. Frame rate adaptation is typically part of the operation in a hypothetical reference decoder (HRD), as described, for example, in Appendix C of reference [2]. In an embodiment, it is proposed to signal, via SEI messaging or other means, a syntax element that defines the picture presentation time (PPT) as a function of a 90kHz clock. This is a repetition of the nominal decoder picture buffer (DPB) output time as specified in the HRD, but now using the 90kHz ClockTicks precision as specified in MPEG-2 Systems. The benefits of this SEI message are that: a) if HRD is not enabled, the PPT SEI message can still be used to indicate the timing of each frame; b) the conversion of bitstream timing and system timing can be facilitated.

[0297] Table 25 describes an example of the syntax of the proposed PPT timing message, which matches the syntax of the presentation timestamp (PTS) variable used in MPEG-2 Transport (H.222) (reference [4]).

[0298] Table 25: Example syntax for image presentation time messaging

[0299] picture_presentation_time(payloadSize){ Descriptors PPT u(33) }

[0300] PPT (picture presentation time)

[0301] - The presentation time shall be related to the decoding time as follows: The PPT is a 33-bit number encoded in three separate fields. The PPT indicates the presentation time tpn(k) in the system target decoder of presentation unit k of elementary stream n. The value of the PPT is specified in units of system clock frequency periods divided by 300 (resulting in 90 kHz). The picture presentation time is obtained from the PPT according to the equation below.

[0302] PPT(k)=((system_clock_frequcy×tp n (k)) / 300)%2 33

[0303] Among them, tp n (k) is the presentation unit P n (k) presentation time.

[0304] References

[0305] Each of the references listed herein is incorporated by reference in its entirety.

[0306] [1] High efficiency video coding, H.265, H series, Coding of moving video, ITU, (02 / 2018).

[0307] [2] B. Bross, J. Chen and S. Liu, “Versatile Video Coding (Draft 5)”, JVET output document, JVET-N1001, v5, uploaded on May 14, 2019.

[0308] [3] C. Carbonara, J. De Filippis, M. Korpi, “High Frame Rate Capture and Production,” SMPTE 2015 Annual Technical Conference and Exhibition, October 26–29, 2015.

[0309] [4] Infrastructure of audiovisual services-Transmission multiplexing and synchronization, H.222.0, H series, Generic coding of moving pictures and associated audio information: Systems, ITU, 08 / 2018.

[0310] Example Computer System Implementation

[0311] Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more such systems, devices, or components. The computer and / or IC may execute, control, or implement instructions related to frame rate scalability, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to frame rate scalability described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0312] Certain embodiments of the present invention include a computer processor that executes software instructions, and the software instructions cause the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement the method related to frame rate scalability as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention can also be provided in the form of a program product. The program product may include any non-transient and tangible medium carrying a set of computer-readable signals, and the set of computer-readable signals includes instructions, and the instructions cause the data processor to perform the method of the present invention when executed by a data processor. According to the program product of the present invention, any of various non-transient and tangible forms can be used. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash memory RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.

[0313] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to the components (including references to "means") should be interpreted to include any components that perform the functions of the described components as equivalents (e.g., functionally equivalent) to the components, including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.

[0314] Equivalents, extensions, alternatives and miscellaneous

[0315] Thus, example embodiments related to frame rate scalability are described. In the foregoing description, embodiments of the present invention have been described with reference to many specific details that may vary depending on the implementation. Therefore, the sole and exclusive indication of what the invention is and what the applicant intends the invention to be is the set of claims granted in accordance with this application in the specific form in which the claims are granted, including any subsequent corrections. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Therefore, limitations, elements, characteristics, features, advantages or attributes that are not expressly recited in a claim should not limit the scope of such claim in any way. Therefore, the specification and drawings should be regarded in an illustrative and not restrictive sense.

[0316] appendix

[0317] This appendix provides a copy of Table D.2 and the associated pic_struct related information from the H.265 specification (reference [1]).

[0318] Table D.2 - Explanation of pic_struct

[0319]

[0320] Semantics of pic_struct syntax elements

[0321] The pic_struct indicates whether the picture should be displayed as a frame or as one or more fields, and for display of frames, when fixed_pic_rate_within_cvs_flag is equal to 1, may indicate a frame doubling or tripling repetition period for the display using a fixed frame refresh interval equal to DpbOutputElementalInterval[n] as given by Equation E-73. The interpretation of the pic_struct is specified in Table D.2. The pic_struct values ​​not listed in Table D.2 are reserved for future use by ITU-T|ISO / IEC and shall not be present in bitstreams conforming to this version of this specification. A decoder shall ignore the reserved values ​​of pic_struct.

[0322] When present, bitstream conformance requires that the value of pic_struct shall be constrained such that exactly one of the following conditions is true:

[0323] - For all pictures in CVS, the value of pic_struct is equal to 0, 7 or 8.

[0324] - For all pictures in CVS, the value of pic_struct is equal to 1, 2, 9, 10, 11 or 12.

[0325] - For all pictures in CVS, the value of pic_struct is equal to 3, 4, 5 or 6.

[0326] When fixed_pic_rate_within_cvs_flag is equal to 1, frame doubling is indicated by pic_struct equal to 7, indicating that the frame should be displayed twice consecutively on the display, with the frame refresh interval equal to DpbOutputElementalInterval[n] as given in Equation E-73, and frame tripling is indicated by pic_struct equal to 8, indicating that the frame should be displayed three times consecutively on the display, with the frame refresh interval equal to DpbOutputElementalInterval[n] as given in Equation E-73.

[0327] NOTE 3—Frame doubling may be used to facilitate display of, for example, 25 Hz progressive scan video on a 50 Hz progressive scan display or 30 Hz progressive scan video on a 60 Hz progressive scan display. Use of frame doubling and frame tripling in alternating combinations of every other frame may be used to facilitate display of 24 Hz progressive scan video on a 60 Hz progressive scan display.

[0328] The field association indicator (pic_struct equals 9 to 12) provides a hint to associate fields of complementary parity together as a frame. The parity of a field can be top or bottom, and the parities of two fields are considered complementary when the parity of one field is top and the parity of the other field is bottom.

[0329] When frame_field_info_present_flag is equal to 1, the constraints specified in the third column of bitstream conformance requirements Table D.2 shall apply.

[0330] NOTE 4 - When frame_field_info_present_flag is equal to 0, then in many cases a default value may be inferred or indicated by other means. In the absence of other indication of the intended display type of the picture, when frame_field_info_present_flag is equal to 0, the decoder shall infer that the value of pic_struct is equal to 0.

Claims

1. A non-transitory processor-readable medium having instructions stored thereon, wherein: When the instructions are executed by a processor, the processor performs a method for decoding a video bitstream, the method comprising: Receiving a coded video bitstream, the coded video bitstream comprising a coded picture portion and a signaling portion, the coded picture portion comprising a coding of a video picture sequence, and the signaling portion comprising a shutter interval parameter, wherein the shutter interval parameter comprises: a shutter interval time scale parameter indicating the number of time units that elapse in one second; a shutter interval clock period parameter indicating a number of time units of a clock operating at a frequency of the shutter interval time scale parameter; a shutter interval duration flag indicating whether exposure duration information is fixed for all temporal sub-layers in the coded picture portion; and If the shutter interval duration flag indicates that the exposure duration information is fixed, decoding a decoded version of the video picture sequence for all the temporal sub-layers in the coded picture portion by calculating an exposure duration value based on the shutter interval time scale parameter and the shutter interval clock period parameter, otherwise the signaling portion comprising one or more sub-layer parameter arrays, wherein values ​​in the one or more sub-layer parameter arrays combined with the shutter interval time scale parameter are used to calculate, for each sub-layer, a corresponding sub-layer exposure duration value for displaying a decoded version of a temporal sub-layer of the sequence of video pictures; and decoding the video picture sequence based on the shutter interval parameter, wherein the one or more sub-layer parameter arrays include an array of sub-layer shutter interval clock period values, each sub-layer shutter interval clock period value indicating the number of time units of the clock operating at the frequency of the shutter interval time scale parameter, The exposure duration value is calculated as the quotient of the shutter interval clock cycle parameter divided by the shutter interval time scale parameter.

2. The non-transitory processor-readable medium of claim 1, wherein: The signaling part includes a supplemental enhancement information SEI message delivery part or a video user information VUI message delivery part.