Frame Rate Scalable Video Coding

Abstract

To provide a frame-rate scalable video coding method.SOLUTION: A decoder reconstructs output video sequence C at a desired frame rate and shutter angle, which may change from scene to scene, with an input video sequence A with variable frame rate and variable shutter angle which is represented in a coded bitstream B with a fixed frame rate. For example, to construct sequence B, some of the input frames are duplicated, some are coded as they are (with no duplication), and some are copied four times. Then, to construct sequence C, any one frame from each set of duplicate frames is selected to generate output frames, matching the original frame rate and shutter angle.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority from U.S. Provisional Application No. 62 / 816,521, filed March 11, 2019, U.S. Provisional Application No. 62 / 850,985, filed May 21, 2019, U.S. Provisional Application No. 62 / 883,195, filed August 6, 2019, and U.S. Provisional Application No. 62 / 904,744, filed September 24, 2019, each of which is incorporated by reference in its entirety.

[0002] More particularly, an embodiment of the present invention relates to frame rate scalable video coding. [Background technology]

[0003] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest gray (black) to brightest white (highlight). In this sense, DR relates to "scene-referred" intensity. DR may also relate to the ability of a display device to properly or approximately render an intensity range of a particular width. In this sense, DR relates to "display-referred" intensity. Unless expressly designated to have a particular meaning at any point in the description herein, it should be presumed that the terms may be used in either sense, e.g., interchangeably.

[0004] As used herein, the term high dynamic range (HDR) refers to a DR width that spans 14-15 orders of magnitude of the human visual system (HVS). In practice, the DR that allows humans to simultaneously perceive a wide range of intensity ranges can be somewhat truncated with respect to HDR.

[0005] In practice, an image contains one or more color components (e.g., luma Y and chroma Cb and Cr), each color component represented with n bits of precision per pixel (e.g., n=8). Using linear luminance encoding, images with n≦8 (e.g., color 24-bit JPEG images) are considered standard dynamic range (SDR) images, and images with n>8 are considered enhanced dynamic range images. HDR images can also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

[0006] Currently, the delivery of video high dynamic range content such as Dolby Vision from Dolby Labs or HDR10 on Blue-Ray is limited to 4K resolution (e.g., 4096×2160 or 3840×2160) and 60 frames per second (fps) by the capabilities of many playback devices. It is expected that in future versions, content up to 8K resolution (e.g., 7680×4320) and 120 fps may be available for delivery and playback. To simplify the HDR playback content ecosystem such as Dolby Vision, it is desirable for future content types to be compatible with existing playback devices. Ideally, content producers should be able to adopt and deliver future HDR technologies without having to derive and deliver special versions of content compatible with existing HDR devices (e.g., HDR10 or Dolby Vision). As appreciated by the inventors, improved techniques for scalable delivery of video content, particularly HDR content, are desired.

[0007] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise noted, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, it should not be inferred that problems identified with one or more of the approaches have been recognized in any prior art based on this section, unless otherwise noted. [Brief description of the drawings]

[0008] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which: [Figure 1] 1 illustrates an example process of a video delivery pipeline. [Diagram 2] 4 illustrates an exemplary process for combining successive original frames to render a target frame rate at a target shutter angle, according to one embodiment of the present invention. [Diagram 3] 1 illustrates an exemplary display of an input sequence with a variable input frame rate and a variable shuttle angle within a container with a fixed frame rate, according to one embodiment of the present invention. [Figure 4] 13 shows an exemplary display for temporal scalability at various frame rates and shutter angles with backward compatibility according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Exemplary embodiments relating to frame rate scalability for video encoding are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obfuscating the present invention.

[0010] overview Exemplary embodiments described herein relate to frame rate scalability in video coding. In one embodiment, a system having a processor receives an encoded bitstream including encoded video frames, where one or more encoded frames are encoded at a first frame rate and a first shutter angle. The processor receives a first flag indicating a presence of a group of encoded frames to be decoded at a second frame rate and a second shutter angle, accesses from the encoded bitstream values ​​of the second frame rate and the second shutter angle for the group of encoded frames, and generates a decoded frame at the second frame rate and the second shutter angle based on the group of encoded frames, the first frame rate, the first shutter angle, the second frame rate, and the second shutter angle.

[0011] In a second embodiment, there is provided a decoder having a processor, receiving an encoded bitstream including a group of encoded video frames, where all of the encoded video frames within the encoded bitstream are encoded at a first frame rate; receiving a number of the combined N frames; receiving a value for a baseline frame rate; accessing a group of N consecutive encoded frames, where an i-th encoded frame in the group of N consecutive encoded frames represents an average of the first i-th input video frames encoded at the encoder at a baseline frame rate and an i-th shutter angle based on a first shutter angle and a first frame rate, where i=1, 2, ...N; accessing from the encoded bitstream or from user input values ​​for the second frame rate and the second shutter angle to decode a group of N consecutive encoded frames at the second frame rate and the second shutter angle; generating a decoded frame at a second frame rate and a second shutter angle based on the group of N consecutive encoded frames, the first frame rate, the first shutter angle, the second frame rate, and the second shutter angle.

[0012] In a third embodiment, the encoded video stream structure is: a coded picture section including a coding of a sequence of video pictures; A signaling section, a shutter interval time scale parameter, indicating the number of time units that elapse in one second; a shutter interval clock tick parameter indicating the number of time units of a clock running at the frequency of the shutter interval time scale parameter, where the shutter interval divided by the shutter interval time scale parameter indicates an exposure duration value; a shutter interval duration flag indicating whether the exposure duration information is fixed for all temporal sub-layers in the coded picture section; and If the shutter interval duration flag indicates that the exposure duration information is fixed, then the coded version of the sequence of video pictures for every temporal sub-ray in the coded picture section is decoded by calculating exposure duration values ​​based on the shutter interval time scale parameter and the shutter interval clock tick parameter; otherwise, The signaling section includes one or more arrays of sub-layer parameters, and values ​​in the one or more arrays of sub-layer parameters combined with the shutter interval time scale parameter are used to calculate a corresponding sub-layer exposure duration value for each sub-layer to display a decoded version of the temporal sub-layers of the sequence of video pictures.

[0013] Example video delivery processing pipeline FIG. 1 illustrates an exemplary process of a conventional video distribution pipeline (100) showing various stages from video capture to video content display. A series of video frames (102) are captured or generated using an image generation block (105). The video frames (102) may be digitally captured (e.g., by a digital camera) or computer generated (e.g., using computer animation) to provide video data (107). Alternatively, the video frames (102) may be captured on film by a film camera. The film is converted to a digital format to provide the video data (107). In a production phase (110), the video data (107) is edited to provide a video production stream (112).

[0014] The video data of the production stream (112) is then provided to a processor at block (115) for post-production editing. Post-production editing at block (115) may include adjusting or modifying the color or brightness in certain areas of the image to enhance image quality or achieve a particular look of the image according to the video creator's creative intent. This is called "color timing" or "color grading". Other editing (e.g., scene selection and sequencing, image cropping, adding computer-generated visual special effects, judder or blur control, frame rate control, etc.) may be performed at block (115) and the video image is displayed on a reference display (125) during the final version of the production for distribution. Following post-production (115), the video data of the final production (117) may be delivered to an encoding block (120) for downstream delivery to decoding and playback devices such as television sets, set-top boxes, movie theaters, etc. In some embodiments, the encoding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other distribution formats, to generate an encoded bitstream (122). At the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a decoded signal (132) that represents an identical or approximation of the signal (117). The receiver may be attached to a target display (140), which may have characteristics quite different from the reference display (125). In that case, a display management block (135) may be used to map the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a display map signal (137).

[0015] Scalable Coding Scalable coding is already part of many video coding standards such as MPEG-2, AVC, and HEVC. In an embodiment of the present invention, scalable coding is extended to improve performance and flexibility, especially as it relates to very high resolution HDR content.

[0016] As used herein, the term "shutter angle" refers to an adjustable shutter setting that controls the percentage of time the film is exposed to light during each frame interval. For example, in one embodiment,

number

[0017] The term has its origins in traditional mechanical rotating shutters, but modern digital cameras can also adjust the shutter electronically. Cinematographers can use the shutter angle to control the amount of motion blur or judder recorded in each frame. Note that instead of "exposure time", alternative terms such as "exposure duration", "shutter interval", or "shutter speed" may be used. Similarly, instead of "frame interval", the term "frame duration" may be used. Alternatively, "frame interval" may be replaced by "1 / frame rate". Exposure time values ​​are usually less than or equal to the duration of a frame. For example, a shutter angle of 180 degrees indicates that the exposure time is half the frame duration. In some situations, the exposure time may be longer than the frame duration of the encoded video, for example, if the encoded frame rate is 120 fps and the frame rate of the associated video content before encoding and display is 60 fps.

[0018] Consider, without limitation, an embodiment in which original content is shot (or generated) at an original frame rate (e.g., 120 fps) with a 360 degree shutter angle. A receiving device can then render video output at various frame rates below the original frame rate by intelligently combining the original frames, for example, by averaging or other operations known in the art.

[0019] Although the combining process may be performed on non-linearly encoded signals (e.g., using gamma, PQ or HLG), the best image quality is obtained by combining the frames in the linear light domain by first converting the non-linearly encoded signals to a linear light representation, then combining the converted frames, and finally re-encoding the output with a non-linear transfer function. This process provides a more accurate simulation of physical camera exposure than combining in the non-linear domain.

[0020] In general, the process of combining frames can be expressed in terms of original frame rate, target frame rate, target shutter angle, and number of frames to be combined as follows:

number

number

[0021] where n_frames is the number of combined frames, original_frame_rate is the frame rate of the original content, target_frame_rate is the rendered frame rate (where target_frame_rate≦original_frame_rate), and target_shutter_angle indicates the amount of motion blur desired. In this example, the maximum value of target_shutter_angle is 360 degrees, which corresponds to maximum motion blur. The minimum value of target_shutter_angle can be expressed as 360*(target_frame_rate / original_frame_rate), which corresponds to minimum motion blur. The maximum value of n_frames can be expressed as (original_frame_rate / target_frame_rate). The values ​​of target_frame_rate and target_shutter_angle should be selected such that the value of n_frame is a non-zero integer.

[0022] In the special case where the original frame rate is 120 fps, equation (2) becomes:

number

number

[0023] 2 illustrates an exemplary process of combining consecutive original frames to render a target frame rate at a target shutter angle, according to one embodiment. Given an input sequence (205) at 120 fps and a shutter angle of 360 degrees, the process generates an output video sequence (210) at 24 fps and a shutter angle of 216 degrees by combining three of the input frames (e.g., the first three consecutive frames) in a set of five consecutive frames and dropping the other two. Note that in some embodiments, output frame 01 of (210) may be generated by combining any of the input frames (205), such as frames 1, 3, and 5, or frames 2, 4, and 5, although combining consecutive frames is believed to result in a better quality video output.

[0024] For example, it is desirable to support original content with variable frame rates to manage artistic and stylistic effects. Also, it is desirable for the variable input frame rates of the original content to be packaged in a "container" with a fixed frame rate to simplify content generation, exchange, and distribution. As an example, three embodiments of how to represent variable frame rate video data in a fixed frame rate container are presented. For clarity, and without limitation, the following description uses a fixed 120 fps container, but the approach can be easily extended to alternative frame rate containers.

[0025] First embodiment (Variable Frame Rate) The first embodiment is an explicit description of original content with variable (non-constant) frame rates packaged into a container with a constant frame rate. For example, original content with different frame rates, e.g., 24, 30, 40, 60, or 120 fps for different scenes, may be packaged into a container with a constant frame rate of 120 fps. In this example, each input frame can be duplicated 5x, 4x, 3x, 2x, or 1x times and packaged into a common 120 fps container.

[0026] Figure 3 shows an example of an input video sequence A with variable frame rate and variable shutter angle, represented by an encoded bitstream B with a fixed frame rate. Then, at the decoder, the decoder reconstructs an output video sequence C with a desired frame rate and shutter angle, which may vary from scene to scene. For example, as shown in Figure 3, to construct sequence B, some of the input frames are duplicated, some are coded as is (without duplication), and some are copied four times. Then, to construct sequence C, an arbitrary one of the duplicated frames is selected to generate an output frame with a matching original frame rate and shutter angle.

[0027] In this embodiment, metadata is inserted into the bitstream to indicate the original (base) frame rate and shutter angle. The metadata can be signaled using high level syntax such as the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), slice or tile group headers. The presence of metadata allows the encoder and decoder to perform useful functions such as: a) The encoder can ignore the duplicated frame, thereby increasing the encoding speed and simplifying the process. For example, all coding tree units (CTUs) in the duplicated frame can be encoded using SKIP mode and reference index 0 in LIST0 of reference frames, which refers to the decoded frame to which the duplicated frame is copied. b) The decoder can bypass decoding of duplicate frames, thereby simplifying the process: for example, metadata in the bitstream can indicate that a frame is a duplicate of a previously decoded frame, which the decoder can recreate by copying and without decoding a new frame. c) By indicating the base frame rate, the playback device can optimize downstream processes, for example by adjusting frame rate conversion or noise reduction algorithms.

[0028] This embodiment allows end users to view content rendered at the frame rate intended by the content creator. This embodiment does not provide backward compatibility with devices that do not support the container's frame rate, e.g., 120 fps.

[0029] Tables 2 and 3 show example syntax for the Raw Byte Sequence Payload (RBSP) of the Sequence Parameter Set and Tile Group Header, where the proposed new syntax elements are shown in highlighted font. The remaining syntax follows the syntax in the proposed specification for the Generic Video Codec (VVC) (Reference [2]).

[0030] As an example, in SPS (see Table 2) you can add a flag to enable variable frame rates. sps_vfr_enabled_flag equal to 1 specifies that the Coded Video Sequence (CVS) can contain variable frame rate content. sps_vfr_enabled_flag equal to 0 specifies that the CVS can contain constant frame rate content. In the tile_group header() (see Table 3), tile_group_vrf_info_present_flag equal to 1 specifies that the syntax elements tile_group_true_fr and tile_group_shutterangle are present in the syntax. tile_group_vrf_info_present_flag equal to 0 specifies that the syntax elements tile_group_true_fr and tile_group_shutterangle are not present in the syntax. If tile_group_vrf_info_present_flag is not present, it is implied to be 0. tile_group_true_fr indicates the actual frame rate of the video data transmitted in this bitstream. tile_group_shutterangle indicates the shutter angle corresponding to the actual frame rate of the video data transmitted in this bitstream. tile_group_skip_flag equal to 1 specifies that the current tile group will be copied from another tile group. tile_group_skip_flag equal to 0 specifies that the current tile group will not be copied from another tile group. tile_group_copy_pic_order_cnt_lsb specifies the picture order count modulo MaxPicOrderCntLsb of previously decoded pictures that the current picture copies when tile_group_skip_flag is set to 1. [Table 2] [Table 3-1] [Table 3-2]

[0031] Second embodiment - fixed frame rate container The second embodiment enables a use case where an original content with a fixed frame rate and shutter angle can be rendered by a decoder with alternative frame rates and variable simulated shutter angles, as illustrated in FIG. 2. For example, if the original content has a frame rate of 120 fps and a shutter angle of 360 degrees (meaning the shutter is open for 1 / 120 seconds), the decoder can render multiple frame rates below 120 fps. For example, to decode 24 fps with a simulated shutter angle of 216 degrees, as described in Table 1, the decoder can combine three decoded frames and display at 24 fps. Table 4 expands on Table 1 and shows how to combine different numbers of encoded frames to render at an output target frame rate and desired target shutter angle. Combining the frames can be performed by simple pixel averaging, by weighted pixel averaging where pixels from one frame are weighted more than pixels from other frames and the sum of all weights is 1, or by other filter interpolation schemes known in the art. In Table 4, the function Ce(a,b) indicates a combination of the encoding frames a to b, and this combination can be performed by averaging, weighted averaging, filtering, or the like. [Table 4-1] [Table 4-2]

[0032] If the target shutter angle value is less than 360 degrees, the decoder can combine different sets of decoded frames. For example, from Table 1, given an original stream at 120 fps and 360 degrees, to generate a stream at 40 fps and 240 degree shutter angle, the decoder needs to combine two of three possible frames. Thus, it can combine either the first and second frames, or the second and third frames. The choice of which frames to combine can be described in terms of a "decoding phase" expressed as follows:

number

number

[0033] In general, decode_phase_idx is in the range of [0, n_frames_max-n_frames]. For example, for an original sequence at 120 fps and 360 degree shutter angle, for a target frame rate of 40 fps at 240 degree shutter angle, n_frames_max=120 / 40=3. From equation (2), n_frames=2, so decode_phase_idx is in the range of [0,1]. Thus, decode_phase_idx=0 indicates to select frames with indexes 0 and 1, and decode_phase_idx=1 indicates to select frames with indexes 1 and 2.

[0034] In this embodiment, the rendered variable frame rate intended by the content creator may be signaled as metadata, such as a supplemental enhancement information (SEI) message, or as video usability information (VUI). Optionally, the rendered frame rate may be controlled by the receiver or the user. An example of frame rate conversion SEI messaging specifying the content creator's preferred frame rate and shutter angle is shown in Table 5. The SEI message may also indicate whether the combined frames are performed in the coded signal domain (e.g., gamma, PQ, etc.) or in the linear light domain. Note that post-processing requires a frame buffer in addition to the decoder picture buffer (DPB). The SEI message may indicate the number of additional frame buffers required, or some alternative method for combining the frames. For example, to reduce complexity, the frames may be recombined at a reduced spatial resolution.

[0035] As shown in Table 4, for certain combinations of frame rate and shutter angle (e.g., 30 fps and 360 degrees, or 24 fps and 288 or 360 degrees), the decoder needs to combine three or more decoded frames, which increases the amount of buffer space required by the decoder. To reduce the burden of extra buffer space in the decoder, in some embodiments, certain combinations of frame rate and shutter angle may be limited (e.g., by setting appropriate coding profiles and levels) for the set of allowed decoding parameters.

[0036] Again, as an example, consider the case of playback at 24 fps, where the decoder may decide to display the same frame five times that would be displayed at an output frame rate of 120 fps. This is exactly the same as showing the frame once at an output frame rate of 24 fps. The advantage of maintaining a constant output frame rate is that the display can run at a constant clock speed, which makes all the hardware much simpler. If the display can dynamically change the clock speed, it makes more sense to display the frame only once (1 / 24th of a second) instead of repeating the same frame five times (each 1 / 120th of a second). The former approach may result in slightly higher image quality, slightly better optical efficiency, or slightly better power efficiency. Similar considerations are applicable to other frame rates.

[0037] Table 5 shows an example of frame rate conversion SEI messaging syntax according to one embodiment. [Table 5]

[0038] framerate_conversion_cancel_flag=1 indicates that the SEI message cancels the persistence of any previous frame rate conversion SEI messages in output order. framerate_conversion_cancel_flag=0 indicates that frame rate conversion information follows. base_frame_rate specifies the desired frame rate. The base_shutter_angle specifies the desired shutter angle. decode_phase_idx_present_flag=1 specifies that the decoding phase information is present, and decode_phase_idx_present_flag=0 specifies that the decoding phase information is not present. decode_phase_idx indicates an offset index in a series of sequential frames with index value 0..(n_frames_max-1), where n_frames_max=120 / base_frame_rate. The value of decode_phase_idx must be in the range 0..(n_frames_max-n_frames), where n_frames=base_shutter_angle / (3*base_frame_rate). If decode_phase_idx is not present, it is inferred to be 0. conversion_domain_idc=0 specifies that frame combining is performed in the linear domain. conversion_domain_idc=1 specifies that frame combining is performed in the non-linear domain. num_frame_buffers specifies the number of additional frame buffers (not counting DPBs). framerate_conversion_persistence_flag specifies the persistence of the frame rate conversion SEI message of the current layer. When framerate_conversion_persistence_flag=0, it specifies that the frame rate conversion SEI message applies only to the current decoded picture. Let picA be the current picture. When framerate_conversion_persistence_flag=1, it specifies that the frame rate conversion SEI message of the current layer persists in output order until one or more of the following conditions become true: - A new Coded Layer-Wise Video Sequence (CLVS) for the current layer begins. - The bitstream ends. - Picture picB in the current layer in the access unit containing the framer rate conversion SEI message applicable to the current layer is output (PicOrderCnt(picB) is greater than PicOrderCnt(picA)), where PicOrderCnt(picB) and PicOrderCnt(picA) are the PicOrderCntVal values ​​of picB and picA, respectively, immediately after the invocation of the decoding process for the picture order count of picB.

[0039] Third embodiment - Multiple shutter angle coded input The third embodiment is a coding scheme that allows for the extraction of sub-frame rates from the bitstream, thus supporting backward compatibility. In HEVC, this is achieved by temporal scalability. Temporal layer scalability is enabled by assigning different values ​​to the temporal_id syntax element of the decoded frames. This allows the bitstream to be simply extracted based on the temporal_id value. However, the HEVC-style approach to temporal scalability does not allow for rendering output frame rates with different shutter angles. For example, a base frame rate of 60 fps extracted from a 120 fps original will always have a shutter angle of 180 degrees.

[0040] In ATSC3.0, another method is described in which a 60fps frame with a 360 degree shutter angle is emulated as a weighted average of two 120fps frames. The emulated 60fps frames are assigned a temporal_id value of 0 and are interleaved with the original 120fps frames assigned a temporal_id value of 1. If 60fps is required, the decoder only needs to decode frames with temporal_id=0. If 120fps is required, the decoder can recover the corresponding original 120fps frames that were not explicitly transmitted by subtracting each temporal_id=1 frame (i.e., 120fps frame) from the scaled version of each corresponding temporal_id=0 frame (i.e., emulated 60fps frame), thereby reconstructing all the original 120fps frames.

[0041] In an embodiment of the present invention, a new algorithm is described that supports multiple target frame rates and target shutter angles in a backward compatible (BC) manner. It is proposed to pre-process the original 120fps content at a base frame rate at several shutter angles. Then, in the decoder, other frame rates at various other shutter angles can be easily derived. The ATSC3.0 approach can be considered as a special case of the proposed scheme, where frames with temporal_id=0 carry frames at 60fps@360 shutter angle, and frames with temporal_id=1 carry frames at 60fps@180 shutter angle.

[0042] As a first example, consider an input sequence at 120 fps and 360 shutter angle, which is used to encode a sequence with a base layer frame rate of 40 fps and shutter angles of 120, 240, and 360 degrees, as depicted in FIG. 4. In this scheme, the encoder calculates a new frame by combining up to three of the original input frames. For example, encoded frame 2 (En-2), which represents the input at 40 fps and 240 degrees, is generated by combining input frames 1 and 2, and encoded frame 3 (En-3), which represents the input at 40 fps and 360 degrees, is generated by combining frame En-2 with input frame 3. At the decoder, to reconstruct the input sequence, decoded frame 2 (Dec-2) is generated by subtracting frame En-1 from frame En-2, and decoded frame 3 (Dec-3) is generated by subtracting frame En-2 from frame En-3. The three decoded frames represent the output at a base frame rate of 120 fps and a shutter angle of 360 degrees. Additional frame rates and shutter angles can be extrapolated using the decoded frames as shown in Table 6. In Table 6, the function Cs(a,b) indicates the combination of input frames a to b, which can be performed by averaging, weighted averaging, filtering, etc. [Table 6-1] [Table 6-2]

[0043] The advantage of this approach is that all 40 fps versions can be decoded without further processing, as shown in Table 6. Another advantage is that other frame rates can be derived at various shutter angles. For example, consider a decoder decoding at 30 fps and a shutter angle of 360. From Table 4, the output corresponds to a sequence of frames generated by Ce(1,4)=Cs(1,4), Cs(5,8), Cs(9,12), etc., which also match the decoded sequence shown in Table 6, but in Table 6, Cs(5,8)=e6-e4+e8. In one embodiment, a look-up table (LUT) can be used to define how the decoded frames need to be combined to generate an output sequence at a specified output frame rate and emulated shutter angle.

[0044] In another example, as shown below, it is proposed to combine up to five frames in the encoder to simplify the extraction of a 24fps base layer at shutter angles of 72, 144, 216, 288, and 360 degrees. This is desirable for cinema content, which is best presented at 24fps on legacy televisions. [Table 7-1] [Table 7-2]

[0045] As shown in Table 7, if the decoded frame rate matches the baseline frame rate (24 fps), in each group of five frames (e.g., e1-e5), the decoder can simply select one frame with the desired shutter angle (e.g., e2 for a 144 degree shutter angle). To decode at a different frame rate and a particular shutter angle, the decoder needs to determine how to appropriately combine the decoded frames (e.g., by addition or subtraction). For example, to decode at 30 fps and a 180 degree shutter angle, the following steps can be followed: a) A decoder can consider a hypothetical encoder transmitting at 120fps and 360 degrees without any consideration of backward compatibility, and then from Table 1, the decoder needs to combine 2 out of 4 frames to generate an output sequence at the desired frame rate and shutter angle. For example, as shown in Table 4, the sequence may include Ce(1,2)=Avg(s1,s2), Ce(5,6)=Avg(s5,s6), etc., where Avg(s1,s2) may denote the average of frames s1 and s2. b) Given that by definition the coded frames can be expressed as e1 = s1, e2 = Avg(s1,s2), e3 = Avg(s1,s3), etc., it can be easily derived that the sequence of frames in step a) can also be expressed as -Ce(1,2)=Avg(s1,s2)=e2 -Ce(5,6)=Avg(s5,s6)=Avg(s1,s5)-Avg(s1,s4)+s6=e5-e4+e6 -etc. As before, the appropriate combinations for the decoded frames can be pre-computed and made available as a LUT.

[0046] The advantage of the proposed method is that it provides options for both content creators and users, i.e., directory / editorial choices as well as user choices. For example, pre-processing content in the encoder allows creating a base frame rate with various shutter angles. Each shutter angle can be assigned a temporal_id value in the range [0,(n_frames-1)], where n_frames has a value equal to 120 divided by the base frame rate. (For example, if the base frame rate is 24 fps, the temporal_id is in the range [0,4].) The choice can be made to optimize compression efficiency or for aesthetic reasons. In some use cases, e.g., for top-streaming, multiple bitstreams with different base layers can be encoded and stored and offered to the user to choose from.

[0047] In a second example of the disclosed method, multiple backward compatible frame rates may be supported. Ideally, one would like to be able to decode at 24 frames / sec to get a 24 fps base layer, at 30 frames / sec to get a 30 fps sequence, at 60 frames / sec to get a 60 fps sequence, and so on. If a target shutter angle is not specified, a default target shutter angle as close to 180 degrees as possible among the shutter angles allowed for the source and target frame rates is recommended. For example, with the values ​​shown in Table 7, the preferred target shutter angles for 120, 60, 40, 30, and 24 fps are 360 ​​degrees, 180 degrees, 120 degrees, 180 degrees, and 216 degrees.

[0048] From the above examples, it can be seen that the choice of how to encode content can affect the complexity of decoding a particular base layer frame rate. One embodiment of the present invention is to adaptively select an encoding scheme based on the desired base layer frame rate. For movie content, this may be, for example, 24 fps, while for sports, it may be 60 fps.

[0049] An exemplary syntax for the BC embodiment of the present invention is shown in Tables 8 and 9 below. In the SPS (Table 8), two syntax elements are added: SPS_hfr_BC_enabled_flag and SPS_base_framerate (if SPS_hfr_BC_enabled_flag is set to 1). sps_hfr_BC_enabled_flag=1 specifies that backwards compatible high frame rate is enabled for Coded Video Sequences (CVS). sps_hfr_BC_enabled_flag=0 specifies that backwards compatible high frame rate is not enabled for CVS. sps_base_framerate specifies the base frame rate of the current CVS. In the tile group header, if sps_hfr_BC_enabled_flag is set to 1, the syntax number_avg_frames is transmitted in the bitstream. number_avg_frames specifies the number of frames at the highest frame rate (e.g., 120 fps) that are combined to produce the current picture at the base frame rate. [Table 8] [Table 9]

[0050] Modification of the second embodiment (fixed frame rate) The HEVC (H.265) coding standard (Reference [1]) and the developing Versatile Video Coding Standard (commonly referred to as VVC, see Reference [2]) define a syntax element pic_struct that indicates whether a picture is to be represented as a frame or as one or more fields, and whether the decoded picture is to be repeated. A copy of Table D.2 "Interpretation of pic_struct" from HEVC is provided in the Appendix for reference.

[0051] As recognized by the inventors, it is important to note that the existing pic_struct syntax element can only support a certain subset of content frame rates when using a fixed frame rate coding container. For example, when using a 60 fps fixed frame rate container, the existing pic_struct syntax can support 30 fps by using frame doubling and 24 fps by using frame doubling and frame triplet, alternating frame by frame of each other, when fixed_pic_rate_within_cvs_flag=1. However, when using a 120 fps fixed frame rate container, the current pic_struct syntax cannot support frame rates of 24 fps or 30 fps. Two new methods have been proposed to mitigate this issue, one is an extension of the HEVC version and the other is not.

[0052] Method 1: Non-backwards compatible pic_struct VVC is still under development, so the syntax can be designed with maximum freedom. In one embodiment, it is proposed to remove the options for frame doubling and frame tripling in pic_struct and add a new syntax element num_frame_repetition_minus2, which indicates any frame repetition using a specific value in pic_struct and specifies the number of frames to repeat. An example of the proposed syntax is given in the table below, where Table 10 shows the changes to Table D.2.3 in HEVC and Table 11 shows the changes to Table D.2 shown in the Appendix. [Table 10] num_frame_repetition_minus2+2 indicates that if fixed_pic_rate_within_cvs_flag=1, frames should be presented on the display num_frame_repetition_minus2+2 times in succession, with a frame update interval equal to DpbOutputElementalInterval[n] as given by equation E-73. [Table 11]

[0053] Method 2: Extended version of HEVC version of pic_struct Since AVC and HEVC decoders have already been adopted, it may be desirable to simply extend the existing pic_struct syntax without removing the old options. In an embodiment, a new pic_struct=13, "frame repetition extension" value, and a new syntax element num_frame_repetition_minus4 are added. An example of the proposed syntax is given in Tables 12 and 13. For pic_struct values ​​0 through 12, the proposed syntax is the same as that in Table D.2 (as shown in the Appendix), so these values ​​have been omitted for simplicity. [Table 12] num_frame_repetition_minus4+4 indicates that if fixed_pic_rate_within_cvs_flag=1, then frames should be presented on the display num_frame_repetition_minus4+4 consecutive times, with a frame update interval equal to DpbOutputElementalInterval[n] as given by equation E-73. [Table 13]

[0054] In HEVC, the parameter Frame_field_info_present_flag exists in the video usability information (VUI), while the syntax elements pic_struct, source_scan_type, and duplicate_flag are in the pic_timing() SEI message. In an embodiment, it is proposed to move all related syntax elements together with frame_field_info_present_flag to the VUI. An example of the proposed syntax is shown in Table 14. [Table 14]

[0055] Alternative signaling of shutter angle information When dealing with variable frame rates, it is desirable to identify both the desired frame rate and the desired shutter angle. In conventional video coding standards, "Video Usability Information" (VUI) provides essential information for proper display of video content such as aspect ratio, color primaries, chroma subsampling, etc. Although the VUI may provide frame rate information when a fixed picrate is set to 1, there is no support for shutter angle information. The embodiments allow for the use of different shutter angles for different temporal layers, and the decoder can use the shutter angle information to improve the final appearance on the display.

[0056] For example, HEVC supports temporal sublayers that essentially use frame dropping techniques to go from a higher frame rate to a lower frame rate. The main problem with this is that with each frame drop, the effective shutter angle is reduced. For example, 60fps can be derived from a 120fps video by dropping every other frame. 30fps can be derived by dropping 3 out of 4 frames, and 24fps by dropping 4 out of 5 frames. Assuming a full 360 degree shutter for 120Hz, simple frame dropping results in shutter angles of 180, 90, and 72 degrees for 60fps, 30fps, and 24fps, respectively [3]. Experience has shown that shutter angles less than 180 degrees are generally unacceptable, especially for frame rates below 50Hz. By providing shutter angle information, for example, if it is desired for a display to generate a cinematic effect from a 120Hz video with reduced shutter angles for each temporal layer, smart techniques can be applied to improve the final appearance.

[0057] In another example, we may want to support different temporal layers (e.g., 60 fps sub-bitstreams in a 120 fps bitstream) with the same shutter angle. Then, the main problem is that when a 120 fps video is displayed at 120 Hz, even / odd frames have different effective shutter angles. If the display has the relevant information, smart techniques can be applied to improve the final appearance. An example of the proposed syntax is shown in Table 15, where the E.2.1 VUI parameters syntax table in HEVC (reference [1]) is modified to support the shutter angle information as described above. Note that in another embodiment, instead of expressing the shutter angle syntax in absolute degrees, it can also be expressed as a ratio of frame rate to shutter speed (see equation (1)). [Table 15] vui_shutter_angle_info_present_flag=1 specifies that shutter angle information is present in the vui_parameters() syntax structure. vui_shutter_angle_info_present_flag=0 specifies that shutter angle information is not present in the vui_parameters() syntax structure. fixed_shutter_angle_within_cvs_flag=1 specifies that the shutter angle information is the same for all temporal sublayers in the CVS. fixed_shutter_angle_within_cvs_flag=0 specifies that the shutter angle information is not the same for all temporal sublayers in the CVS. fixed_shutter_angle specifies the shutter angle in CVS in degrees. The value of fixed_shutter_angle ranges from 0 to 360. sub_layer_shutter_angle[i] specifies the shutter angle in degrees when HighestTid is equal to i. The value of sub_layer_shutter_angle[i] ranges from 0 to 360.

[0058] Gradual Frame Rate Update in Coded Video Sequences (CVS) Experiments show that for HDR content displayed on an HDR display, the frame rate must be increased based on the brightness of the content to perceive the same motion juddering as in standard dynamic range (SDR) playback at 100 nits. In most standards (AVC, HEVC, VVC, etc.), the video frame rate can be indicated in the VUI (contained in the SPS) using the vui_time_scale, vui_num_units_in_tick, elemental_duration_in_tc_minus1[temporal_id_max] syntax elements. For example, see Table 16 below (see Section E.2.1 of Reference [1]). [Table 16] As discussed in reference [1], the variable ClockTick is derived as follows and is called the clock tick: ClockTick=vui_num_units_in_tick÷vui_time_scale picture_duration=ClockTick*(elemental_duration_in_tc_minus1[i]+1) frame_rate=1 / pic_duration

[0059] However, the frame rate can only be changed at certain times, e.g., in HEVC, only at Intra Random Access Point (IRAP) frames or only at the start of a new CVS. In HDR playback, the frame rate or picture duration may need to be changed on a picture-by-picture basis, for example, in the case of fade-in or fade-out, since the brightness of the picture changes on a frame-by-frame basis. In order to allow the frame rate or picture duration to be refreshed at any time (even on a frame-by-frame basis), in one embodiment, a new SEI message for "Progressive Refresh Rate" is proposed, as shown in Table 17. [Table 17]

[0060] The new syntax num_units_in_tick has the same definition as vui_num_units_in_tick, and time_scale has the same definition as vui_time_scale. num_units_in_tick is the number of time units of a clock running at a frequency time_scale Hz that correspond to one increment (called a clock tick) of the clock tick counter. num_units_in_tick shall be greater than 0. The number of clock ticks in a second is equal to num_units_in_tick divided by time_scale. For example, if the picture rate of a video signal is 25 Hz, time_scale may be equal to 27 000 000, num_units_in_tick equal to 1 080 000 and therefore a clock tick equal to 0.04 seconds. time_scale is the number of time units that elapse in one second. For example, a time coordinate system that measures time using a 27 MHz clock has a time_scale of 27 000 000. The value of time_scale shall be greater than 0. The picture duration for a picture using the gradual_refresh_rate SEI message is defined as follows: picture_duration=num_units_in_tick÷time_scale.

[0061] Signaling shutter angle information by SEI message As mentioned above, Table 15 provides an example of VUI parameter syntax with shutter angle support. By way of example, and not limitation, Table 18 lists the same syntax elements, but now as part of the SEI message for shutter angle information. Note that the SEI messaging is used only as an example, and similar messaging may be constructed at other layers of high level syntax, such as sequence parameter set (SPS), picture parameter set (PPS), slice or tile group headers, etc. [Table 18]

[0062] The shutter angle is usually expressed as 0-360 degrees. For example, a shutter angle of 180 degrees indicates that the exposure duration is 1 / 2 the frame duration. The shutter angle can be expressed as shutter_angle=frame_rate*360*shutter_speed, where shutter_speed is the exposure duration and frame_rate is the inverse of the frame duration. The frame_rate for a given temporal sublayer Tid can be indicated by num_units_in_tick, time_scale, elemental_duration_in_tc_minus1[Tid]. For example, if fixed_pic_rate_within_cvs_flag[Tid]=1, then frame_rate= time_scale / (num_units_in_tick*(elemental_duration_in_tc_minus1[Tid]+1)).

[0063] In some embodiments, the shutter angle value (e.g., fixed_shutter_angle) may not be an integer, for example, 135.75 degrees. To allow for greater precision, in Table 21, u(9) (unsigned 9-bit) may be replaced with u(16) or some other suitable bit depth (e.g., 12-bit, 14-bit, or greater than 16-bit).

[0064] In some embodiments, it may be useful to express the shutter angle information in terms of "clock ticks." In VVC, the variable ClockTick is derived as follows:

number

number

number

number

[0065] Table 19 shows an example of the SEI messaging shown by equation (11). In this example, the shutter angle must be greater than 0 for a real-world camera. [Table 19] As mentioned above, the use of u(16) (unsigned 16-bit) for the shutter angle precision is shown as an example and is 360 / 2 16 = 0.0055 precision. The precision can be adjusted based on the actual application. For example, when using u(8), the precision is 360 / 2 8 =1.4063. NOTE: Shutter angles are expressed as greater than 0 degrees, but less than or equal to 360 degrees. For example, a shutter angle of 180 degrees indicates that the exposure duration is 1 / 2 the frame duration. fixed_shutter_angle_within_cvs_flag=1 specifies that the shutter angle value is the same for all temporal sub-layers in the CVS. fixed_shutter_angle_in_cvs_flag=0 specifies that the shutter angle value is not the same for all temporal sub-layers in the CVS. fixed_shutter_angle_numer_minus1+1 specifies the numerator used to derive the shutter angle value. The value of fixed_shutter_angle_numer_minus1 must be in the range of 0 to 65535. fixed_shutter_angle_demom_minus1+1 specifies the denominator used to derive the shutter angle value. The value of fixed_shutter_angle_demom_minus1 must be in the range 0 to 65535. The value of fixed_shutter_angle_numer_minus1 must be less than or equal to the value of fixed_shutter_angle_demom_minus1. The variable shutter angle, in degrees, is derived as follows: shutterAngle=360*(fixed_shutter_angle_numer_minus1+1)÷(fixed_shutter_angle_demom_minus1+1)) sub_layer_shutter_angle_numer_minus1[i]+1 specifies the numerator used to derive the shutter angle value when HighestTid is equal to i. The value of sub_layer_shutter_angle_numer_minus1[i] must be in the range from 0 to 65535, inclusive. sub_layer_shutter_angle_demom_minus1[i]+1 specifies the denominator used to derive the shutter angle value when HighestTid is equal to i. The value of sub_layer_shutter_angle_demom_minus1[i] must be in the range from 0 to 65535, inclusive. The value of sub_layer_shutter_angle_numer_minus1[i] must be less than or equal to the value of sub_layer_shutter_angle_denom_minus1[i]. The variable sublayer shutter angle [i], in degrees, is derived as follows: subLayerShutterAngle[i]=360* (sub_layer_shutter_angle_numer_minus1[i]+1)÷(sub_layer_shutter_angle_demom_minus1[i]+1)

[0066] In another embodiment, the frame duration (e.g., frame_duration) may be specified by some other means. For example, in DVB / ATSC, when fixed_pic_rate_within_cvs_flag[Tid]=1, frame_rate=time_scale / (num_units_in_tick*(elemental_duration_in_tc_minus1[Tid]+1)),frame_duration=1 / frame_rate.

[0067] Although the syntax in Table 19 and some of the subsequent tables assumes that the shutter angle is always greater than zero, a shutter angle=0 may be used to signal a creative intent that the content should be displayed without any motion blur. This is the case for moving graphics, animations, CGI textures, matte screens, etc. Thus, for example, signaling shutter angle=0 may be useful for mode decisions in transcoders (e.g., to select an edge-preserving transcoding mode) as well as in displays that receive shutter angle metadata over the CTA interface or 3GPP interface. For example, a shutter angle=0 may be used to instruct a display that should not perform motion processing such as noise reduction, frame interpolation, etc. In such an embodiment, the syntax elements fixed_shutter_angle_nume_minus1 and sub_layer_shutter_angle_numer_minus1[i] may be replaced by the syntax elements fixed_shutter_angle_numer and sub_layer_shutter_angle_nume[i], where fixed_shutter_angle_numer specifies the numerator used to derive the shutter angle value. The value of fixed_shutter_angle_numer must be in the range from 0 to 65535, inclusive. sub_layer_shutter_angle_numer[i] specifies the numerator used to derive the shutter angle value when HighestTid is equal to i. The value of sub_layer_shutter_angle_numer[i] must be in the range 0 to 65535, inclusive.

[0068] In another embodiment, fixed_shutter_angle_denom_minus1 and sub_layer_shutter_angle_denom_minus1[i] are also replaced by the syntax elements fixed_shutter_angle_denom and sub_layer_shutter_angle_denom[i].

[0069] In one embodiment, the num_units_In_tick and time_scale syntax defined in SPS can be reused by setting general_hrd_parameters_present_flag=1 in VVC as shown in Table 20. Under this scenario, the SEI message can be renamed as exposure duration SEI message. [Table 20] fixed_exposure_duration_within_cvs_flag=1 specifies that the effective exposure duration value is the same for all temporal sub-layers in the CVS. fixed_exposure_duration_numer_minus1+1 specifies the numerator used to derive the exposure duration value. The value of fixed_exposure_duration_numer_minus1 must be in the range 0 to 65535, inclusive. fixed_exposure_duration_demom_minus1+1 specifies the denominator used to derive the exposure duration value. fixed_exposure_duration_demom_minus1 must be in the range 0 to 65535, inclusive. The value of fixed_exposure_during_numer_minus1 must be less than or equal to the value of fixed_exposure_duration_demom_minus1. The variable fixedExposureDuration is derived as follows: fixedExposureDuration=(fixed_exposure_duration_numer_minus1+1)÷(fixed_exposure_duration_demom_minus1+1)*ClockTicks. sub_layer_exposure_duration_duration_numer_minus1[i]+1 specifies the numerator used to derive the exposure duration value when HighestTid is equal to i. The value of sub_layer_exposure_duration_numer_minus1[i] must be in the range from 0 to 65535, inclusive. sub_layer_exposure_duration_demom_minus1[i]+1 specifies the denominator used to derive the exposure duration value when HighestTid is equal to i. The value of sub_layer_exposure_duration_demom_minus1[i] must be in the range from 0 to 65535, inclusive. The value of sub_layer_exposure_duration_numer_minus1[i] must be less than or equal to the value of sub_layer_exposure_duration_demom_minus1[i]. The variable subLayerExposureDuration[i] for HigestTid equal to i is derived as follows: subLayerExposureDuration[i]=(sub_layer_exposure_duration_numer_minus1[i]+1)÷(sub_layer_exposure_duration_demom_minus1[i]+1)*ClockTick.

[0070] In another embodiment, clockTick can be explicitly defined by the syntax elements expo_num_units_in_tick and expo_time_scale as shown in Table 21. The advantage here is that, like the previous embodiment, there is no dependency on whether the general_hrd_parameters_present_flag is set to 1 in VVC.

number

[0071] As mentioned above, the syntax parameters sub_layer_exposure_duration_numer_minus1[i] and sub_layer_exposure_duration_denom_minus1[i] are also replaced by sub_layer_exposure_duration_numer[i] and sub_layer_exposure_duration_denom[i].

[0072] In another embodiment, the parameter ShutterInterval (i.e., exposure duration) can be defined by the syntax elements sii_num_units_in_shutter_interval and sii_time_scale, as shown in Table 22, where:

number

[0073] In an alternative embodiment, instead of using a numerator and denominator for signaling the sub-layer shutter interval, a single value is used. An example of such a syntax is shown in Table 23. [Table 23] Shutter interval information SEI message semantics The shutter interval information SEI message indicates the shutter interval for the associated video content before encoding and display, e.g., for camera captured content, the amount of time the image sensor is exposed to generate a picture. sii_num_units_in_shutter specifies the number of time units of a clock running at a frequency of sii_time_scale Hz that correspond to one increment of the shutter clock tick counter. The shutter interval, defined by the variable ShutterInterval, is equal to sii_num_units_in_shutter_interval divided by sii_time_scale, in seconds. For example, when ShutterInterval is equal to 0.04 seconds, sii_time_scale may be equal to 27 000 000 and sii_num_units_in_shutter_interval may be equal to 1 080 000. sii_time_scale specifies the number of time units that elapse in one second. For example, in a time coordinate system that measures time using a 27MHz clock, sii_time_scale is 27 000 000. If the value of sii_time_scale is greater than 0, the value of ShutterInterval is specified by: ShutterInterval=sii_num_units_in_shutter_interval÷sii_time_scale. Otherwise (the value of sii_time_scale is equal to 0), ShutterInterval should be interpreted as unknown or unspecified. Note 1: A value of ShutterInterval equal to 0 may indicate that the associated video content includes screen-captured content, computer-generated content, or other non-camera-captured content. NOTE 2: A value of ShutterInterval greater than the reciprocal of the coded picture rate, the coded picture interval, may indicate that the coded picture rate is greater than the picture rate at which the associated video content was generated, e.g., if the coded picture rate is 120 Hz and the picture rate of the associated video content before encoding and display is 60 Hz. The coded picture interval for a given temporal sublayer Tid may be indicated by ClockTick and elemental_duration_in_tc_minus1[Tid]. For example, when fixed_pic_rate_within_cvs_flag[Tid]=1, the picture interval for a given temporal sublayer Tid defined by the variable PictureInterval[Tid] is specified by: PictureInterval[Tid]=ClockTick*(elemental_duration_in_tc_minus1[Tid]+1) fixed_shutter_interval_into_cvs_flag=1 specifies that the value of ShutterInterval is the same for all temporal sublayers in the CVS. fixed_shutter_interval_within_cvs_flag=0 specifies that the value of ShutterInterval is not the same for all temporal sublayers in the CVS. sub_layer_num_units_in_shutter_interval[i] specifies the number of time units of a clock running at a frequency of sii_time_scale Hz that correspond to one increment of the shutter clock tick counter. The sublayer shutter interval defined by the variable subLayerShutterInterval[i] is equal to the quotient of sub_layer_num_units_in_shutter_interval[i] divided by sii_time_scale when HighestTid is equal to i, in seconds. When the value of fixed_shutter_interval_within_cvs_flag is equal to 0 and the value of sii_time_scale is greater than 0, the value of subLayerShutterInterval[i] is specified by: subLayerShutterInterval[i]=sub_layer_num_units_in_shutter_interval[i]÷sii_time_scale Otherwise (the value of sii_time_scale is equal to 0), subLayerShutterInterval[i] shall be interpreted as unknown or unspecified. If the value of fixed_shutter_interval_within_cvs_flag is not equal to 0, then subLayerShutterInterval[i] = ShutterInterval.

[0074] Table 24 provides an overview of the six approaches discussed in Tables 18-23 for providing SEI messaging related to shutter angle or exposure duration. [Table 24]

[0075] Variable Frame Rate Signaling In many applications, it is desired that the decoder support playback at variable frame rates, as described in U.S. Provisional Application No. 62 / 883,195, filed August 06, 2019. Frame rate adaptation is typically part of the operation in a hypothetical reference decoder (HRD), as described, for example, in Appendix C of reference [2]. In one embodiment, it is proposed to signal via SEI messaging or other means a syntax element that defines the Picture Presentation Time (PPT) as a function of the 90 kHz clock. This is a kind of repetition of the nominal Decoder Picture Buffer (DPB) output time specified in the HRD, but now using the 90 kHz ClockTicks precision specified in the MPEG-2 system. The advantages of this SEI message are that a) if the HRD is not enabled, the PPT SEI message can still be used to indicate the timing of each frame, and b) it can facilitate conversion of bitstream timing and system timing.

[0076] Table 25 illustrates an example of the proposed PPT timing message syntax, which matches the syntax of the presentation timestamp (PTS) variable used in MPEG-2 Transport (H.222) (Reference [4]). [Table 25] PPT (Picture Presentation Time) Presentation time is related to decoding time as follows: The PPT is a 33-bit number coded in three separate fields. It represents the time of presentation at the system target decoder of presentation unit k of elementary stream n, tp n The PPT value is specified in units of the system clock frequency divided by 300 (which produces 90 kHz). The Picture Presentation Time is derived from the PPT according to the following formula: PPT(k)=((system_clock_frequency x tp n (k)) / 300)%2 33 . Here, tp n (k) is presentation unit P n (k) Presentation time.

[0077] References Each of the references cited herein is incorporated by reference in its entirety. [1] High Efficiency Video Coding (HEVC), H.265, Series H, Video Encoding, ITU, (02 / 2018) [2] B. Bross, J. Chen, and S. Liu, "Diverse Video Coding (VVC) (Draft 5)," JVET Output Document, JVET-N1001, v5, (updated May 14, 2019) [3] C. Carbonara, J. DeFilippis, M. Korpi, "High Frame Rate Capture and Generation," SMPTE, 2015 Technical Conference and Exhibition, October 26-29, 2015. [4] Infrastructure for audiovisual services - Transmission multiplexing and synchronization, H.222.0, Series H, Generic coding of moving pictures and associated audio information: Systems, ITU, 08 / 2018

[0078] Computer system implementation example The embodiments of the present invention may be implemented in a computer system, a system composed of electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or perform instructions related to frame rate scalability as described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to frame rate scalability described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0079] Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc., can implement the frame rate scalability-related methods described above by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention may also be provided in the form of a program product. The program product may include any non-transitory, tangible medium that carries a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the methods of the present invention. A program product according to the present invention may be in any of a wide variety of non-transitory, tangible forms. The program product may include, for example, a physical medium such as a magnetic data storage medium including a floppy diskette, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signals on the program product may optionally be compressed or encrypted. Where a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, the reference to that component (including references to a "means") should be interpreted as including any component equivalent that performs the function of the described component (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structures that perform that function in exemplary embodiments of the invention.

[0080] Equivalents, Extensions, Substitutes and Others Thus, exemplary embodiments related to frame rate scalability are described. In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the only and exclusive indication of what the invention is and is intended by the applicant to be is the set of claims issued from this application in the particular form in which such claims are issued, including any subsequent amendments. Definitions expressly set forth herein for terms contained in the claims take precedence over the meaning of the terms used in the claims. Thus, no limitations, elements, properties, features, advantages or attributes not expressly set forth in the claims should limit the scope of the claims in any manner. Thus, the specification and drawings should be regarded in an illustrative and not restrictive sense.

[0081] appendix This appendix provides a copy of Table D.2 and associated pic_struct related information from the H.265 specification (Reference [1]). [Table D.2-1] [Table D.2-2] Semantics of the pic_struct syntax element pic_struct indicates whether a picture is to be presented as a frame or as one or more fields, and, for presentation of a frame when fixed_pic_rate_within_cvs_flag=1, may indicate the frame doubling or tripling repetition period for displays using a fixed frame update interval equal to DpbOutputElementalInterval[n] given by equation E-73. The interpretation of pic_struct is specified in Table D.2. Values ​​of pic_struct not listed in Table D.2 are reserved for future use by ITU-T|ISO / IEC and shall not be present in bitstreams conforming to this version of this specification. Decoders shall ignore reserved values ​​of pic_struct. If present, it is a requirement of bitstream conformance that the value of pic_struct be constrained such that exactly one of the following conditions is true: The value of -pic_struct will be equal to 0, 7, or 8 for all pictures in the CVS. The value of -pic_struct will be equal to 1, 2, 9, 10, 11, or 12 for all pictures in the CVS. The value of -pic_struct will be equal to 3, 4, 5, or 6 for all pictures in the CVS. When fixed_pic_rate_within_cvs_flag=1, frame doubling is indicated by pic_struct equal to 7, which indicates that the frame should be displayed twice consecutively on the display with a frame refresh interval equal to DpbOutputElementalInterval[n], as given by equation E-73, and frame tripling is indicated by pic_struct equal to 8, which indicates that the frame should be displayed three times consecutively on the display with a frame refresh interval equal to DpbOutputElementalInterval[n], as given by equation E-73. Note 3: Frame doubling can be used to facilitate the display, for example, of 25 Hz progressive scan video on a 50 Hz progressive scan display and 30 Hz progressive scan video on a 60 Hz progressive scan display. A combination of frame doubling and frame tripling, alternating on each other frame, can be used to facilitate the display of 24 Hz progressive scan video on a 60 Hz progressive scan display. The nominal vertical and horizontal sampling positions of the samples in the top and bottom fields for chroma formats 4:2:0, 4:2:2, and 4:4:4 are shown in Figures D.1, D.2, and D.3, respectively. The field association indicator (pic_struct equal to 9 through 12) provides a hint to associate fields of complementary parity as a frame. Field parity can be top or bottom, and if one field's parity is top and the other field's parity is bottom, then the parity of the two fields is considered complementary. When frame_field_info_present_flag=1, it is a requirement of bitstream conformance that the constraints specified in the third column of Table D.2 apply. NOTE 4: When frame_field_info_present_flag=0, in many cases a default value can be inferred or indicated by other means. In the absence of other indications of the picture's intended presentation type, decoders should infer the value of pic_struct as 0 when frame_field_info_present_flag=0.

Claims

1. An apparatus for encoding a video bitstream, comprising: an input for receiving a sequence of video pictures and associated shutter interval information; A processor, the processor comprising: generating an encoded video bitstream comprising an encoded picture section comprising an encoding of a sequence of video pictures and a signaling section comprising shutter interval parameters for said sequence of video pictures; The shutter interval parameter is a shutter interval time scale parameter indicating the number of time units that elapse in one second; a shutter interval clock tick parameter indicating the number of time units of a clock running at the frequency of said shutter interval time scale parameter; a shutter interval duration flag indicating whether exposure duration information for all temporal sub-layers within the coded picture section is fixed; if the shutter interval duration flag indicates that the exposure duration information is fixed, then the coded version of the sequence of video pictures for all temporal sub-layers in the coded picture section is decoded by calculating exposure duration values ​​based on the shutter interval time scale parameter and the shutter interval clock tick parameter; otherwise, the signaling section includes one or more arrays of sub-layer parameters, values ​​in the one or more arrays of sub-layer parameters combined with the shutter interval time scale parameter are used to calculate a corresponding sub-layer exposure duration value for each sub-layer to display a decoded version of a temporal sub-layer of the sequence of video pictures; the one or more arrays of sub-layer parameters include an array of sub-layer shutter interval clock tick values, each value indicating a number of time units of a clock running at a frequency of the shutter interval time scale parameter; the exposure duration value is calculated as the shutter interval clock tick parameter divided by the shutter interval time scale parameter; An apparatus comprising:

2. The apparatus of claim 1 , wherein the signaling section comprises a supplemental enhancement information (SEI) messaging section or a video user information (VUI) messaging section.

Images