Mixed NAL Unit Type Picture Constraints

Abstract

To disclose a video coding mechanism.SOLUTION: This mechanism includes receiving a bitstream comprising a flag and a plurality of sub-pictures which are associated with a picture. The plurality of sub-pictures are contained in a plurality of video coding layer (VCL) network abstraction layer (NAL) units. The mechanism determines: that one or more of VCL NAL units among the sub-pictures of the picture all have a first particular value of an NAL unit type; and that other VCL NAL units in the picture all have a different second particular value of the NAL unit type based on a value of the flag. One or more of the sub-pictures are decoded based on the first particular value of the NAL unit type and the second particular value of the NAL unit type. One or more of the sub-pictures are forwarded for display as part of a decoded video sequence.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0002] FIELD OF THE DISCLOSURE This disclosure relates generally to video coding, and more particularly to coding sub-pictures of pictures in video coding. [Background technology]

[0003] The amount of video data required to depict even a relatively short video can be significant, which can pose challenges when the data is to be streamed or otherwise communicated over communication networks with limited bandwidth capacity. Therefore, video data is typically compressed before being communicated over modern telecommunications networks. Because memory resources may be limited, video size can also be an issue when the video is stored on a storage device. Video compression devices often use software and / or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data needed to represent a digital video image. The compressed data is then received at the destination by a video decompression device, which decodes the video data. Due to limited network resources and an ever-increasing demand for higher video quality, improved compression and decompression techniques that improve compression ratios with little or no sacrifice in image quality are desirable. Summary of the Invention [Means for solving the problem]

[0004] In an embodiment, the present disclosure includes a method implemented in a decoder, the method comprising: receiving, by a receiver of the decoder, a bitstream comprising a flag and a plurality of sub-pictures associated with a picture, the sub-pictures being included in a plurality of video coding layer (VCL) network abstraction layer (NAL) units; determining, by a processor, based on the value of the flag, that one or more VCL NAL units of the sub-pictures of the picture all have a first particular value of NAL unit type and that other VCL NAL units of the picture all have a different second particular value of NAL unit type; and decoding, by the processor, one or more of the sub-pictures based on the first particular value of NAL unit type or the second particular value of NAL unit type.

[0005] A picture may be partitioned into multiple sub-pictures. Such sub-pictures may be coded into separate sub-bitstreams, which may then be aggregated into a bitstream for transmission to a decoder. For example, sub-pictures may be used for virtual reality (VR) applications. As a specific example, a user may only view a portion of a VR picture at any given time. Therefore, to improve coding efficiency, different sub-pictures may be transmitted at different resolutions so that more bandwidth can be allocated to sub-pictures that are more likely to be displayed and sub-pictures that are less likely to be displayed can be compressed. Furthermore, a video stream may be encoded by using intra random access point (IRAP) pictures. IRAP pictures are coded according to intra prediction and can be decoded without reference to other pictures. Non-IRAP pictures may be coded according to inter prediction and can be decoded by reference to other pictures. Non-IRAP pictures are much more condensed than IRAP pictures. However, because IRAP pictures contain enough data to be decoded without reference to other pictures, a video sequence must begin decoding at an IRAP picture. IRAP pictures can be used in subpictures and can enable dynamic resolution changes. Thus, to further increase coding efficiency, a video system may send more IRAP pictures for subpictures that are more likely to be seen (e.g., based on the user's current viewport) and fewer IRAP pictures for subpictures that are less likely to be seen. However, the subpictures are part of the same picture. Thus, this scheme may result in pictures that include both IRAP and non-IRAP subpictures. Some video systems are not capable of handling mixed pictures with both IRAP and non-IRAP regions. This disclosure includes a flag that indicates whether a picture is mixed, and therefore whether it includes both IRAP and non-IRAP components.Furthermore, this flag constrains a picture so that a mixed picture contains exactly two NAL unit types, including one IRAP type and one non-IRAP type. Based on this flag, a decoder can treat different subpictures differently during decoding in order to properly decode and display the picture / subpicture. This flag may be stored in the PPS and may be referred to as mixed_nalu_types_in_pic_flag. Thus, the disclosed mechanism enables the implementation of additional functionality. Furthermore, the disclosed mechanism enables dynamic resolution changes when utilizing subpicture bitstreams. Thus, the disclosed mechanism allows lower-resolution subpicture bitstreams to be transmitted when streaming VR video without significantly impairing the user experience. Thus, the disclosed mechanism increases coding efficiency, thereby reducing the use of network resources, memory resources, and / or processing resources in the encoder and decoder.

[0006] Optionally, in any of the preceding aspects, another implementation of the aspect provides that a first particular value of the NAL unit type indicates that the picture includes a single type of intra random access point (IRAP) subpicture, and that a second particular value of the NAL unit type indicates that the picture includes a single type of non-IRAP subpicture.

[0007] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the bitstream includes a picture parameter set (PPS) that includes the flag.

[0008] Optionally, in any of the preceding aspects, another implementation of the aspect specifies that the first particular value of the NAL unit type is equal to an instantaneous decoding refresh (IDR) with random access decodable leading pictures (IDR_W_RADL), an IDR without leading pictures (IDR_N_LP), or a clean random access (CRA) NAL unit type (CRA_NUT).

[0009] Optionally, in any of the preceding aspects, another implementation of the aspect specifies that the second particular value of the NAL unit type is equal to a trailing picture NAL unit type (TRAIL_NUT), a random access decodable leading picture NAL unit type (RADL_NUT), or a random access skip leading picture (RASL) NAL unit type (RASL_NUT).

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect specifies that this flag is mixed_nalu_types_in_pic_flag.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect specifies that mixed_nalu_types_in_pic_flag is equal to 1 when it specifies that each picture that references the PPS has more than one VCL NAL unit and that the VCL NAL units do not have the same value of NAL unit type (nal_unit_type), and that mixed_nalu_types_in_pic_flag is equal to 0 when each picture that references the PPS has one or more VCL NAL units and the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type.

[0012] In an embodiment, the present disclosure includes a method implemented in an encoder, the method comprising: determining, by a processor, that a picture includes multiple sub-pictures of different types; encoding, by the processor, the sub-pictures of the picture into multiple video coding layer (VCL) network abstraction layer (NAL) units in a bitstream; encoding, by the processor, a flag into the bitstream that is set to indicate that one or more VCL NAL units of the sub-pictures of the picture all have a first particular value of the NAL unit type and that other VCL NAL units in the picture all have a second particular value of the NAL unit type; and storing, by a memory coupled to the processor, the bitstream for communication to a decoder.

[0013] A picture may be partitioned into multiple sub-pictures. Such sub-pictures may be coded into separate sub-bitstreams, which may then be aggregated into a bitstream for transmission to a decoder. For example, sub-pictures may be used for virtual reality (VR) applications. As a specific example, a user may only view a portion of a VR picture at any given time. Therefore, to improve coding efficiency, different sub-pictures may be transmitted at different resolutions so that more bandwidth can be allocated to sub-pictures that are more likely to be displayed and sub-pictures that are less likely to be displayed can be compressed. Furthermore, a video stream may be encoded by using intra random access point (IRAP) pictures. IRAP pictures are coded according to intra prediction and can be decoded without reference to other pictures. Non-IRAP pictures may be coded according to inter prediction and can be decoded by reference to other pictures. Non-IRAP pictures are much more condensed than IRAP pictures. However, because IRAP pictures contain enough data to be decoded without reference to other pictures, a video sequence must begin decoding at an IRAP picture. IRAP pictures can be used in subpictures and can enable dynamic resolution changes. Thus, to further increase coding efficiency, a video system may send more IRAP pictures for subpictures that are more likely to be seen (e.g., based on the user's current viewport) and fewer IRAP pictures for subpictures that are less likely to be seen. However, the subpictures are part of the same picture. Thus, this scheme may result in pictures that include both IRAP and non-IRAP subpictures. Some video systems are not capable of handling mixed pictures with both IRAP and non-IRAP regions. This disclosure includes a flag that indicates whether a picture is mixed, and therefore whether it includes both IRAP and non-IRAP components.Furthermore, this flag constrains a picture so that a mixed picture contains exactly two NAL unit types, including one IRAP type and one non-IRAP type. Based on this flag, a decoder can treat different subpictures differently during decoding in order to properly decode and display the picture / subpicture. This flag may be stored in the PPS and may be referred to as mixed_nalu_types_in_pic_flag. Thus, the disclosed mechanism enables the implementation of additional functionality. Furthermore, the disclosed mechanism enables dynamic resolution changes when utilizing subpicture bitstreams. Thus, the disclosed mechanism allows lower-resolution subpicture bitstreams to be transmitted when streaming VR video without significantly impairing the user experience. Thus, the disclosed mechanism increases coding efficiency, thereby reducing the use of network resources, memory resources, and / or processing resources in the encoder and decoder.

[0014] Optionally, in any of the preceding aspects, another implementation of the aspect provides that a first particular value of the NAL unit type indicates that the picture includes a single type of IRAP subpicture, and that a second particular value of the NAL unit type indicates that the picture includes a single type of non-IRAP subpicture.

[0015] Optionally, in any of the preceding aspects, another implementation of the aspect further comprises encoding the PPS into the bitstream, and provides that the flag is encoded into the PPS.

[0016] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first particular value of the NAL unit type is equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT.

[0017] Optionally, in any of the preceding aspects, another implementation of the aspect specifies that the second particular value of the NAL unit type is equal to TRAIL_NUT, RADL_NU, or RASL_NU.

[0018] Optionally, in any of the preceding aspects, another implementation of the aspect specifies that this flag is mixed_nalu_types_in_pic_flag.

[0019] Optionally, in any of the preceding aspects, another implementation of the aspect specifies that mixed_nalu_types_in_pic_flag is equal to 1 when it specifies that each picture that references the PPS has more than one VCL NAL unit and that the VCL NAL units do not have the same value of nal_unit_type, and that mixed_nalu_types_in_pic_flag is equal to 0 when each picture that references the PPS has one or more VCL NAL units and the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type.

[0020] In an embodiment, the present disclosure includes a video coding device comprising a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are configured to perform the method of any of the preceding aspects.

[0021] In an embodiment, the present disclosure includes a non-transitory computer-readable medium comprising a computer program product for use by a video coding device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium that, when executed by a processor, cause the video coding device to perform the method of any of the preceding aspects.

[0022] In an embodiment, the present disclosure includes a decoder comprising: receiving means for receiving a bitstream comprising a flag and a plurality of sub-pictures associated with a picture, the plurality of sub-pictures being included in a plurality of VCL NAL units; determining means for determining, based on a value of the flag, that one or more VCL NAL units of the sub-pictures of the picture all have a first particular value of NAL unit type and that other VCL NAL units in the picture all have a different second particular value of NAL unit type; decoding means for decoding one or more of the sub-pictures based on the first particular value of NAL unit type or the second particular value of NAL unit type; and forwarding means for forwarding one or more of the sub-pictures for display as part of a decoded video sequence.

[0023] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the decoder is further configured to perform the method of any of the preceding aspects.

[0024] In an embodiment, the present disclosure includes an encoder comprising: determining means for determining that a picture includes multiple sub-pictures of different types; encoding means for encoding the sub-pictures of the picture into multiple VCL NAL units in a bitstream and encoding into the bitstream a flag that is set to indicate that one or more VCL NAL units of the sub-pictures of the picture all have a first particular value of the NAL unit type and that other VCL NAL units in the picture all have a second particular value of the NAL unit type, and storage means for storing the bitstream for communication to a decoder.

[0025] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the encoder is further configured to perform the method of any of the preceding aspects.

[0026] For clarity, any one of the above-described embodiments may be combined with any one or more of the other above-described embodiments to create new embodiments within the scope of the present disclosure.

[0027] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

[0028] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a flowchart of an exemplary method for coding a video signal. [Figure 2] 1 is a schematic diagram of an example coding and decoding (codec) system for video coding. [Figure 3] FIG. 1 is a schematic diagram illustrating an exemplary video encoder. [Figure 4] FIG. 1 is a schematic diagram illustrating an exemplary video decoder. [Figure 5] FIG. 2 is a schematic diagram illustrating an exemplary coded video sequence. [Figure 6] FIG. 1 is a schematic diagram illustrating multiple sub-picture video streams split from a virtual reality (VR) picture video stream. [Figure 7] FIG. 1 is a schematic diagram illustrating an exemplary bitstream containing pictures of mixed Network Abstraction Layer (NAL) unit types. [Figure 8] 1 is a schematic diagram of an exemplary video coding device. [Figure 9] 1 is a flowchart of an exemplary method for encoding a video sequence including pictures of mixed NAL unit types into a bitstream. [Figure 10]1 is a flowchart of an example method for decoding a video sequence including pictures of mixed NAL unit types from a bitstream. [Figure 11] 1 is a schematic diagram of an example system for coding a video sequence containing pictures of mixed NAL unit types into a bitstream. DETAILED DESCRIPTION OF THE INVENTION

[0030] While illustrative implementations of one or more embodiments are provided below, it should be understood at the outset that the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The present disclosure should not be limited in any way to the illustrative implementations, diagrams, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0031] The following acronyms are used herein: Coded Video Sequence (CVS), Decoded Picture Buffer (DPB), Instantaneous Decoding Refresh (IDR), Intra Random Access Point (IRAP), Least Significant Bit (LSB), Most Significant Bit (MSB), Network Abstraction Layer (NAL), Picture Order Count (POC), Raw Byte Sequence Payload (RBSP), Sequence Parameter Set (SPS), and Working Draft (WD).

[0032] Many video compression techniques can be utilized to reduce the size of video files while minimizing data loss. For example, video compression techniques may include performing spatial (e.g., intra-picture) prediction and / or temporal (e.g., inter-picture) prediction to reduce or remove data redundancy in a video sequence. For block-based video coding, video slices (e.g., video pictures or portions of video pictures) may be partitioned into video blocks, which may also be referred to as tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in intra-coded (I) slices of a picture are coded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in inter-coded unidirectionally predicted (P) or bidirectionally predicted (B) slices of a picture may be coded by utilizing spatial prediction with respect to reference samples in neighboring blocks of the same picture or temporal prediction with respect to reference samples in other reference pictures. Pictures may be referred to as frames and / or images, and reference pictures may be referred to as reference frames and / or reference images. Spatial or temporal prediction results in a predictive block representing an image block. Residual data represents pixel differences between the original image block and the predictive block. Thus, inter-coded blocks are coded according to a motion vector pointing to a block of reference samples forming the predictive block and residual data indicating the difference between the coded block and the predictive block. Intra-coded blocks are coded according to an intra-coding mode and the residual data. For further compression, the residual data can be transferred from the pixel domain to the transform domain. These result in residual transform coefficients that can be quantized. The quantized transform coefficients can first be arranged in a two-dimensional array. The quantized transform coefficients can be scanned to produce a one-dimensional vector of transform coefficients. Entropy coding can be applied to achieve further compression.Such video compression techniques are discussed in more detail below.

[0033] To ensure that the encoded video can be decoded correctly, the video is encoded and decoded according to a corresponding video coding standard. Video coding standards include International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, Advanced Video Coding (AVC), also known as ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC) and Multiview Video Coding plus Depth (MVC+D), and three-dimensional (3D) AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The ITU-T and ISO / IEC Joint Video Experts Team (JVET) has initiated development of a video coding standard called Versatile Video Coding (VVC). VVC is contained in a Working Draft (WD), which includes JVET-M1001-v6, which provides a description of the algorithm, a description of the encoder side of the VVC WD, and reference software.

[0034] A video coding system may encode video by utilizing IRAP pictures and non-IRAP pictures. An IRAP picture is a picture coded according to inter-prediction that serves as a random access point for a video sequence. In intra-prediction, blocks of a picture are coded with reference to other blocks within the same picture. This contrasts with non-IRAP pictures, which utilize inter-prediction. In inter-prediction, blocks of a current picture are coded with reference to other blocks in a reference picture different from the current picture. Because IRAP pictures are coded without reference to other pictures, they can be decoded without first decoding other pictures. Thus, a decoder can begin decoding a video sequence at any IRAP picture. In contrast, because non-IRAP pictures are coded with reference to other pictures, a decoder generally cannot begin decoding a video sequence at a non-IRAP picture. IRAP pictures also refresh the DPB. This is because an IRAP picture is the starting point of a CVS, and pictures in a CVS do not reference pictures in previous CVSs. Therefore, IRAP pictures can also stop inter-prediction-related coding errors because such errors cannot propagate through IRAP pictures. However, IRAP pictures are much larger than non-IRAP pictures in terms of data size. Therefore, video sequences generally contain fewer interspersed IRAP pictures along with many non-IRAP pictures to balance coding efficiency and functionality. For example, a 60-frame CVS may contain one IRAP picture and 59 non-IRAP pictures.

[0035] In some cases, video coding systems may be used to code virtual reality (VR) video, which may also be called 360-degree video. VR video may include a sphere of video content that is displayed as if the user were at the center of the sphere. Only a portion of the sphere, called a viewport, is displayed to the user. For example, a user may use a head-mounted display (HMD) that selects and displays a viewport of the sphere based on the user's head movement. This creates the impression of physically being present in the virtual space depicted by the video. To achieve this result, each picture in a video sequence includes a full sphere of video data at the corresponding instant in time. However, only a small portion of the picture (e.g., a single viewport) is displayed to the user. The remainder of the picture is discarded without being rendered. Typically, the entire picture is transmitted so that different viewports can be dynamically selected and displayed in response to the user's head movement. This approach can result in very large video file sizes.

[0036] To improve coding efficiency, some systems divide pictures into subpictures. A subpicture is a defined spatial region of a picture. Each subpicture contains a corresponding viewport of the picture. Video can be coded at two or more resolutions. Each resolution is coded into a different sub-bitstream. When a user streams VR video, the coding system can aggregate the sub-bitstreams into a bitstream for transmission based on the current viewport used by the user. Specifically, the current viewport is obtained from the high-resolution sub-bitstream, and the unviewed viewport is obtained from the low-resolution bitstream. In this way, the highest quality video is displayed to the user, and lower-quality video is discarded. When the user selects a new viewport, lower-resolution video is presented to the user. The decoder can request that the new viewport receive higher-resolution video. The encoder can then modify the aggregation process accordingly. When an IRAP picture is reached, the decoder can begin decoding the higher-resolution video sequence in the new viewport. This approach significantly improves video compression without adversely affecting the user's viewing experience.

[0037] One problem with the above approach is that the amount of time required to change resolution is based on the amount of time it takes to reach an IRAP picture. This is because, as explained above, a decoder cannot start decoding a different video sequence at a non-IRAP picture. One approach to reducing such latency is to include more IRAP pictures. However, this results in an increase in file size. To balance functionality and coding efficiency, different viewports / subpictures may include IRAP pictures at different frequencies. For example, viewports that are more likely to be viewed may have more IRAP pictures than other viewports. For example, in a basketball situation, viewports related to the basket and / or center court may include IRAP pictures more frequently than viewports showing the stands or ceiling, because such viewports are less likely to be viewed by a user.

[0038] This approach leads to other problems. Specifically, the subpictures containing the viewport are part of a single picture. When different subpictures have IRAP pictures at different frequencies, some parts of a picture contain both IRAP and non-IRAP subpictures. This is problematic because pictures are stored in the bitstream using NAL units. A NAL unit is a storage unit that contains a picture's parameter set or slices and the corresponding slice headers. An access unit is a unit that contains an entire picture. Thus, an access unit contains all of the NAL units for a picture. A NAL unit also contains a type that indicates the type of picture that contains the slice. Some video systems require that all NAL units for a single picture (e.g., contained in the same access unit) have the same type. Therefore, the NAL unit storage mechanism may not work correctly when a picture contains both IRAP and non-IRAP subpictures.

[0039] Disclosed herein is a mechanism for adjusting the NAL storage scheme to support pictures that contain both IRAP and non-IRAP subpictures. This in turn enables VR videos that contain different IRAP subpicture frequencies for different viewports. In a first example, a flag is disclosed herein that indicates whether a picture is mixed. For example, this flag may indicate that the picture contains both IRAP and non-IRAP subpictures. Based on this flag, a decoder can treat different types of subpictures differently when decoding in order to properly decode and display the picture / subpicture. This flag may be stored in a picture parameter set (PPS) and may be referred to as mixed_nalu_types_in_pic_flag.

[0040] In a second example, a flag indicating whether a picture is mixed is disclosed herein. For example, the flag may indicate that the picture includes both IRAP and non-IRAP subpictures. Furthermore, the flag constrains the picture so that the mixed picture includes exactly two NAL unit types, including one IRAP type and one non-IRAP type. For example, the picture may include IRAP NAL units that include only one of instantaneous decoding refresh (IDR) with random-access decodable leading pictures (IDR_W_RADL), IDR without leading pictures (IDR_N_LP), or clean random access (CRA) NAL unit types (CRA_NUT). Furthermore, the picture may include non-IRAP NAL units that include only one of trailing picture NAL unit types (TRAIL_NUT), random-access decodable leading picture NAL unit types (RADL_NUT), or random-access skip leading picture (RASL) NAL unit types (RASL_NUT). Based on this flag, the decoder can treat different sub-pictures differently when decoding in order to decode and display the picture / sub-picture appropriately. This flag may be stored in the PPS and may be called mixed_nalu_types_in_pic_flag.

[0041] 1 is a flowchart of an exemplary operational method 100 of coding a video signal. Specifically, a video signal is encoded in an encoder. The encoding process compresses the video signal by utilizing various mechanisms to reduce the video file size. The smaller file size allows the compressed video file to be transmitted to a user while reducing the associated bandwidth overhead. A decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally mirrors the encoding process to enable the decoder to reliably reconstruct the video signal.

[0042] In step 101, a video signal is input to an encoder. For example, the video signal may be an uncompressed video file stored in memory. As another example, the video file may be captured by a video capture device, such as a video camera, and encoded to support live streaming of the video. The video file may include both an audio component and a video component. The video component includes a series of image frames that, when viewed in sequence, create the visual effect of movement. A frame includes pixels represented in terms of light, referred to herein as luma components (or luma samples), and pixels represented in terms of color, referred to herein as chroma components (or color samples). In some examples, the frame may also include depth values to support three-dimensional viewing.

[0043] In step 103, the video is partitioned into blocks. Partitioning involves subdividing the pixels of each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame may first be divided into coding tree units (CTUs), which are blocks of a predetermined size (e.g., 64 pixels by 64 pixels). CTUs contain both luma and chroma samples. A coding tree may be utilized to divide the CTUs into blocks and then recursively subdivide the blocks until a configuration that supports further encoding is achieved. For example, the luma component of a frame may be subdivided until each block contains relatively uniform illumination values. Furthermore, the chroma component of a frame may be subdivided until each block contains relatively uniform color values. Thus, the partitioning scheme varies depending on the content of the video frame.

[0044] In step 105, various compression mechanisms are utilized to compress the image blocks partitioned in step 103. For example, inter-prediction and / or intra-prediction may be utilized. Inter-prediction is designed to take advantage of the fact that objects in a common scene tend to appear in consecutive frames. Thus, a block depicting an object in a reference frame need not be repeatedly described in adjacent frames. Specifically, an object such as a table may remain in a constant position across multiple frames. Thus, the table may be described once, and adjacent frames may reference the reference frame. A pattern matching mechanism may be utilized to match objects across multiple frames. Furthermore, a moving object may be depicted across multiple frames, for example, due to object motion or camera motion. As a specific example, a video may show a car moving around on the screen across multiple frames. A motion vector may be utilized to describe such motion. A motion vector is a two-dimensional vector that provides an offset from the coordinates of the object in a frame to the coordinates of the object in a reference frame. Thus, inter-prediction may encode an image block in a current frame as a set of motion vectors indicating an offset from a corresponding block in a reference frame.

[0045] Intra prediction encodes blocks within a common frame. Intra prediction takes advantage of the fact that luma and chroma components tend to be clustered in a frame. For example, a green spot in a tree tends to be positioned next to a similar green spot. Intra prediction utilizes multiple directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. Directional mode indicates that the current block is similar / the same as samples of neighboring blocks in the corresponding direction. Planar mode indicates that a series of blocks along a row / column (e.g., a plane) can be interpolated based on neighboring blocks at the end of the row. Planar mode essentially indicates a smooth transition of light / color across a row / column by utilizing a relatively constant gradient by varying values. DC mode is utilized for boundary smoothing and indicates that the block is similar / the same as the average value associated with samples of all neighboring blocks associated with the angular direction of the directional prediction mode. Thus, intra-predicted blocks can represent image blocks as various related prediction mode values instead of actual values. Furthermore, inter-predicted blocks can represent image blocks as motion vector values instead of actual values. In either case, the prediction block may in some cases not exactly represent the image block. Any differences are stored in a residual block. To further compress the file, a transform may be applied to the residual block.

[0046] In step 107, various filtering techniques may be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction discussed above may result in the creation of blocky images in a decoder. Furthermore, the block-based prediction scheme may encode a block and then reconstruct the encoded block for later use as a reference block. The in-loop filtering scheme iteratively applies a noise suppression filter, a deblocking filter, an adaptive loop filter, and a sample adaptive offset (SAO) filter to a block / frame. These filters mitigate such blocking artifacts so that the encoded file can be accurately reconstructed. Furthermore, because these filters mitigate artifacts in the reconstructed reference block, the artifacts are less likely to produce additional artifacts in subsequent blocks that are coded based on the reconstructed reference block.

[0047] Once the video signal has been segmented, compressed, and filtered, the resulting data is encoded in a bitstream in step 109. The bitstream includes the data discussed above as well as any signaling data desired to support proper video signal reconstruction at the decoder. For example, such data may include segmentation data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream may be stored in memory for transmission to the decoder on demand. The bitstream may also be broadcast and / or multicast to multiple decoders. Creation of the bitstream is an iterative process. Thus, steps 101, 103, 105, 107, and 109 may occur sequentially and / or simultaneously across multiple frames and blocks. The order shown in FIG. 1 is presented for clarity and ease of discussion and is not intended to limit the video coding process to any particular order.

[0048] In step 111, a decoder receives the bitstream and begins the decoding process. Specifically, the decoder converts the bitstream into corresponding syntax and video data using an entropy decoding scheme. In step 111, the decoder uses syntax data from the bitstream to determine a partition for the frame. This partition must match the result of the block partitioning in step 103. Entropy encoding / decoding as used in step 111 is now described. The encoder makes many choices during the compression process, such as selecting a block partitioning scheme from several possible options based on the spatial positioning of values in the input image. Signaling the exact selection may utilize multiple bins. Herein, a bin is a binary value (e.g., a bit value that can change depending on the situation) treated as a variable. Entropy coding allows the encoder to discard any options that are clearly not feasible for a particular case, leaving a set of acceptable options. Each acceptable option is then assigned a codeword. The length of the codeword is based on the number of allowable choices (e.g., one bin for two choices, two bins for three to four choices, etc.). The encoder then encodes the codeword for the selected choice. This scheme reduces the size of the codeword because the codeword is as large as desired to uniquely indicate a choice from a small subset of allowable choices, rather than uniquely indicating a choice from a large possible set of all possible choices. The decoder then decodes the choice by determining the set of allowable choices in a similar manner as the encoder. By determining the set of allowable choices, the decoder can read the codeword and determine the choice made by the encoder.

[0049] In step 113, the decoder performs block decoding. Specifically, the decoder generates a residual block using an inverse transform. The decoder then uses the residual block and a corresponding predictive block to reconstruct an image block according to the partition. The predictive block may include both intra-predicted blocks and inter-predicted blocks, such as those generated by the encoder in step 105. The reconstructed image block is then positioned into a frame of the reconstructed video signal according to the partition data determined in step 111. The syntax for step 113 may also be signaled in the bitstream via entropy coding, as discussed above.

[0050] In step 115, filtering is performed on the frames of the reconstructed video signal at the encoder in a manner similar to step 107. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and an SAO filter may be applied to the frames to remove blocking artifacts. Once the frames have been filtered, the video signal may be output to a display in step 117 for viewing by an end user.

[0051] 2 is a schematic diagram of an exemplary coding and decoding (codec) system 200 for video coding. Specifically, codec system 200 provides functionality to support the implementation of operational method 100. Codec system 200 is generalized to depict components utilized in both encoders and decoders. Codec system 200 receives and segments a video signal as discussed with respect to steps 101 and 103 in operational method 100, which results in a segmented video signal 201. When operating as an encoder, as discussed with respect to steps 105, 107, and 109 of method 100, codec system 200 then compresses the segmented video signal 201 into a coded bitstream. When operating as a decoder, codec system 200 generates an output video signal from the bitstream as discussed with respect to steps 111, 113, 115, and 117 of operational method 100. Codec system 200 includes a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, an intra-picture prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header formatting and context-adaptive binary arithmetic coding (CABAC) component 231. Such components are coupled as shown. In FIG. 2, black lines indicate the movement of data to be encoded / decoded, and dashed lines indicate the movement of control data that controls the operation of other components. The components of codec system 200 may all reside in an encoder. A decoder may include a subset of the components of codec system 200.For example, the decoder may include an intra-picture prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded picture buffer component 223. These components are now described.

[0052] The partitioned video signal 201 is a captured video sequence that has been partitioned into blocks of pixels by a coding tree. The coding tree utilizes various partitioning modes to subdivide the blocks of pixels into smaller blocks of pixels. These blocks may then be further subdivided into smaller blocks. The blocks may be referred to as nodes on the coding tree. Larger parent nodes are divided into smaller child nodes. The number of times a node is subdivided is referred to as the depth of the node / coding tree. In some cases, the partitioned blocks may be included in a coding unit (CU). For example, a CU may be a subpart of a CTU that includes a luma block, a red differential chroma (Cr) block, and a blue differential chroma (Cb) block, along with corresponding syntax instructions for the CU. Partitioning modes may include a binary tree (BT), a ternary tree (TT), and a quad tree (QT), each of which is utilized to partition a node into two, three, or four child nodes whose shape changes depending on the partitioning mode utilized. The segmented video signal 201 is forwarded to a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, a filter control analysis component 227, and a motion estimation component 221 for compression.

[0053] The generic coder control component 211 is configured to make decisions regarding the coding of images of a video sequence into a bitstream according to application constraints. For example, the generic coder control component 211 manages the optimization of bitrate / bitstream size versus reconstruction quality. Such decisions may be made based on storage space / bandwidth availability and image resolution requirements. The generic coder control component 211 also manages buffer utilization, taking transmission speed into account, to mitigate buffer underrun and overrun issues. To manage these issues, the generic coder control component 211 manages segmentation, prediction, and filtering by other components. For example, the generic coder control component 211 may dynamically increase compression complexity to improve resolution and bandwidth utilization, or decrease compression complexity to reduce resolution and bandwidth utilization. Thus, the generic coder control component 211 controls other components of the codec system 200 to balance video signal reconstruction quality and bitrate issues. The generic coder control component 211 generates control data, which controls the operation of other components. Control data is also forwarded to the Header Formatting and CABAC component 231 and encoded in the bitstream to signal parameters for decoding at the decoder.

[0054] The partitioned video signal 201 is also sent to a motion estimation component 221 and a motion compensation component 219 for inter-prediction. A frame or slice of the partitioned video signal 201 may be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-predictive coding of the received video blocks relative to one or more blocks in one or more reference frames to perform temporal prediction. The codec system 200 may perform multiple coding passes to, for example, select an appropriate coding mode for each block of video data.

[0055] The motion estimation component 221 and the motion compensation component 219 may be highly integrated but are shown separately for conceptual purposes. Motion estimation, performed by the motion estimation component 221, is the process of generating motion vectors that estimate the motion of video blocks. A motion vector may indicate, for example, the displacement of a coded object relative to a predictive block. A predictive block is a block that is found to closely match a block to be coded in terms of pixel differences. A predictive block may also be referred to as a reference block. Such pixel differences may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference measures. HEVC utilizes several coded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, CTUs may be divided into CTBs, which may then be divided into CBs for inclusion in CUs. CUs may be coded as prediction units (PUs) containing prediction data and / or transform units (TUs) containing transformed residual data for the CUs. The motion estimation component 221 generates motion vectors, PUs, and TUs by using rate-distortion analysis as part of a rate-distortion optimization process. For example, motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame and may select the reference block, motion vector, etc. with the best rate-distortion performance, which balances the quality of the video reconstruction (e.g., the amount of data lost due to compression) and coding efficiency (e.g., the size of the final encoding).

[0056] In some examples, the codec system 200 may calculate values for sub-integer pixel positions of reference pictures stored in the decoded picture buffer component 223. For example, the video codec system 200 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference pictures. Accordingly, the motion estimation component 221 may perform motion searches for integer and fractional pixel positions to output fractional-pixel precision motion vectors. The motion estimation component 221 calculates motion vectors for PUs of video blocks in inter-coded slices by comparing the positions of the PUs with the positions of predictive blocks of the reference pictures. The motion estimation component 221 outputs the calculated motion vectors to the header formatting and CABAC component 231 as motion data for encoding and outputs motion to the motion compensation component 219.

[0057] The motion compensation performed by the motion compensation component 219 may involve fetching or generating a predictive block based on a motion vector determined by the motion estimation component 221. Again, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated in some examples. Upon receiving the motion vector for the PU of the current video block, the motion compensation component 219 may locate the predictive block to which the motion vector points. A residual video block is then formed by subtracting pixel values of the predictive block from pixel values of the current video block being coded to form pixel difference values. Generally, the motion estimation component 221 performs motion estimation on the luma component, and the motion compensation component 219 uses the motion vector calculated based on the luma component for both the chroma and luma components. The predictive block and the residual block are forwarded to the transform scaling and quantization component 213.

[0058] The partitioned video signal 201 is also sent to an intra-picture estimation component 215 and an intra-picture prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-picture estimation component 215 and the intra-picture prediction component 217 may be highly integrated but are shown separately for conceptual purposes. The intra-picture estimation component 215 and the intra-picture prediction component 217 intra-predict the current block relative to blocks in the current frame as an alternative to the inter-prediction performed by the motion estimation component 221 and the motion compensation component 219 between frames, as described above. Specifically, the intra-picture estimation component 215 determines the intra-prediction mode to use to encode the current block. In some examples, the intra-picture estimation component 215 selects an appropriate intra-prediction mode for encoding the current block from multiple tested intra-prediction modes. The selected intra-prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0059] For example, the intra picture estimation component 215 calculates a rate-distortion value using a rate-distortion analysis for various tested intra prediction modes and selects an intra prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between an encoded block and the original uncoded block that was coded to produce the encoded block, as well as the bit rate (e.g., number of bits) used to produce the encoded block. The intra picture estimation component 215 calculates a ratio from the distortion and rate for the various coded blocks to determine which intra prediction mode exhibits the best rate-distortion value for the block. In addition, the intra picture estimation component 215 may be configured to code depth blocks of a depth map using a depth modeling mode (DMM) based on rate-distortion optimization (RDO).

[0060] The intra-picture prediction component 217, when implemented on an encoder, may generate a residual block from the prediction block based on a selected intra-prediction mode determined by the intra-picture estimation component 215, or, when implemented on a decoder, may read the residual block from the bitstream. The residual block includes value differences between the prediction block and the original block, represented as a matrix. The residual block is then forwarded to the transform scaling and quantization component 213. The intra-picture estimation component 215 and the intra-picture prediction component 217 may operate on both the luma and chroma components.

[0061] The transform scaling and quantization component 213 is configured to further compress the residual block. The transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform, to the residual block, resulting in a video block comprising residual transform coefficient values. A wavelet transform, an integer transform, a subband transform, or other types of transforms may also be used. The transform may convert the residual information from the pixel value domain to a transform domain, such as the frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling involves applying a scale factor to the residual information so that different frequency information is quantized with different granularity, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also configured to quantize the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the transform scaling and quantization component 213 may then perform a scan of a matrix containing the quantized transform coefficients, which are forwarded to the header formatting and CABAC component 231 to be encoded in the bitstream.

[0062] The scaling and inverse transform component 229 applies the inverse operations of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, transform, and / or quantization to reconstruct residual blocks in the pixel domain for later use as reference blocks, which may become predictive blocks for another current block, for example. The motion estimation component 221 and / or motion compensation component 219 may calculate reference blocks by adding the residual blocks back to the corresponding predictive blocks for use in motion estimation of later blocks / frames. A filter is applied to the reconstructed reference blocks to mitigate artifacts created during scaling, quantization, and transform. Such artifacts may otherwise cause inaccurate predictions (and create further artifacts) when subsequent blocks are predicted.

[0063] The filter control analysis component 227 and the in-loop filter component 225 apply filters to residual blocks and / or reconstructed image blocks. For example, a transformed residual block from the scaling and inverse transform component 229 may be combined with a corresponding prediction block from the intra-picture prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. The filter may then be applied to the reconstructed image block. In some examples, the filter may instead be applied to the residual block. Like the other components in FIG. 2, the filter control analysis component 227 and the in-loop filter component 225 may be highly integrated and implemented together, but are illustrated separately for conceptual purposes. The filters applied to reconstructed reference blocks are applied to specific spatial regions and include multiple parameters for adjusting how such filters are applied. The filter control analysis component 227 analyzes the reconstructed reference blocks to determine where such filters should be applied and sets the corresponding parameters. Such data is forwarded to the header formatting and CABAC component 231 as filter control data for encoding. The in-loop filter component 225 applies such filters based on the filter control data. The filters may include a deblocking filter, a noise suppression filter, a SAO filter, and an adaptive loop filter. Such filters may be applied in the spatial / pixel domain (e.g., on reconstructed pixel blocks) or in the frequency domain, depending on the example.

[0064] When operating as an encoder, the filtered reconstructed image blocks, residual blocks, and / or predictive blocks are stored in the decoded picture buffer component 223 for later use in motion estimation as discussed above. When operating as a decoder, the decoded picture buffer component 223 stores and forwards the reconstructed and filtered blocks to a display as part of the output video signal. The decoded picture buffer component 223 may be any memory device capable of storing predictive blocks, residual blocks, and / or reconstructed image blocks.

[0065] The header formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes such data into a coded bitstream for transmission to a decoder. Specifically, the header formatting and CABAC component 231 generates various headers to encode control data, such as general control data and filter control data. Additionally, prediction data, including intra-prediction and motion data, and residual data in the form of quantized transform coefficient data, are all encoded in the bitstream. The final bitstream contains all information desired by a decoder to reconstruct the original partitioned video signal 201. Such information may also include an intra-prediction mode index table (also called a codeword mapping table), definitions of coding contexts for various blocks, indications of the most probable intra-prediction mode, indications of partition information, and so on. Such data may be encoded by utilizing entropy coding. For example, the information may be encoded by utilizing context-adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioned entropy (PIPE) coding, or another entropy coding technique. Following entropy coding, the coded bitstream may be transmitted to another device (eg, a video decoder) or may be archived for later transmission or retrieval.

[0066] 3 is a block diagram illustrating an example video encoder 300. Video encoder 300 may be utilized to implement the encoding functionality of codec system 200 and / or to implement steps 101, 103, 105, 107, and / or 109 of method of operation 100. Encoder 300 segments an input video signal, resulting in a segmented video signal 301 that is substantially similar to segmented video signal 201. Segmented video signal 301 is then compressed and encoded into a bitstream by components of encoder 300.

[0067] Specifically, the partitioned video signal 301 is forwarded to an intra-picture prediction component 317 for intra prediction. The intra-picture prediction component 317 may be substantially similar to the intra-picture estimation component 215 and the intra-picture prediction component 217. The partitioned video signal 301 is also forwarded to a motion compensation component 321 for inter prediction based on a reference block in a decoded picture buffer component 323. The motion compensation component 321 may be substantially similar to the motion estimation component 221 and the motion compensation component 219. The prediction block and residual block from the intra-picture prediction component 317 and the motion compensation component 321 are forwarded to a transform and quantization component 313 for transforming and quantizing the residual block. The transform and quantization component 313 may be substantially similar to the transform scaling and quantization component 213. The transformed and quantized residual block and the corresponding prediction block (along with associated control data) are forwarded to an entropy coding component 331 for coding into a bitstream. The entropy coding component 331 may be substantially similar to the header formatting and CABAC component 231 .

[0068] The transformed and quantized residual block and / or the corresponding prediction block are also forwarded from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into a reference block used by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially similar to the scaling and inverse transform component 229. An in-loop filter in the in-loop filter component 325 is also applied to the residual block and / or the reconstructed reference block, depending on the example. The in-loop filter component 325 may be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may include multiple filters such as those discussed with respect to the in-loop filter component 225. The filtered block is then stored in the decoded picture buffer component 323 for use as a reference block by the motion compensation component 321. The decoded picture buffer component 323 may be substantially similar to the decoded picture buffer component 223.

[0069] 4 is a block diagram illustrating an exemplary video decoder 400. Video decoder 400 may be utilized to implement the decoding functionality of codec system 200 and / or to perform steps 111, 113, 115, and / or 117 of method of operation 100. Decoder 400 receives a bitstream, for example, from encoder 300, and generates a reconstructed output video signal based on the bitstream for display to an end user.

[0070] The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 may utilize header information to provide context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding the video signal, such as general control data, filter control data, partition information, motion information, prediction data, and quantized transform coefficients from residual blocks. The quantized transform coefficients are forwarded to the inverse transform and quantization component 429 for reconstruction into residual blocks. The inverse transform and quantization component 429 may be similar to the inverse transform and quantization component 329.

[0071] The reconstructed residual block and / or predictive block are forwarded to the intra-picture prediction component 417 for reconstruction into an image block based on an intra-prediction operation. The intra-picture prediction component 417 may be similar to the intra-picture estimation component 215 and the intra-picture prediction component 217. Specifically, the intra-picture prediction component 417 utilizes a prediction mode to locate a reference block within a frame and applies the residual block to the result to reconstruct an intra-predicted image block. The reconstructed intra-predicted image block and / or residual block and corresponding inter-prediction data are forwarded to the decoded picture buffer component 423 via an in-loop filter component 425, which may be substantially similar to the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block, and / or predictive block, and such information is stored in the decoded picture buffer component 423. The reconstructed image blocks from the decoded picture buffer component 423 are forwarded to the motion compensation component 421 for inter prediction. The motion compensation component 421 may be substantially similar to the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 utilizes motion vectors from reference blocks to generate prediction blocks and applies a residual block to the result to reconstruct an image block. The resulting reconstructed blocks may also be forwarded to the decoded picture buffer component 423 via an in-loop filter component 425. The decoded picture buffer component 423 continues to store additional reconstructed image blocks, which may be reconstructed into frames via partition information. Such frames may also be arranged in a sequence. The sequence is output to a display as a reconstructed output video signal.

[0072] 5 is a schematic diagram illustrating an exemplary CVS 500. For example, the CVS 500 may be encoded by an encoder, such as the codec system 200 and / or the encoder 300, in accordance with the method 100. Furthermore, the CVS 500 may be decoded by a decoder, such as the codec system 200 and / or the decoder 400. The CVS 500 includes pictures coded in a decoding order 508. The decoding order 508 is the order in which the pictures are positioned in a bitstream. The pictures of the CVS 500 are then output in a presentation order 510. The presentation order 510 is the order in which the pictures should be displayed by a decoder to properly display the resulting video. For example, the pictures of the CVS 500 may generally be positioned in the presentation order 510. However, to increase coding efficiency, for example, by placing similar pictures closer together to support inter-prediction, some pictures may be moved to different positions. Moving such pictures in this manner results in the decoding order 508. In the illustrated example, the pictures are indexed in the decoding order 508 from 0 to 4. In presentation order 510, the pictures at index 2 and index 3 have been moved before the picture at index 0.

[0073] The CVS 500 includes an IRAP picture 502. The IRAP picture 502 is a picture coded according to intra prediction that serves as a random access point for the CVS 500. Specifically, blocks of the IRAP picture 502 are coded with reference to other blocks in the IRAP picture 502. Because the IRAP picture 502 is coded without reference to other pictures, the IRAP picture 502 can be decoded without first decoding any other pictures. Thus, a decoder can start decoding the CVS 500 at the IRAP picture 502. Furthermore, the IRAP picture 502 may cause the DPB to be refreshed. For example, pictures presented after the IRAP picture 502 may not depend on pictures (e.g., picture index 0) before the IRAP picture 502 for inter prediction. Thus, the picture buffer may be refreshed when the IRAP picture 502 is decoded. This has the effect of stopping any inter-prediction related coding errors because such errors cannot propagate through the IRAP picture 502. The IRAP picture 502 may include various types of pictures. For example, the IRAP picture may be coded as an IDR or a CRA. An IDR is an intra-coded picture that starts a new CVS 500 and refreshes the picture buffer. A CRA is an intra-coded picture that acts as a random access point without starting a new CVS 500 or refreshing the picture buffer. In this way, a leading picture 504 associated with a CRA may reference a picture before the CRA, but a leading picture 504 associated with an IDR may not reference a picture before the IDR.

[0074] The CVS 500 also includes various non-IRAP pictures. These include a leading picture 504 and a late-end picture 506. The leading picture 504 is a picture positioned after the IRAP picture 502 in decoding order 508 but before the IRAP picture 502 in presentation order 510. The late-end picture 506 is positioned after the IRAP picture 502 in both decoding order 508 and presentation order 510. Both the leading picture 504 and the late-end picture 506 are coded according to inter prediction. The late-end picture 506 is coded with reference to the IRAP picture 502 or a picture positioned after the IRAP picture 502. Thus, the late-end picture 506 can always be decoded once the IRAP picture 502 is decoded. The leading pictures 504 may include random access skip ahead (RASL) pictures and random access decodable ahead (RADL) pictures. The RASL picture is coded with reference to a picture before the IRAP picture 502, but is coded at a position after the IRAP picture 502. Because the RASL picture depends on the previous picture, the RASL picture cannot be decoded when the decoder starts decoding at the IRAP picture 502. Therefore, the RASL picture is skipped and not decoded when the IRAP picture 502 is used as a random access point. However, when the decoder uses the previous IRAP picture (before index 0, not shown) as a random access point, the RASL picture is decoded and displayed. The RADL picture is coded with reference to the IRAP picture 502 and / or a picture after the IRAP picture 502, but is positioned before the IRAP picture 502 in presentation order 510. Because the RADL picture does not depend on the picture before the IRAP picture 502, the RADL picture can be decoded and displayed when the IRAP picture 502 is a random access point.

[0075] Each picture from the CVS 500 may be stored in an access unit. Furthermore, a picture may be partitioned into slices, and slices may be included in NAL units. An NAL unit is a storage unit that includes a parameter set or slice for a picture and a corresponding slice header. NAL units are assigned a type to indicate to a decoder the type of data included in the NAL unit. For example, a slice from the IRAP picture 502 may be included in an IDR with RADL (IDR_W_RADL) NAL unit, an IDR without leading pictures (IDR_N_LP) NAL unit, a CRA NAL unit, etc. The IDR_W_RADL NAL unit indicates that the IRAP picture 502 is an IDR picture associated with a RADL leading picture 504. The IDR_N_LP NAL unit indicates that the IRAP picture 502 is an IDR picture not associated with any leading pictures 504. The CRA NAL unit indicates that the IRAP picture 502 is a CRA picture that may be associated with a leading picture 504. Slices of non-IRAP pictures may also be arranged into NAL units. For example, a slice of the trailing-end picture 506 may be arranged in a trailing-end picture NAL unit type (TRAIL_NUT), which indicates that the trailing-end picture 506 is an inter-predictively coded picture. A slice of the leading picture 504 may be included in a RASL NAL unit type (RASL_NUT) and / or a RADL NAL unit type (RADL_NUT), which may indicate that the corresponding picture is an inter-predictively coded leading picture 504 of the corresponding type. By signaling the slices of a picture in the corresponding NAL unit, a decoder can easily determine the appropriate decoding mechanism to apply to each picture / slice.

[0076] 6 is a schematic diagram illustrating multiple sub-picture video streams 601, 602, and 603 split from a VR picture video stream 600. For example, each of the sub-picture video streams 601-603 and / or the VR picture video stream 600 may be coded in CVS 500. Accordingly, the sub-picture video streams 601-603 and / or the VR picture video stream 600 may be encoded by an encoder, such as codec system 200 and / or encoder 300, in accordance with method 100. Furthermore, the sub-picture video streams 601-603 and / or the VR picture video stream 600 may be decoded by a decoder, such as codec system 200 and / or decoder 400.

[0077] The VR picture video stream 600 includes multiple pictures presented over time. Specifically, VR works by coding a sphere of video content, which can be displayed as if the user were at the center of the sphere. Each picture includes the entire sphere, while only a portion of the picture, known as a viewport, is displayed to the user. For example, a user may utilize a head-mounted display (HMD) that selects and displays a viewport of the sphere based on the user's head movement. This creates the impression of physically being present in the virtual space depicted by the video. To achieve this result, each picture in a video sequence includes a full sphere of video data at the corresponding instant in time. However, only a small portion of the picture (e.g., a single viewport) is displayed to the user. The remainder of the picture is discarded without being rendered. Typically, the entire picture is transmitted so that different viewports can be dynamically selected and displayed in response to the user's head movement.

[0078] In the illustrated example, each picture in the VR picture video stream 600 may be subdivided into sub-pictures based on available viewports. Thus, each picture and corresponding sub-picture has a certain temporal position (e.g., picture order) as part of the temporal presentation. When the sub-division is applied consistently over time, sub-picture video streams 601-603 are created. Such consistent sub-division yields the sub-picture video streams 601-603, each of which includes a set of sub-pictures with a certain size, shape, and spatial position relative to the corresponding picture in the VR picture video stream 600. Furthermore, the sets of sub-pictures in the sub-picture video streams 601-603 vary in temporal position over presentation time. Thus, the sub-pictures in the sub-picture video streams 601-603 may be aligned in the temporal domain based on their temporal positions. The sub-pictures from the sub-picture video streams 601-603 at each time position may then be aggregated in the spatial domain based on the predetermined spatial positions to reconstruct the VR picture video stream 600 for display. Specifically, the sub-picture video streams 601-603 may each be encoded into a separate sub-bitstream. When such sub-bitstreams are aggregated together, they result in a bitstream that includes the entire set of pictures over time. The resulting bitstream may be sent to a decoder for decoding and display based on the user's currently selected viewport.

[0079] One issue with VR video is that all of the sub-picture video streams 601-603 can be transmitted to the user at high quality (e.g., high resolution). This allows the decoder to dynamically select the user's current viewport and display sub-pictures from the corresponding sub-picture video streams 601-603 in real time. However, the user may only see a single viewport, e.g., from the sub-picture video stream 601, while the sub-picture video streams 602-603 are discarded. Therefore, transmitting the sub-picture video streams 602-603 at high quality may waste a large amount of bandwidth. To increase coding efficiency, the VR video may be encoded into multiple video streams 600, with each video stream 600 encoded at a different quality / resolution. In this way, the decoder can transmit a request for the current sub-picture video stream 601. In response, the encoder (or intermediate slicer or other content server) can select a higher quality sub-picture video stream 601 from the higher quality video stream 600 and a lower quality sub-picture video stream 602-603 from the lower quality video stream 600. The encoder can then combine such sub-bitstreams together into a completed encoded bitstream for transmission to the decoder. In this way, the decoder receives a series of pictures where the current viewport is of higher quality and the other viewports are of lower quality. Furthermore, the highest quality sub-pictures are generally displayed to the user (no head movement) and the lower quality sub-pictures are generally discarded, which balances functionality and coding efficiency.

[0080] If the user turns around to look at sub-picture video stream 602 instead of sub-picture video stream 601, the decoder requests that the new current sub-picture video stream 602 be transmitted at a higher quality. The encoder can then modify the integration mechanism accordingly. As noted above, the decoder can start decoding the new CVS 500 only at the IRAP picture 502. Thus, the sub-picture video stream 602 is displayed at a lower quality until the IRAP picture / sub-picture is reached. The IRAP picture can then be decoded at a higher quality to begin decoding the higher quality version of the sub-picture video stream 602. This approach significantly improves video compression without adversely affecting the user's viewing experience.

[0081] One problem with the above approach is that the amount of time required to change resolution is based on the amount of time it takes to reach an IRAP picture in the video stream. This is because the decoder cannot start decoding a different version of the sub-picture video stream 602 at a non-IRAP picture. One approach to reducing such latency is to include more IRAP pictures. However, this results in an increase in file size. To balance functionality and coding efficiency, different viewport / sub-picture video streams 601-603 may include IRAP pictures at different frequencies. For example, viewport / sub-picture video streams 601-603 that are more likely to be viewed may have more IRAP pictures than other viewport / sub-picture video streams 601-603. For example, in a basketball situation, viewport / subpicture video streams 601-603 relating to the basket and / or center court may contain IRAP pictures more frequently than viewport / subpicture video streams 601-603 showing the stands or ceiling, because such viewport / subpicture video streams 601-603 are less likely to be seen by a user.

[0082] This approach leads to additional problems. Specifically, sub-pictures from sub-picture video streams 601-603 that share a POC are part of a single picture. As mentioned above, slices from a picture are included in NAL units based on picture type. In some video coding systems, all NAL units for a single picture are constrained to contain the same NAL unit type. When different sub-picture video streams 601-603 have IRAP pictures with different frequencies, some pictures contain both IRAP and non-IRAP sub-pictures. This violates the constraint that each single picture should only utilize the same type of NAL unit.

[0083] This disclosure addresses this issue by removing the constraint that all NAL units for slices in a picture utilize the same NAL unit type. For example, a picture is contained in an access unit. By removing this constraint, an access unit may contain both IRAP and non-IRAP NAL unit types. Furthermore, a flag may be coded to indicate when a picture / access unit contains a mix of IRAP and non-IRAP NAL unit types. In some examples, this flag is a mixed NAL unit types flag in a picture (mixed_nalu_types_in_pic_flag). Additionally, a constraint may be applied to require that a single mixed picture / access unit may contain only one type of IRAP NAL unit and one type of non-IRAP NAL unit. This prevents unintended mixing of NAL unit types. If such mixing is allowed, the decoder must be designed to manage such mixing. This unnecessarily increases the required hardware complexity without providing any additional benefit to the coding process. For example, a mixed picture may include one type of IRAP NAL unit selected from IDR_W_RADL, IDR_N_LP, or CRA_NUT. Additionally, a mixed picture may include one type of non-IRAP NAL unit selected from TRAIL_NUT, RADL_NUT, and RASL_NUT. An exemplary implementation of this scheme is discussed in more detail below.

[0084] 7 is a schematic diagram illustrating an example bitstream 700 including pictures of mixed NAL unit types. For example, bitstream 700 may be generated by codec system 200 and / or encoder 300 for decoding by codec system 200 and / or decoder 400 according to method 100. Furthermore, bitstream 700 may include VR picture video stream 600 aggregated from multiple sub-picture video streams 601-603 of multiple video resolutions, each sub-picture video stream including CVS 500 at a different spatial location.

[0085] The bitstream 700 includes a sequence parameter set (SPS) 710, one or more picture parameter sets (PPS) 711, multiple slice headers 715, and image data 720. The SPS 710 includes sequence data common to all pictures in a video sequence included in the bitstream 700. Such data may include picture size, bit depth, coding tool parameters, bit rate limits, etc. The PPS 711 includes parameters that apply to the entire picture. Thus, each picture in the video sequence may reference the PPS 711. Note that while each picture references the PPS 711, in some examples, a single PPS 711 may include data for multiple pictures. For example, multiple similar pictures may be coded according to similar parameters. In such cases, a single PPS 711 may include data for such similar pictures. The PPS 711 may indicate the coding tools, quantization parameters, offsets, etc. available for slices in the corresponding picture. The slice header 715 includes parameters specific to each slice in the picture. Thus, there may be one slice header 715 for each slice in a video sequence. The slice header 715 may include slice type information, a picture order count (POC), a reference picture list, prediction weights, tile entry points, deblocking parameters, etc. Note that the slice header 715 may also be referred to as a tile group header in some contexts.

[0086] The image data 720 includes video data coded according to inter-prediction and / or intra-prediction, as well as corresponding transformed and quantized residual data. For example, a video sequence includes multiple pictures 721 coded as image data 720. Because a picture 721 is a single frame of a video sequence, it is generally displayed as a single unit when displaying the video sequence. However, sub-pictures 723 may be displayed to implement certain technologies, such as virtual reality. Each picture 721 references a PPS 711. A picture 721 may be divided into sub-pictures 723, tiles, and / or slices. A sub-picture 723 is a spatial region of a picture 721 that is consistently applied throughout the coded video sequence. Thus, the sub-picture 723 may be displayed by an HMD in the context of VR. Furthermore, a sub-picture 723 with a specified POC may be obtained from the sub-picture video streams 601-603 of the corresponding resolution. The sub-picture 723 may reference an SPS 710. In some systems, a slice 725 is referred to as a tile group that contains the tiles. A slice 725 and / or a tile group of tiles references a slice header 715. A slice 725 may be defined as an integer number of complete tiles or an integer number of consecutive complete CTU rows within a tile of a picture 721 that is exclusively contained in a single NAL unit. Thus, a slice 725 is further divided into CTUs and / or CTBs. The CTUs / CTBs are further divided into coding blocks based on a coding tree. The coding blocks may then be encoded / decoded according to a prediction mechanism.

[0087] Parameter sets and / or slices 725 are coded in NAL units. A NAL unit may be defined as bytes containing a syntax structure that includes an indication of the type of data that follows, and data in the form of RBSPs, interspersed with emulation prevention bytes, if necessary. More specifically, a NAL unit is a storage unit that includes a parameter set or slice 725 of a picture 721 and a corresponding slice header 715. Specifically, a VCL NAL unit 740 is a NAL unit that includes a slice 725 of a picture 721 and a corresponding slice header 715. Furthermore, non-VCL NAL units 730 include parameter sets, such as an SPS 710 and a PPS 711. Several types of NAL units may be utilized. For example, the SPS 710 and the PPS 711 may be included in the SPS NAL unit type (SPS_NUT) 731 and the PPS NAL unit type (PPS_NUT) 732, respectively, both of which are non-VCL NAL units 730.

[0088] As mentioned above, an IRAP picture, such as IRAP picture 502, may be included in an IRAP NAL unit 745. Non-IRAP pictures, such as leading picture 504 and trailing picture 506, may be included in a non-IRAP NAL unit 749. Specifically, an IRAP NAL unit 745 is any NAL unit that includes a slice 725 taken from an IRAP picture or a subpicture. A non-IRAP NAL unit 749 is any NAL unit that includes a slice 725 taken from any picture that is not an IRAP picture or a subpicture (e.g., a leading picture or a trailing picture). Because both IRAP NAL unit 745 and non-IRAP NAL unit 749 include slice data, they are both VCL NAL units 740. In an exemplary embodiment, an IRAP NAL unit 745 may include a slice 725 from an IDR associated with an IDR picture or a RADL picture without a leading picture in an IDR_N_LP NAL unit 741 or an IDR_w_RADL NAL unit 742, respectively. Additionally, an IRAP NAL unit 745 may include a slice 725 from a CRA picture in a CRA_NUT 743. In an exemplary embodiment, a non-IRAP NAL unit 749 may include a slice 725 from an RASL picture, a RADL picture, or a trailing picture in a RASL_NUT 746, a RADL_NUT 747, or a TRAIL_NUT 748, respectively. In an exemplary embodiment, a complete list of possible NAL units is shown below as categorized by NAL unit type:

[0089] [Table 1A]

[0090] [Table 1B]

[0091] [Table 1C]

[0092] As mentioned above, a VR video stream may include sub-pictures 723 with IRAP pictures at different frequencies. This allows fewer IRAP pictures to be used for spatial regions that a user is less likely to view, and more IRAP pictures to be used for spatial regions that a user is more likely to view frequently. In this way, spatial regions that a user is more likely to switch to regularly can be quickly adjusted to a higher resolution. When this approach results in a picture 721 that includes both IRAP NAL units 745 and non-IRAP NAL units 749, the picture 721 is called a mixed picture. This condition may be signaled by the mixed NAL unit type in the picture flag (mixed_nalu_types_in_pic_flag) 727. mixed_nalu_types_in_pic_flag 727 may be set in the PPS 711. Furthermore, mixed_nalu_types_in_pic_flag 727 may be set equal to 1 when specifying that each picture 721 that references the PPS 711 has more than one VCL NAL unit 740 and that the VCL NAL units 740 do not have the same value of NAL unit type (nal_unit_type). Furthermore, mixed_nalu_types_in_pic_flag 727 may be set equal to 0 when each picture 721 that references the PPS 711 has one or more VCL NAL units 740 and the VCL NAL units 740 of each picture 721 that references the PPS 711 all have the same value of nal_unit_type.

[0093] Furthermore, when mixed_nalu_types_in_pic_flag 727 is set, a constraint may be utilized such that one or more VCL NAL units 740 of subpictures 723 of picture 721 all have a first particular value of NAL unit type and all other VCL NAL units 740 in picture 721 have a different second particular value of NAL unit type. For example, the constraint may require that mixed picture 721 include a single type of IRAP NAL unit 745 and a single type of non-IRAP NAL unit 749. For example, picture 721 may include one or more IDR_N_LP NAL units 741, one or more IDR_w_RADL NAL units 742, or one or more CRA_NUT 743, but cannot include any combination of such IRAP NAL units 745. Furthermore, a picture 721 may contain one or more RASL_NUTs 746, one or more RADL_NUTs 747, or one or more TRAIL_NUTs 748, but cannot contain any combination of such IRAP NAL units 745.

[0094] In certain example implementations, picture types are utilized to define the decoding process. Such processes include deriving picture identification information, for example, by a picture order count (POC), marking reference picture status in a decoded picture buffer (DPB), outputting pictures from the DPB, etc. Pictures may be identified by a type based on the NAL unit type, including all or part of the coded picture. In some video coding systems, picture types may include instantaneous decoding refresh (IDR) pictures and non-IDR pictures. In other video coding systems, picture types may include back-end pictures, temporal sub-layer access (TSA) pictures, step-by-step temporal sub-layer access (STSA) pictures, random access decodable ahead (RADL) pictures, random access skip ahead (RASL) pictures, broken link access (BLA) pictures, instantaneous random access pictures, and clean random access pictures. Such picture types may be further distinguished based on whether the picture is a sub-layer reference picture or a sub-layer non-reference picture. BLA pictures may be further distinguished as BLA with a leading picture, BLA with a RADL picture, and BLA without a leading picture, and IDR pictures may be further distinguished as IDR with a RADL picture and IDR without a leading picture.

[0095] Such picture types may be utilized to implement various video-related functions. For example, IDR, BLA, and / or CRA pictures may be utilized to implement an IRAP picture. An IRAP picture may provide the following functions / advantages: The presence of an IRAP picture may indicate that the decoding process can start from that picture. This function enables the implementation of a random access feature, such that the decoding process starts at a specified position in the bitstream as long as the IRAP picture is present at that position. Such a position is not necessarily at the beginning of the bitstream. The presence of an IRAP picture also refreshes the decoding process, such that coded pictures starting at the IRAP picture, except for RASL pictures, are coded without reference to pictures positioned before the IRAP picture. Therefore, an IRAP picture positioned in the bitstream stops the propagation of decoding errors. Therefore, a decoding error of a coded picture positioned before the IRAP picture cannot propagate through the IRAP picture to pictures that follow the IRAP picture in decoding order.

[0096] Although IRAP pictures provide various functions, they also create a penalty in compression efficiency. Therefore, the presence of IRAP pictures can cause a sudden increase in bit rate. This penalty in compression efficiency has various causes. For example, IRAP pictures are intra-predicted pictures that are represented by many more bits than inter-predicted pictures used as non-IRAP pictures. Furthermore, the presence of IRAP pictures disrupts the temporal prediction used in inter prediction. Specifically, IRAP pictures refresh the decoding process by removing previous reference pictures from the DPB. Removing previous reference pictures reduces the availability of reference pictures for use in coding pictures that follow the IRAP picture in decoding order, thereby reducing the efficiency of this process.

[0097] IDR pictures may utilize a different signaling and derivation process than other IRAP picture types. For example, the IDR-related signaling and derivation process may set the most significant bit (MSB) portion of the POC to 0 rather than deriving the MSB from the previous key picture. Furthermore, the slice header of an IDR picture may not contain information used to assist reference picture management. On the other hand, other picture types, such as CRA, back-end, and TSA, may contain reference picture information, such as a reference picture set (RPS) or reference picture list, which may be utilized to implement the reference picture marking process. The reference picture marking process is the process of determining the status of a reference picture in the DPB as either used for reference or not used for reference. In an IDR picture, such information does not need to be signaled because the presence of IDR indicates that the decoding process should simply mark all reference pictures in the DPB as not used for reference.

[0098] In addition to picture type, picture identification by POC is also utilized for multiple purposes, such as managing the use of reference pictures in inter prediction, outputting pictures from the DPB, scaling motion vectors, and weighted prediction. For example, in some video coding systems, pictures in the DPB may be marked as used for short-term reference, used for long-term reference, or unused for reference. When a picture is marked as unused for reference, it can no longer be used for prediction. When such a picture is no longer needed for output, it may be removed from the DPB. In other video coding systems, reference pictures may be marked as short-term and long-term. A reference picture may be marked as unused for reference when it is no longer needed for prediction reference. The conversion between these statuses may be controlled by a decoded reference picture marking process. An implicit sliding window process and / or an explicit memory management control operation (MMCO) process may be utilized as the decoded reference picture marking mechanism. The sliding window process marks a short-term reference picture as unused for reference when the number of reference frames equals a specified maximum number, denoted as max_num_ref_frames in the SPS. Short-term reference pictures may be stored in a first-in first-out manner, so that the most recently decoded short-term pictures are kept in the DPB. An explicit MMCO process may include multiple MMCO instructions. An MMCO instruction may mark one or more short-term or long-term reference pictures as unused for reference, mark all pictures as unused for reference, or mark the current reference picture or an existing short-term reference picture as a long-term reference picture and assign a long-term picture index to the long-term reference picture.

[0099] In some video coding systems, the reference picture marking operation and the process for outputting and removing pictures from the DPB are performed after a picture is decoded. Other video coding systems utilize the RPS for reference picture management. The most fundamental difference between the RPS mechanism and the MMCO / sliding window process is that, for each particular slice, the RPS provides the complete set of reference pictures used by the current picture or any subsequent pictures. Thus, the complete set of all pictures to be kept in the DPB for use by the current picture or future pictures is signaled in the RPS. This differs from the MMCO / sliding window method, in which only relative changes to the DPB are signaled. With the RPS mechanism, information from earlier pictures in the decoding order is not required to maintain the correct status of reference pictures in the DPB. To take advantage of the RPS to improve error resilience, the order of picture decoding and DPB operations is changed in some video coding systems. In some video coding systems, picture marking and buffering operations, including both outputting and removing decoded pictures from the DPB, may be applied after the current picture is decoded. In other video coding systems, the RPS may be first decoded from the slice header of the current picture, and then picture marking and buffering operations may be applied before decoding the current picture.

[0100] In VVC, the reference picture management approach can be summarized as follows: Two reference picture lists, denoted List 0 and List 1, are directly signaled and derived. They are not based on the RPS or sliding window and MMCO processes discussed above. Reference picture marking is directly based on Reference Picture Lists 0 and 1, which utilize both active and inactive entries in the reference picture lists, but only active entries can be used as reference indexes in inter prediction of a CTU. Information for the derivation of the two reference picture lists is signaled by syntax elements and syntax structures in the SPS, PPS, and slice header. A predefined RPL structure is signaled in the SPS for use by referencing it in the slice header. Two reference picture lists are generated for all types of slices, including bidirectional inter-predicted (B) slices, unidirectional inter-predicted (P) slices, and intra-predicted (I) slices. The two reference picture lists can be constructed without using the reference picture list initialization process or the reference picture list modification process. A long-term reference picture (LTRP) is identified by the POC LSB. A delta POC MSB cycle can be signaled for the LTRP as determined on a picture-by-picture basis.

[0101] To code a video image, the image is first partitioned, and the partitions are coded into a bitstream. Various picture partitioning schemes are available. For example, an image may be partitioned into regular slices, dependent slices, tiles, and / or according to wavefront parallelism (WPP). For simplicity, HEVC constrains the encoder to use regular slices, dependent slices, tiles, WPP, and combinations thereof when partitioning slices into groups of CTBs for video coding. Such partitioning may be applied to support maximum transmission unit (MTU) size adaptation, parallel processing, and reduced end-to-end delay. The MTU represents the maximum amount of data that can be transmitted in a single packet. If a packet payload exceeds the MTU, the payload is split into two packets through a process called fragmentation.

[0102] A regular slice, also referred to simply as a slice, is a partitioned portion of an image that can be reconstructed independently of other regular slices within the same picture, despite some interdependence due to loop filtering operations. Each regular slice is encapsulated in a unique Network Abstraction Layer (NAL) unit for transmission. Furthermore, intra-picture prediction (intra-sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across slice boundaries can be disabled to support independent reconstruction. Such independent reconstruction supports parallelization. For example, regular slice-based parallelization utilizes minimal inter-processor or inter-core communication. However, because each regular slice is independent, each slice is associated with a separate slice header. The use of regular slices can incur significant coding overhead due to the bit cost of the slice header for each slice and the lack of prediction across slice boundaries. Furthermore, regular slices can be utilized to support compliance with MTU size requirements. Specifically, because regular slices are encapsulated in separate NAL units and can be coded independently, each regular slice must be smaller than the MTU in the MTU scheme to avoid breaking the slice into multiple packets. Thus, the goals of parallelization and MTU size adaptation may impose conflicting requirements on the slice layout in a picture.

[0103] Dependent slices are similar to regular slices but have shortened slice headers, allowing for partitioning of picture treeblock boundaries without breaking intra-picture prediction. Dependent slices therefore allow regular slices to be fragmented into multiple NAL units, which results in reduced end-to-end delay by allowing parts of a regular slice to be sent out before the coding of the entire regular slice is complete.

[0104] A picture may be divided into tile groups / slices and tiles. A tile is a sequence of CTUs that covers a rectangular area of the picture. A tile group / slice contains a certain number of tiles of the picture. Raster scan tile group mode and rectangular tile group mode may be used to create tiles. In raster scan tile group mode, a tile group contains a sequence of tiles in a tile raster scan of the picture. In rectangular tile group mode, a tile group contains a certain number of tiles of the picture that collectively form a rectangular area of the picture. The tiles within a rectangular tile group are in the order of the tile raster scan of the tile group. For example, a tile may be a partitioned portion of an image created by horizontal and vertical boundaries that create columns and rows of tiles. Tiles may be coded in raster scan order (right to left and top to bottom). The scan order of CTBs is local within a tile. Therefore, the CTB of the first tile is coded in raster scan order before proceeding to the CTB of the next tile. Similar to regular slices, tiles break the dependency of intra-picture prediction as well as the dependency of entropy decoding. However, because tiles may not be included in individual NAL units, tiles may not be used for MTU size adaptation. Each tile may be processed by one processor / core, and inter-processor / inter-core communication utilized for intra-picture prediction between processing units decoding neighboring tiles may be limited to carrying a shared slice header (when adjacent tiles are in the same slice) and performing loop filtering-related sharing of reconstructed samples and metadata. When more than one tile is included in a slice, entry point byte offsets for each tile other than the first entry point offset in the slice may be signaled in the slice header. For each slice and tile, at least one of the following conditions should be satisfied: 1) all coded treeblocks in the slice belong to the same slice, and 2) all coded blocks in the tile belong to the same slice.

[0105] In WPP, an image is partitioned into single rows of CTBs. The entropy decoding and prediction mechanisms may use data from CTBs in other rows. Parallel processing is enabled through parallel decoding of CTB rows. For example, the current row can be decoded in parallel with the previous row. However, the decoding of the current row lags behind the decoding process of the previous row by the two CTBs. This delay ensures that data for the CTBs above and to the right of the current CTB in the current row is available before the current CTB is coded. This approach appears graphically as a wavefront. This staggered start allows parallelization with up to the same number of processors / cores as the number of CTB rows the image contains. Because intra-picture prediction between neighboring treeblock rows within a picture is allowed, inter-processor / inter-core communication to enable intra-picture prediction can be significant. WPP partitioning takes into account the NAL unit size. Therefore, WPP does not support MTU size adaptation. However, regular slicing can be used with WPP, with some coding overhead, to perform MTU size adaptation if required. Finally, a wavefront segment may contain exactly one CTB row. Furthermore, when using WPP, and a slice starts within a CTB row, the slice should end on the same CTB row.

[0106] A tile may also include a motion constrained tile set. A motion constrained tile set (MCTS) is a tile set designed such that associated motion vectors are constrained to point to integer sample positions within the MCTS and fractional sample positions that require only integer sample positions within the MCTS for interpolation. Furthermore, the use of motion vector candidates for temporal motion vector prediction derived from blocks outside the MCTS is not permitted. In this way, each MCTS can be independently decoded without the presence of tiles not included in the MCTS. A temporal MCTS Supplemental Enhancement Information (SEI) message indicates the presence of an MCTS in a bitstream and can be used to signal the MCTS. The MCTS SEI message provides supplemental information that can be used in MCTS sub-bitstream extraction (defined as part of the semantics of the SEI message) to generate a conforming bitstream for the MCTS set. The information includes several extraction information sets, each of which defines the number of MTCS sets and contains raw byte sequence payload (RBSP) bytes of replacement video parameter sets (VPS), sequence parameter sets (SPS), and picture parameter sets (PPS) to be used during the MCTS sub-bitstream extraction process. When extracting a sub-bitstream according to the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, and PPS) may be rewritten or replaced, and the slice header may be updated, because one or all of the slice address-related syntax elements (including first_slice_segment_in_pic_flag and slice_segment_address) may utilize different values in the extracted sub-bitstream.

[0107] VR applications, also known as 360-degree video applications, may display only a portion of a complete sphere and, consequently, only a subset of the entire picture. Viewport-dependent 360-degree delivery via the Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH) mechanism may be used to reduce the bitrate and support delivery of 360-degree video via a streaming mechanism. This mechanism splits the sphere / projected picture into multiple MCTSs, for example, by using cube-map projection (CMP). Two or more bitstreams with different spatial resolutions or qualities may be encoded. When delivering data to a decoder, an MCTS from a higher-resolution / high-quality bitstream is transmitted for the viewport to be displayed (e.g., the previous viewport). An MCTS from a lower-resolution / lower-quality bitstream is transmitted for the other viewport. These MCTSs are packed in some manner and then transmitted to the receiver for decoding. The viewport seen by the user is expected to be represented by a high-resolution / high-quality MCTS to create a good viewing experience. When the user's head moves to look at a different viewport (e.g., a left or right viewport), the displayed content comes from the lower resolution / lower quality viewport for a short period of time while the system fetches a higher resolution / higher quality MCTS for the new viewport. When the user's head moves to look at a different viewport, there is a delay between the time of the user's head movement and the time the higher resolution / higher quality representation of the viewport is seen. This delay depends on how quickly the system can fetch the higher resolution / higher quality MCTS for that viewport, which in turn depends on the IRAP period. The IRAP period is the interval between the occurrence of two IRAPs. This delay is related to the IRAP period because the MCTS for the new viewport can only be decodable from the IRAP picture.

[0108] For example, if the IRAP period is coded every second, the following applies: The best-case scenario for delay is the same as the network round-trip delay when the user's head moves to look at the new viewport just before the system starts fetching the new segment / IRAP period. In this scenario, the system can immediately request a higher-resolution / higher-quality MCTS for the new viewport, so the only delay is the network round-trip delay, which is the delay between the request to fetch and the transmission of the requested MCTS, assuming that the minimum buffer delay can be set to near zero and that sensor delay is small and negligible. The network round-trip delay may be around 200 milliseconds, for example. The worst-case scenario for delay is the IRAP period plus the network round-trip delay when the user's head moves to look at the new viewport just after the system has already made a request for the next segment. To improve on the worst-case scenario above, the bitstream can be coded using more frequent IRAP pictures so that the IRAP period is shorter, as this reduces the overall delay. However, this approach increases bandwidth requirements due to reduced compression efficiency.

[0109] In one example implementation, sub-pictures of the same coded picture are allowed to contain different nal_unit_type values. This mechanism is explained as follows: A picture may be divided into sub-pictures. A sub-picture is a rectangular set of tile groups / slices starting with the tile group with tile_group_address equal to 0. Each sub-picture may have a separate tile partition, as it may reference a corresponding PPS. The presence of a sub-picture may be indicated in the PPS. Each sub-picture is treated like a picture in the decoding process. In-loop filtering across sub-picture boundaries may always be disabled. The width and height of a sub-picture may be specified in units of the luma CTU size. The position of a sub-picture in a picture may not be signaled but may be derived using the following rules: A sub-picture occupies the next unoccupied position in the CTU raster scan order within a picture that is large enough to contain the sub-picture within the picture's boundaries. The reference picture for decoding each subpicture is generated by extracting an area from a reference picture in the decoded picture buffer that is co-located with the current subpicture. Since the extracted area is the decoded subpicture, inter-prediction is performed between subpictures of the same size and the same location within the picture. In such cases, allowing different nal_unit_type values within a coded picture allows subpictures originating from random access pictures and subpictures originating from non-random access pictures to be merged into the same coded picture without substantial difficulty (e.g., without VCL-level modifications). Such benefits also apply to MCTS-based coding.

[0110] Allowing different nal_unit_type values within a coded picture may be beneficial in other scenarios. For example, a user may view some areas of 360-degree video content more frequently than other areas. To obtain a better trade-off between coding efficiency and average comparable-quality viewport switching latency in MCTS / subpicture-based viewport-dependent 360-degree video delivery, more frequent IRAP pictures can be coded for areas that are more commonly viewed than other areas. Comparable-quality viewport switching latency is the latency a user experiences when switching from a first viewport to a second viewport until the presentation quality of the second viewport reaches a presentation quality comparable to that of the first viewport.

[0111] Another implementation utilizes the following solution for support of mixed NAL unit types within a picture, including POC derivation and reference picture management: A flag (sps_mixed_tile_groups_in_pic_flag) exists in the parameter set, which is directly or indirectly referenced by the tile group, to specify whether there can be a picture with mixed IRAP and non-IRAP subpictures. For NAL units containing IDR tile groups, a flag (poc_msb_reset_flag) exists in the corresponding tile group header to specify whether the POC MSB is reset in the POC derivation for the picture. A variable called PicRefreshFlag is defined and associated with a picture. This flag specifies whether the POC derivation and DPB state should be refreshed when decoding the picture. The value of PicRefreshFlag is derived as follows: If the current tile group is contained in the first access unit in the bitstream, PicRefreshFlag is set equal to 1. Otherwise, if the current tile group is an IDR tile group, PicRefreshFlag is set equal to sps_mixed_tile_groups_in_pic_flag?poc_msb_reset_flag:1. Otherwise, if the current tile group is a CRA tile group, the following applies: If the current access unit is the first access unit of a coded sequence, PicRefreshFlag is set equal to 1. The current access unit is the first access unit of a coded sequence when the access unit immediately follows the last of the sequence NAL units or when the associated variable HandleCraAsFirstPicInCvsFlag is set equal to 1. Otherwise (for example, the current tile group does not belong to the first access unit in the bitstream and is not an IRAP tile group), PicRefreshFlag is set equal to 0.

[0112] When PicRefreshFlag is equal to 1, the value of the POC MSB (PicOrderCntMsb) is reset to 0 during the derivation of the POC for a picture. Information utilized for reference picture management, such as a reference picture set (RPS) or a reference picture list (RPL), is signaled in the tile group / slice header regardless of the corresponding NAL unit type. A reference picture list is constructed at the beginning of decoding of each tile group regardless of the NAL unit type. The reference picture list may include RefPicList[0] and RefPicList[1] for the RPL approach, RefPicList0[] and RefPicList1[] for the RPS approach, or similar lists containing reference pictures for inter-prediction operations of the picture. When PicRefreshFlag is equal to 1, during the reference picture marking process, all reference pictures in the DPB are marked as unused for reference.

[0113] Such an implementation is associated with several problems. For example, when mixing nal_unit_type values within a picture is not allowed, and when the derivation of whether a picture is an IRAP picture and the derivation of the variable NoRaslOutputFlag are described at the picture level, a decoder can perform these derivations after receiving the first VCL NAL unit of any picture. However, with support for mixed NAL unit types within a picture, the decoder must wait for the arrival of other VCL NAL units of the picture before performing the above derivations. In the worst case, the decoder must wait for the arrival of the last VCL NAL unit of the picture. Furthermore, such a system may signal a flag in the tile group header of an IDR NAL unit to specify whether the POC MSB is reset in the POC derivation for the picture. This mechanism has the following problems: The case of mixed IRAP NAL unit types and non-IRAP NAL unit types is not supported by this mechanism. Furthermore, signaling this information in the tile-group / slice headers of VCL NAL units requires a value change during bitstream extraction or aggregation when there is a change to the status of whether IRAP (IDR or CRA) NAL units are mixed with non-IRAP NAL units in a picture. Such rewriting of slice headers occurs whenever a user requests video, requiring a large amount of hardware resources. Furthermore, any other mixing of different NAL unit types within a picture other than mixing specific IRAP and specific non-IRAP NAL unit types is allowed. Such flexibility complicates the codec design while not supporting real-world use cases, which unnecessarily increases the decoder complexity and therefore the associated implementation costs.

[0114] In general, this disclosure describes techniques for supporting sub-picture or MCTS-based random access in video coding. More specifically, this disclosure describes an improved design for supporting mixed NAL unit types within a picture, which is utilized to support sub-picture or MCTS-based random access. The description of the techniques is based on the VVC standard, but also applies to other video / media codec specifications.

[0115] To solve the above problems, the following exemplary implementations are disclosed. Such implementations may be applied individually or in combination. In one example, each picture is associated with an indication of whether the picture contains mixed nal_unit_type values. This indication is signaled in the PPS. This indication supports the decision of whether to reset the POC MSB and / or whether to reset the DPB by marking all reference pictures as unused for reference. When this indication is signaled in the PPS, changes to the value of the PPS may be made during aggregation or separate extraction. However, this is acceptable because the PPS is rewritten and replaced by other mechanisms during such bitstream extraction or aggregation.

[0116] Alternatively, this indication may be signaled in the tile group header but required to be the same for all tile groups of a picture. However, in this case, the value may need to be changed during sub-bitstream extraction of the MCTS / sub-picture sequence. Alternatively, this indication may be signaled in the NAL unit header but required to be the same for all tile groups of a picture. However, in this case, the value may need to be changed during sub-bitstream extraction of the MCTS / sub-picture sequence. Alternatively, this indication may be signaled by defining such an additional VCL NAL unit type as, when used for a picture, all VCL NAL units of that picture should have the same NAL unit type value. However, in this case, the NAL unit type value of the VCL NAL unit may need to be changed during sub-bitstream extraction of the MCTS / sub-picture sequence. Alternatively, this indication may be signaled by defining such an additional IRAP VCL NAL unit type as, when used for a picture, all VCL NAL units of that picture should have the same NAL unit type value. However, in this case, the NAL unit type value of the VCL NAL unit may need to be changed during sub-bitstream extraction of the MCTS / sub-picture sequence. Alternatively, each picture that has at least one VCL NAL unit with any of the IRAP NAL unit types may be associated with an indication of whether the picture contains mixed NAL unit type values.

[0117] Furthermore, constraints may be applied such that a limited mix of nal_unit_type values within a picture is allowed by only allowing mixed IRAP and non-IRAP NAL unit types. For any particular picture, either all VCL NAL units have the same NAL unit type, or some VCL NAL units have a specific IRAP NAL unit type, and the rest have a specific non-IRAP VCL NAL unit type. In other words, no VCL NAL unit of any particular picture can have more than one IRAP NAL unit type and no VCL NAL unit type can have more than one non-IRAP NAL unit type. A picture may be considered to be an IRAP picture only if it does not contain mixed nal_unit_type values and if no VCL NAL units have IRAP NAL unit types. For any IRAP NAL units (including IDRs) that do not belong to an IRAP picture, the POC MSB may not be reset. For any IRAP NAL unit (including IDR) that does not belong to an IRAP picture, the DPB is not reset, so marking all reference pictures as unused for referencing is not performed. TemporalId may be set equal to 0 for a picture if at least one VCL NAL unit of the picture is an IRAP NAL unit.

[0118] The following are specific implementations of one or more of the aspects described above: An IRAP picture may be defined as a coded picture in which the value of mixed_nalu_types_in_pic_flag is equal to 0 and each VCL NAL unit has a nal_unit_type ranging from IDR_W_RADL to RSV_IRAP_VCL13, inclusive. Exemplary PPS syntax and semantics are as follows:

[0119] [Table 2]

[0120] mixed_nalu_types_in_pic_flag is set equal to 0 to specify that each picture that references a PPS has multiple VCL NAL units and that these NAL units do not have the same value of nal_unit_type. mixed_nalu_types_in_pic_flag is set equal to 0 to specify that the VCL NAL units of each picture that references a PPS have the same value of nal_unit_type.

[0121] An example tile group / slice header syntax is as follows:

[0122] [Table 3A]

[0123] [Table 3B]

[0124] The semantics of an exemplary NAL unit header are as follows: For VCL NAL units of any particular picture, one of the following two conditions should be met: All VCL NAL units have the same value of nal_unit_type. Some VCL NAL units have a specific IRAP NAL unit type value (i.e., values of nal_unit_type in the range from IDR_W_RADL to RSV_IRAP_VCL13, inclusive), while all other VCL NAL units have a specific non-IRAP VCL NAL unit type (i.e., values of nal_unit_type in the range from TRAIL_NUT to RSV_VCL_7, inclusive, or in the range from RSV_VCL14 to RSV_VCL15, inclusive). nuh_temporal_id_plus1 minus 1 specifies the temporal identifier of the NAL unit. The value of nuh_temporal_id_plus1 shall not be equal to 0.

[0125] The variable TemporalId is derived as follows: TemporalId=nuh_temporal_id_plus1-1 (7-1)

[0126] For VCL NAL units of a picture, when nal_unit_type is in the range from IDR_W_RADL to RSV_IRAP_VCL13, inclusive, TemporalId shall be equal to 0 for all VCL NAL units of the picture, regardless of the nal_unit_type values of the other VCL NAL units of the picture. The value of TemporalId shall be the same for all VCL NAL units of an access unit. The value of TemporalId of a coded picture or access unit is the value of TemporalId of the VCL NAL units of the coded picture or access unit.

[0127] An exemplary decoding process for a coded picture is as follows: The decoding process operates as follows for the current picture, CurrPic: NAL unit decoding is specified here. The following decoding process uses syntax elements at and above the tile group header layer. The variables and functions for the picture order count are derived as specified herein. This is called only for the first tile group / slice of the picture. At the beginning of the decoding process for each tile group / slice, the decoding process for reference picture list construction is called to derive reference picture list 0 (RefPicList[0]) and reference picture list 1 (RefPicList[1]). If the current picture is an IDR picture, the decoding process for reference picture list construction may be called for bitstream conformance checking purposes, but may not be necessary for decoding the current picture or pictures after the current picture in decoding order.

[0128] The decoding process for reference picture list construction is as follows: This process is invoked at the beginning of the decoding process for each tile group. Reference pictures are addressed through reference indexes, which are indexes into the reference picture list. When decoding an I tile group, no reference picture list is used in decoding the tile group data. When decoding a P tile group, only reference picture list 0 (RefPicList[0]) is used in decoding the tile group data. When decoding a B tile group, both reference picture list 0 and reference picture list 1 (RefPicList[1]) are used in decoding the tile group data. At the beginning of the decoding process for each tile group, reference picture lists RefPicList[0] and RefPicList[1] are derived. The reference picture list is used in marking reference pictures or in decoding the tile group data. For any tile group of an IDR picture or an I tile group of a non-IDR picture, RefPicList[0] and RefPicList[1] may be derived for the purpose of bitstream conformance checking, but their derivation is not required for decoding the current picture or pictures that follow the current picture in decoding order. For P tile groups, RefPicList[1] may be derived for the purpose of bitstream conformance checking, but their derivation is not required for decoding the current picture or pictures that follow the current picture in decoding order.

[0129] 8 is a schematic diagram of an exemplary video coding device 800. The video coding device 800 is suitable for implementing the disclosed examples / embodiments as described herein. The video coding device 800 includes a downstream port 820, an upstream port 850, and / or a transceiver unit (Tx / Rx) 810 including a transmitter and / or receiver for communicating data upstream and / or downstream over a network. The video coding device 800 also includes a processor 830 including a logic unit and / or central processing unit (CPU) for processing data and a memory 832 for storing data. The video coding device 800 may also include electrical components, optical-electrical (OE) components, electrical-optical (EO) components, and / or wireless communication components coupled to the upstream port 850 and / or downstream port 820 for communication of data over a telecommunications network, an optical communication network, or a wireless communication network. The video coding device 800 may also include an input and / or output (I / O) device 860 for communicating data to and from a user. The I / O devices 860 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices 860 may also include input devices such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0130] The processor 830 is implemented by hardware and software. The processor 830 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 830 is in communication with the downstream port 820, the Tx / Rx 810, the upstream port 850, and the memory 832. The processor 830 includes a coding module 814. The coding module 814 performs the disclosed embodiments described herein, such as methods 100, 900, and 1000, which may utilize the CVS 500, the VR picture video stream 600, and / or the bitstream 700. The coding module 814 may also implement any other method / mechanism described herein. Additionally, the coding module 814 may implement the codec system 200, the encoder 300, and / or the decoder 400. For example, coding module 814 may set flags in a PPS to indicate when a picture contains both IRAP NAL units and non-IRAP NAL units, and to constrain such pictures to contain only a single type of IRAP NAL units and a single type of non-IRAP NAL units. Thus, coding module 814 enables video coding device 800 to provide additional functionality and / or coding efficiency when coding video data. Thus, coding module 814 improves the functionality of video coding device 800 as well as addresses problems specific to video coding techniques. Furthermore, coding module 814 implements transformations of video coding device 800 to different states. Alternatively, coding module 814 may be implemented as instructions stored in memory 832 and executed by processor 830 (e.g., as a computer program product stored on a non-transitory medium).

[0131] The memory 832 comprises one or more memory types such as a disk, a tape drive, a solid-state drive, a read-only memory (ROM), a random access memory (RAM), a flash memory, a ternary content addressable memory (TCAM), a static random access memory (SRAM), etc. The memory 832 may be used as an overflow data storage device to store programs when such programs are selected for execution and to store instructions and data read during program execution.

[0132] 9 is a flowchart of an example method 900 for encoding a video sequence, such as CVS 500, that includes pictures with mixed NAL unit types from multiple sub-picture video streams 601-603 at multiple video resolutions into a bitstream, such as bitstream 700, that includes integrated VR picture video stream 600. Method 900 may be utilized by an encoder, such as codec system 200, encoder 300, and / or video coding device 800, when performing method 100.

[0133] Method 900 may begin when an encoder receives a video sequence including multiple pictures, such as VR pictures, and determines to encode the video sequence into a bitstream based on, for example, user input. In step 901, the encoder determines that a picture of the video sequence includes multiple sub-pictures of different types.

[0134] In step 903, the encoder encodes the sub-pictures of the picture into multiple VCL NAL units in the bitstream.

[0135] In step 905, the encoder encodes the PPS into a bitstream. The encoder also encodes a flag into the PPS, and thus into the bitstream. The flag is set to indicate that one or more VCL NAL units of the picture's subpictures all have a first specific value of NAL unit type and that the other VCL NAL units in the picture all have a second, different value of NAL unit type. For example, the first specific value of NAL unit type may indicate that the picture contains a single type of IRAP subpicture. As a specific example, the first specific value of NAL unit type may be equal to one of IDR_W_RADL, IDR_N_LP, or CRA_NUT. Thus, a picture may have any number of IRAP subpictures, but all of the IRAP subpictures must be of the same type (e.g., only one of IDR_W_RADL, IDR_N_LP, or CRA_NUT). Furthermore, the second particular value of the NAL unit type may indicate that the picture contains non-IRAP sub-pictures of a single type. As a particular example, the second particular value of the NAL unit type may be equal to TRAIL_NUT, RADL_NUT, or RASL_NUT. Thus, a picture may have any number of non-IRAP sub-pictures, but all of the non-IRAP sub-pictures must be of the same type (e.g., only one of TRAIL_NUT, RADL_NUT, or RASL_NUT).

[0136] In step 907, the encoder stores the bitstream including a flag for communication to the decoder. In some examples, the flag is mixed_nalu_types_in_pic_flag. In one particular example, mixed_nalu_types_in_pic_flag may be set equal to 1 when specifying that each picture that references the PPS has more than one VCL NAL unit and that the VCL NAL units do not have the same value of NAL unit type (nal_unit_type). Furthermore, mixed_nalu_types_in_pic_flag may be set equal to 0 when each picture that references the PPS has one or more VCL NAL units and the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type.

[0137] 10 is a flowchart of an example method 1000 of decoding a video sequence, such as CVS 500, that includes pictures of mixed NAL unit types from a bitstream, such as bitstream 700, that includes VR picture video stream 600 aggregated from multiple sub-picture video streams 601-603 of multiple video resolutions. Method 1000 may be utilized by a decoder, such as codec system 200, decoder 400, and / or video coding device 800, when performing method 100.

[0138] Method 1000 may begin when a decoder begins receiving a bitstream of coded data representing a video sequence, e.g., as a result of method 900. In step 1001, the decoder receives the bitstream. The bitstream comprises a flag associated with a picture and a plurality of subpictures. The subpictures are partitioned into slices, and the slices are contained in VCL NAL units. Thus, each of the plurality of subpictures is also contained in a plurality of VCL NAL units. The bitstream may also include a PPS. In some examples, the PPS includes a flag. As a particular example, the flag may be mixed_nalu_types_in_pic_flag. Furthermore, mixed_nalu_types_in_pic_flag may be set equal to 1 to specify that each picture that references the PPS has more than one VCL NAL unit and that the VCL NAL units do not have the same value of nal_unit_type. Additionally, mixed_nalu_types_in_pic_flag may be set equal to 0 when each picture that references a PPS has one or more VCL NAL units and the VCL NAL units of each picture that references a PPS have the same value of nal_unit_type.

[0139] In step 1003, the decoder determines, based on the value of the flag, that one or more VCL NAL units of the subpictures of the picture all have a first specific value of NAL unit type and that other (e.g., remaining) VCL NAL units in the picture all have a different second specific value of NAL unit type. For example, the first specific value of NAL unit type may indicate that the picture includes IRAP subpictures of a single type. As a specific example, the first specific value of NAL unit type may be equal to one of IDR_W_RADL, IDR_N_LP, or CRA_NUT. Thus, a picture may have any number of IRAP subpictures, but all of the IRAP subpictures must be of the same type (e.g., only one of IDR_W_RADL, IDR_N_LP, or CRA_NUT). Furthermore, the second specific value of NAL unit type may indicate that the picture includes non-IRAP subpictures of a single type. As one particular example, the second particular value of the NAL unit type may be equal to TRAIL_NUT, RADL_NUT, or RASL_NUT. Thus, a picture may have any number of non-IRAP sub-pictures, but all of the non-IRAP sub-pictures must be of the same type (e.g., only one of TRAIL_NUT, RADL_NUT, or RASL_NUT).

[0140] In step 1005, the decoder decodes one or more of the sub-pictures based on the first particular value of the NAL unit type and the second particular value of the NAL unit type.

[0141] In step 1007, one or more of the sub-pictures are transferred for display as part of the decoded video sequence.

[0142] 11 is a schematic diagram of an example system 1100 for coding a video sequence, such as CVS 500, including pictures of mixed NAL unit types from multiple sub-picture video streams 601-603 at multiple video resolutions into a bitstream, such as bitstream 700 including integrated VR picture video stream 600. System 1100 may be implemented by an encoder and decoder, such as codec system 200, encoder 300, decoder 400, and / or video coding device 800. Furthermore, system 1100 may be utilized when implementing methods 100, 900, and / or 1000.

[0143] The system 1100 includes a video encoder 1102. The video encoder 1102 comprises a determination module 1101 for determining that a picture includes multiple sub-pictures of different types. The video encoder 1102 further comprises an encoding module 1103 for encoding the sub-pictures of the picture into multiple VCL NAL units in a bitstream. The encoding module 1103 is further for encoding into the bitstream a flag that is set to indicate that one or more VCL NAL units of the sub-pictures of the picture all have a first particular value of the NAL unit type and that other VCL NAL units in the picture all have a second particular value of the NAL unit type. The video encoder 1102 further comprises a storage module 1105 for storing the bitstream for communication to a decoder. The video encoder 1102 further comprises a transmission module 1107 for transmitting the bitstream to the video decoder 1110. The video encoder 1102 may be further configured to perform any of the steps of the method 900.

[0144] The system 1100 also includes a video decoder 1110. The video decoder 1110 comprises a receiving module 1111 for receiving a bitstream comprising a flag associated with a picture and multiple sub-pictures, the multiple sub-pictures included in multiple VCL NAL units. The video decoder 1110 further comprises a determining module 1113 for determining, based on the value of the flag, that one or more VCL NAL units of the sub-pictures of the picture all have a first specific value of the NAL unit type and that other VCL NAL units in the picture all have a different second specific value of the NAL unit type. The video decoder 1110 further comprises a decoding module 1115 for decoding one or more of the sub-pictures based on the first specific value of the NAL unit type and the second specific value of the NAL unit type. The video decoder 1110 further comprises a transport module 1117 for transporting one or more of the sub-pictures for display as part of a decoded video sequence. The video decoder 1110 may be further configured to perform any of the steps of the method 1000.

[0145] A first component is directly coupled to a second component when there are no intervening components other than a line, wire, or another medium between the first and second components. A first component is indirectly coupled to a second component when there are intervening components other than a line, wire, or another medium between the first and second components. The term "coupled" and variations thereof include both directly coupled and indirectly coupled. The use of the term "about," unless otherwise stated, means a range that includes ±10% of the number thereafter.

[0146] It should also be understood that the steps of the exemplary methods described herein do not necessarily have to be performed in the order described, and that the order of steps in such methods should be understood to be merely exemplary. Similarly, additional steps may be included in such methods, and some steps may be omitted or combined, in a manner consistent with various embodiments of the present disclosure.

[0147] Although several embodiments have been provided in this disclosure, it will be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the disclosure. The examples should be considered illustrative rather than limiting, and the intention is not to be limited to the details provided herein. For example, in another system, various elements or components may be combined or integrated, or certain features may be omitted or not implemented.

[0148] Additionally, the techniques, systems, subsystems, and methods described and illustrated in various embodiments as individual or separate may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations will be ascertainable by those skilled in the art and could be made without departing from the spirit and scope disclosed herein. [Explanation of symbols]

[0149] 200 Codec System 201 segmented video signal 211 General-purpose coder control component 213 Transform Scaling and Quantization Components 215 Intra-picture Estimation Component 217 Intra-picture Prediction Component 219 Motion Compensation Component 221 Motion Estimation Component 223 Decoded Picture Buffer Component 225 In-Loop Filter Components 227 Filter Control Analysis Component 229 Scaling and Inverse Transformation Components 231 Header Formatting and CABAC Components 300 Encoder 301 Segmented Video Signal 313 Transform and Quantize Components 317 Intra-picture Prediction Component 321 Motion Compensation Component 323 Decoded Picture Buffer Component 325 In-Loop Filter Components 329 Inverse Transform and Quantization Components 331 Entropy Coding Component 400 decoder 417 Intra-picture Prediction Component 421 Motion Compensation Component 423 Decoded Picture Buffer Component 425 In-Loop Filter Components 429 Inverse Transform and Quantization Components 433 Entropy Decoding Component 500 CVS 502 IRAP Picture 504 Leading Picture 506 Rear Picture 508 Decoding Order 510 Presentation order 600 VR picture video streams 601 Subpicture Video Stream 602 Subpicture Video Stream 603 Subpicture Video Stream 700 bitstream 710 SPS 711 PPS 715 slice header 720 image data 721 Pictures 723 Subpicture 725 slices 727 Picture Flags 730 non-VCL NAL units 731 SPS_NUT 732 PPS_NUT 740 VCL NAL Unit 741 IDR_N_LP NAL unit 742 IDR_w_RADL NAL unit 743 CRA_NUT 745 IRAP NAL unit 746 RASL_NUT 747 RADL_NUT 748 TRAIL_NUT 749 Non-IRAP NAL Units 800 Video Coding Device 810 Transceiver Unit (Tx / Rx) 814 Coding Module 820 downstream ports 830 processor 832 memory 850 upstream ports 860 I / O devices 1100 System 1101 Decision Module 1102 Video Encoder 1103 Encoding Module 1105 Storage Module 1107 Transmitter 1110 Video Decoder 1111 receiver 1113 Decision Module 1115 Decryption Module 1117 Transfer Module

Claims

1. 1. A method implemented in a decoder, comprising: receiving a bitstream including a sequence parameter set (SPS), one or more picture parameter sets (PPS), and coded data for a plurality of pictures, the coded data for the plurality of pictures being included in a plurality of video coding layer (VCL) network abstraction layer (NAL) units, the SPS being included in a non-VCL NAL unit having an SPS NAL unit type (SPS_NUT), and the PPS being included in a non-VCL NAL unit having a PPS NAL unit type (PPS_NUT); decoding one or more of the pictures based on the flag; Equipped with the PPS includes the flag, where the flag equal to 1 specifies that each picture referencing the PPS has multiple VCL NAL units and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type), and the flag equal to 0 specifies that each picture referencing the PPS has multiple VCL NAL units and the VCL NAL units of each picture referencing the PPS have the same value of nal_unit_type. method.

2. The method of claim 1 , wherein the flag is mixed_naru_types_in_pic_flag.

3. 3. The method of claim 1, wherein when the flag is equal to 1, one or more of the VCL NAL units of a picture have a first specific NAL unit type value, and all other VCL NAL units of the picture have a second specific NAL unit type value.

4. 4. The method of claim 3, wherein the first specific NAL unit type value is equal to an instantaneous decoding refresh (IDR) with random access decodable leading pictures (IDR_W_RADL), an IDR without leading pictures (IDR_N_LP), or a clean random access (CRA) NAL unit type (CRA_NUT).

5. The method of claim 3 or 4, wherein the second specific NAL unit type value is equal to a trailing picture NAL unit type (TRAIL_NUT).

6. 1. A method implemented in an encoder, comprising: encoding a sequence parameter set (SPS) and one or more picture parameter sets (PPS) into a bitstream; encoding one or more pictures into a bitstream; Equipped with the coded data for the one or more pictures is included in a plurality of Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, the SPS is included in a non-VCL NAL unit having an SPS NAL unit type (SPS_NUT), and the PPS is included in a non-VCL NAL unit having a PPS NAL unit type (PPS_NUT); the PPS includes a flag, the flag equal to 1 specifying that each picture referencing the PPS has multiple VCL NAL units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); the flag equal to zero specifies that each picture that references the PPS has multiple VCL NAL units, and the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type. method.

7. The method of claim 6 , wherein the flag is mixed_naru_types_in_pic_flag.

8. 8. The method of claim 6, wherein when the flag is equal to 1, one or more of the VCL NAL units of a picture have a first specific NAL unit type value, and all other VCL NAL units of the picture have a second specific NAL unit type value.

9. 9. The method of claim 8, wherein the first specific NAL unit type value is equal to an instantaneous decoding refresh (IDR) with random access decodable leading pictures (IDR_W_RADL), an IDR without leading pictures (IDR_N_LP), or a clean random access (CRA) NAL unit type (CRA_NUT).

10. The method according to claim 8 or 9, wherein the second specific NAL unit type value is equal to a trailing picture NAL unit type (TRAIL_NUT).

11. a processor; a memory coupled to the processor; Equipped with A video coding device, wherein the processor is configured to perform the method of any one of claims 1 to 5.

12. a processor; a memory coupled to the processor; Equipped with A video coding device, wherein the processor is configured to perform the method of any one of claims 6 to 10.

13. 11. A non-transitory computer-readable medium comprising a computer program product for use by a video coding device, the computer program product comprising computer-executable instructions that, when executed by a processor, cause the video coding device to perform the method of any one of claims 1 to 5 or the method of any one of claims 6 to 10.

14. A decoder comprising: means for receiving a bitstream including a sequence parameter set (SPS), one or more picture parameter sets (PPS), and coded data of a plurality of pictures, the coded data of the plurality of pictures being included in a plurality of video coding layer (VCL) network abstraction layer (NAL) units, the SPS being included in a non-VCL NAL unit having an SPS NAL unit type (SPS_NUT), and the PPS being included in a non-VCL NAL unit having a PPS NAL unit type (PPS_NUT); means for decoding one or more of the pictures based on the flag; Equipped with the PPS includes the flag, and the flag equal to 1 specifies that each picture that references the PPS has multiple VCL NAL units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); the flag equal to zero specifies that each picture that references the PPS has multiple VCL NAL units, and the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type. decoder.

15. A decoder according to claim 14, further configured to perform the method according to any one of claims 2 to 5.

16. 1. An encoder comprising: means for encoding a sequence parameter set (SPS) and one or more picture parameter sets (PPS) into a bitstream; means for encoding one or more pictures into a bitstream; Equipped with the coded data for the one or more pictures is included in a plurality of Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, the SPS is included in a non-VCL NAL unit having an SPS NAL unit type (SPS_NUT), and the PPS is included in a non-VCL NAL unit having a PPS NAL unit type (PPS_NUT); the PPS includes a flag, the flag equal to 1 specifying that each picture referencing the PPS has multiple VCL NAL units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); the flag equal to zero specifies that each picture that references the PPS has multiple VCL NAL units, and the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type. Encoder.

17. 17. An encoder according to claim 16, further configured to perform a method according to any one of claims 7 to 10.

18. 1. A method for storing a bitstream, comprising: obtaining a bitstream; storing said bitstream in at least one memory; Equipped with The bitstream comprises: a sequence parameter set (SPS), one or more picture parameter sets (PPS), and coded data for a number of pictures; Equipped with the coded data of the pictures is included in a plurality of Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, the SPS is included in a non-VCL NAL unit having an SPS NAL unit type (SPS_NUT), and the PPS is included in a non-VCL NAL unit having a PPS NAL unit type (PPS_NUT); the PPS includes a flag, the flag equal to 1 specifying that each picture referencing the PPS has multiple VCL NAL units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); the flag equal to zero specifies that each picture that references the PPS has multiple VCL NAL units, and the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type. method.

19. The method of claim 18, wherein the flag is mixed_naru_types_in_pic_flag.

20. 20. The method of claim 18 or 19, wherein when the flag is equal to 1, one or more of the VCL NAL units of a picture have a first specific NAL unit type value, and all other VCL NAL units of the picture have a second specific NAL unit type value.

21. 21. The method of claim 20, wherein the first specific NAL unit type value is equal to an instantaneous decoding refresh (IDR) with random access decodable leading pictures (IDR_W_RADL), an IDR without leading pictures (IDR_N_LP), or a clean random access (CRA) NAL unit type (CRA_NUT).

22. 21. The method of claim 20, wherein the second specific NAL unit type value is equal to a trailing picture NAL unit type (TRAIL_NUT).

23. A device for storing a bitstream, comprising: Memory and A transceiver; Equipped with the transceiver is configured to receive or transmit one or more bitstreams; the memory is configured to store the one or more bitstreams; The bitstream comprises: a sequence parameter set (SPS), one or more picture parameter sets (PPS), and coded data for a number of pictures; Equipped with the coded data of the pictures is included in a plurality of Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, the SPS is included in a non-VCL NAL unit having an SPS NAL unit type (SPS_NUT), and the PPS is included in a non-VCL NAL unit having a PPS NAL unit type (PPS_NUT); the PPS includes a flag, the flag equal to 1 specifying that each picture referencing the PPS has multiple VCL NAL units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); the flag equal to zero specifies that each picture that references the PPS has multiple VCL NAL units, and the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type. device.

24. The device of claim 23 , wherein the flag is mixed_naru_types_in_pic_flag.

25. 25. The device of claim 23 or 24, wherein when the flag is equal to 1, one or more of the VCL NAL units of a picture have a first specific NAL unit type value and all other VCL NAL units of the picture have a second specific NAL unit type value.

26. 26. The device of claim 25, wherein the first particular NAL unit type value is equal to an instantaneous decoding refresh (IDR) with random access decodable leading pictures (IDR_W_RADL), an IDR without leading pictures (IDR_N_LP), or a clean random access (CRA) NAL unit type (CRA_NUT).

27. 26. The device of claim 25, wherein the second specific NAL unit type value is equal to a trailing picture NAL unit type (TRAIL_NUT).

Images