Mixed NAL Unit Picture Constraints in Video Coding

Abstract

To provide a video codec mechanism.SOLUTION: A decoding method includes receiving a bitstream of a current picture including a plurality of video coding layer (VCL) network abstraction layer (NAL) units that do not have the same NAL unit type. Active entries of reference picture lists are obtained for slices positioned in a sub-picture A (subpicA) in subsequent pictures following the current picture in decoding order. The active entries do not refer to any reference picture preceding the current picture in the decoding order when the subpicA at the current picture is associated with an intra-random access point (IRAP) NAL unit type. The subsequent pictures are decoded based on the reference picture list active entries and forwarded to display as part of a decoded video sequence.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 871,524, entitled "Constraints for Mixed NAL Unit Types within One Picture in Video Coding," filed July 8, 2019 by Ye-Kui Wang, and incorporated herein by reference.

[0002] This disclosure relates generally to video coding, and more particularly to coding sub-pictures of pictures in video coding. [Background technology]

[0003] The amount of video data required to depict even a relatively short video is substantial, which can be challenging when the data is streamed or otherwise communicated over communications networks with limited bandwidth capacity. Therefore, video data is typically compressed before being communicated over modern telecommunications networks. Video size can also be an issue when the video is stored on a storage device, as memory resources may be limited. Video compression devices often use software and / or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data required to represent a digital video picture. The compressed data is then received at the destination by a video decompression device, which decodes the video data. With limited network resources and ever-increasing demands for higher video quality, improved compression and decompression techniques that improve compression ratios with little sacrifice in picture quality are desirable. Summary of the Invention

[0004] In one embodiment, the present disclosure includes a method implemented in a decoder, the method including: receiving, by a receiver of the decoder, a bitstream including a current picture including multiple video coding layer (VCL) NAL units that do not have the same network abstraction layer (NAL) unit type; obtaining, by a processor of the decoder, an active entry of a reference picture list for a slice located in subpicA (subpicA) in a subsequent picture after the current picture in decoding order, where the active entry does not include a reference to any reference picture that precedes the current picture in decoding order when subpicA in the current picture is associated with an intra random access point (IRAP) NAL unit type; decoding, by the processor, the subsequent picture based on the reference picture list active entry; and forwarding, by the processor, the subsequent picture for display as part of the decoded video sequence.

[0005] A video coding system may encode video by using IRAP pictures and non-IRAP pictures. IRAP pictures are pictures coded according to intra-prediction that serve as random access points in a video sequence. IRAP pictures can be decoded without first decoding any other pictures. Therefore, a decoder can begin decoding a video sequence at any IRAP picture. In contrast, a decoder generally cannot begin decoding a video sequence at a non-IRAP picture. IRAP pictures may also refresh the DPB. This is because IRAP pictures serve as the starting point of a coded video sequence (CVS) and pictures in a CVS do not reference pictures in previous CVSs. In this way, IRAP pictures can also break / stop the inter-prediction chain and stop inter-prediction-related coding errors because such errors cannot propagate through IRAP pictures.

[0006] In some cases, video coding systems may be used to code virtual reality (VR) video. VR video may include a sphere of video content displayed as if the user were at the center of the sphere. Only a portion of the sphere, called the viewport, is displayed to the user. The remainder of the picture is discarded without being rendered. The entire picture is typically transmitted so that different viewports can be dynamically selected and displayed in response to the user's head movements. This approach can result in very large video file sizes. To improve coding efficiency, some systems divide the picture into subpictures. Video can be coded at two or more resolutions. Each resolution is coded into a different set of sub-bitstreams corresponding to subpictures. When a user streams VR video, the coding system can merge the sub-bitstreams into a bitstream for transmission based on the current viewport used by the user. Specifically, the current viewport is derived from the high-resolution sub-bitstream, and the viewports not being displayed are derived from the low-resolution bitstream. In this way, the highest quality video is displayed to the user, and lower quality video is discarded. If the user selects a new viewport, the lower-resolution video is presented to the user. The decoder can request that the new viewport receive higher-resolution video. The encoder can then modify the merging process accordingly. When the IRAP picture is reached, the decoder can begin decoding the higher-resolution video sequence in the new viewport. This approach significantly improves video compression without adversely affecting the user's viewing experience.

[0007] One concern with the above approach is that the length of time required to change resolution is based on the length of time it takes to reach an IRAP picture. This is because the decoder cannot start decoding a different video sequence at a non-IRAP picture, as described above. One approach to reducing this latency is to include more IRAP pictures. However, this results in an increase in file size. To balance functionality and coding efficiency, different viewports / subpictures may include IRAP pictures at different frequencies. For example, viewports with a higher likelihood of being viewed may have more IRAP pictures than other viewports. This approach leads to other problems. Specifically, pictures after an IRAP picture are constrained not to reference pictures preceding the IRAP picture. However, this constraint is enforced at the picture level. A picture containing a mixed NAL unit containing both IRAP and non-IRAP subpictures may not be considered an IRAP picture at the picture level. Therefore, such picture-level constraints may not be applied. This may lead to some pictures after the IRAP subpicture inappropriately referencing pictures preceding the IRAP picture. In this case, the IRAP subpicture does not function properly as an access point because the reference picture / subpicture may not be available, preventing subpictures after the IRAP subpicture from being decodable. Furthermore, an IRAP subpicture should not prevent non-IRAP subpictures from referencing in a way that would defeat the purpose of having mixed NAL units (e.g., inter-coded sequences of different lengths depending on the subpicture position).

[0008] This example includes a mechanism for mitigating coding errors when a picture contains both IRAP NAL units and non-IRAP NAL units. Specifically, a subpicture of a current picture may contain an IRAP NAL unit. When this occurs, slices in pictures after the current picture that are also included in the subpicture are restricted from referencing reference pictures preceding the current picture. This ensures that the IRAP NAL unit stops inter-prediction propagation at the subpicture level. Thus, a decoder can begin decoding with the IRAP subpicture. Slices associated with subpictures in later pictures can always be decoded because such slices do not reference any data preceding the (undecoded) IRAP subpicture. This restriction does not apply to non-IRAP NAL units. Therefore, inter-prediction is not broken for subpictures that contain non-IRAP data. In this way, the disclosed mechanism enables the implementation of additional functionality. For example, the disclosed mechanism supports dynamic resolution changes at the subpicture level when using subpicture bitstreams. Therefore, the disclosed mechanism allows lower resolution sub-picture bitstreams to be transmitted when streaming VR video without significantly impairing the user experience, thus increasing coding efficiency and therefore reducing network, memory, and / or processing resource usage at the encoder and decoder.

[0009] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the IRAP NAL unit type is a clean random access (CRA) NAL unit type.

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the IRAP NAL unit type is an instantaneous decoder refresh (IDR) NAL unit type.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the aspect further includes determining, by the processor, that all slices of the current picture located in subpicA are associated with the same NAL unit type.

[0012] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the aspect further includes determining, by the processor, based on the flag, that the first NAL unit type value for the VCL NAL unit of the current picture is different from the second NAL unit type value for the VCL NAL unit of the current picture.

[0013] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the bitstream includes a picture parameter set (PPS), and the flag is obtained from the PPS.

[0014] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the flag is mixed_nalu_types_in_pic_flag, and mixed_nalu_types_in_pic_flag is equal to 1 when it specifies that each picture referencing the PPS has multiple VCL NAL units and that the VCL NAL units do not have the same value of NAL unit type (nal_unit_type).

[0015] In one embodiment, the disclosure includes a method implemented in an encoder, the method including: determining, by a processor of the encoder, that a current picture includes multiple VCL NAL units that do not have the same NAL unit type; determining, by the processor, that subpicA in the current picture is associated with an IRAP NAL unit type; generating, by the processor, an active entry of a reference picture list for a slice located at subpicA in a subsequent picture after the current picture in decoding order, where the active entry does not include a reference to any reference picture that precedes the current picture in decoding order when subpicA in the current picture is associated with the IRAP NAL unit type; encoding, by the processor, the subsequent picture into a bitstream based on the reference picture list; and storing, by a memory coupled to the processor, the bitstream for communication to a decoder.

[0016] A video coding system may encode video by using IRAP pictures and non-IRAP pictures. IRAP pictures are pictures coded according to intra prediction that serve as random access points in a video sequence. IRAP pictures can be decoded without first decoding any other pictures. Therefore, a decoder can begin decoding a video sequence at any IRAP picture. In contrast, a decoder generally cannot begin decoding a video sequence at a non-IRAP picture. IRAP pictures may also refresh the DPB. This is because IRAP pictures serve as the starting point of a coded video sequence (CVS), and pictures in a CVS do not reference pictures in previous CVSs. In this way, IRAP pictures can also break the inter-prediction chain and stop inter-prediction-related coding errors because such errors cannot propagate through IRAP pictures.

[0017] In some cases, video coding systems may be used to code virtual reality (VR) video. VR video may include a sphere of video content displayed as if the user were at the center of the sphere. Only a portion of the sphere, called the viewport, is displayed to the user. The remainder of the picture is discarded without being rendered. The entire picture is typically transmitted so that different viewports can be dynamically selected and displayed in response to the user's head movements. This approach can result in very large video file sizes. To improve coding efficiency, some systems divide the picture into subpictures. Video can be coded at two or more resolutions. Each resolution is coded into a different set of sub-bitstreams corresponding to subpictures. When a user streams VR video, the coding system can merge the sub-bitstreams into a bitstream for transmission based on the current viewport used by the user. Specifically, the current viewport is derived from the high-resolution sub-bitstream, and the viewports not being displayed are derived from the low-resolution bitstream. In this way, the highest quality video is displayed to the user, and lower quality video is discarded. If the user selects a new viewport, the lower-resolution video is presented to the user. The decoder can request that the new viewport receive higher-resolution video. The encoder can then modify the merging process accordingly. When the IRAP picture is reached, the decoder can begin decoding the higher-resolution video sequence in the new viewport. This approach significantly improves video compression without adversely affecting the user's viewing experience.

[0018] One concern with the above approach is that the length of time required to change resolution is based on the length of time it takes to reach an IRAP picture. This is because the decoder cannot start decoding a different video sequence at a non-IRAP picture, as described above. One approach to reducing this latency is to include more IRAP pictures. However, this results in an increase in file size. To balance functionality and coding efficiency, different viewports / subpictures may include IRAP pictures at different frequencies. For example, a viewport with a higher likelihood of being viewed may have more IRAP pictures than other viewports. This approach leads to other problems. Specifically, pictures after an IRAP picture are constrained not to reference pictures preceding the IRAP picture. However, this constraint is enforced at the picture level. A picture containing a mixed NAL unit containing both IRAP and non-IRAP subpictures may not be considered an IRAP picture at the picture level. Therefore, such picture-level constraints may not be applied. This may lead to some pictures after the IRAP subpicture inappropriately referencing pictures preceding the IRAP picture. In this case, the IRAP subpicture does not function properly as an access point because the reference picture / subpicture may not be available, preventing subpictures after the IRAP subpicture from being decodable. Furthermore, an IRAP subpicture should not prevent non-IRAP subpictures from referencing in a way that would defeat the purpose of having mixed NAL units (e.g., inter-coded sequences of different lengths depending on the subpicture position).

[0019] This example includes a mechanism for mitigating coding errors when a picture contains both IRAP NAL units and non-IRAP NAL units. Specifically, a subpicture of a current picture may contain an IRAP NAL unit. When this occurs, slices in pictures after the current picture that are also included in the subpicture are restricted from referencing reference pictures preceding the current picture. This ensures that the IRAP NAL unit stops inter-prediction propagation at the subpicture level. Thus, a decoder can begin decoding with the IRAP subpicture. Slices associated with subpictures in later pictures can always be decoded because such slices do not reference any data preceding the (undecoded) IRAP subpicture. This restriction does not apply to non-IRAP NAL units. Therefore, inter-prediction is not broken for subpictures that contain non-IRAP data. In this way, the disclosed mechanism enables the implementation of additional functionality. For example, the disclosed mechanism supports dynamic resolution changes at the subpicture level when using subpicture bitstreams. Therefore, the disclosed mechanism allows lower resolution sub-picture bitstreams to be transmitted when streaming VR video without significantly impairing the user experience, thus increasing coding efficiency and therefore reducing network, memory, and / or processing resource usage at the encoder and decoder.

[0020] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the IRAP NAL unit type is a CRA NAL unit type.

[0021] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the IRAP NAL unit type is an IDR NAL unit type.

[0022] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the implementation further includes encoding, by the processor, the current picture into the bitstream by ensuring that all slices of the current picture located in subpicA are associated with the same NAL unit type.

[0023] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the aspect further includes encoding, by the processor, a flag into the bitstream indicating that the first NAL unit type value for the VCL NAL unit of the current picture is different from the second NAL unit type value for the VCL NAL unit of the current picture.

[0024] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the flag is encoded in the bitstream into the PPS.

[0025] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the flag is mixed_nalu_types_in_pic_flag, which is set equal to 1 when each picture that references the PPS has multiple VCL NAL units and the VCL NAL units do not have the same value of nal_unit_type.

[0026] In one embodiment, the present disclosure comprises a video coding device including a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are configured to perform the method of any of the preceding aspects.

[0027] In one embodiment, the present disclosure provides a non-transitory computer-readable medium including a computer program product for use by a video coding device, the computer program product including computer-executable instructions stored on the non-transitory computer-readable medium such that, when executed by a processor, the computer program product causes the video coding device to perform a method of any of the preceding aspects.

[0028] In one embodiment, the disclosure includes a decoder including: receiving means for receiving a bitstream including a current picture including multiple VCL NAL units that do not have the same NAL unit type; obtaining means for obtaining an active entry of a reference picture list for a slice located in subpicA in a subsequent picture after the current picture in decoding order; determining means for determining that the active entry does not include a reference to any reference picture that precedes the current picture in decoding order when subpicA in the current picture is associated with an IRAP NAL unit type; decoding means for decoding the subsequent picture based on the reference picture list active entry; and forwarding means for forwarding the subsequent picture for display as part of a decoded video sequence.

[0029] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the decoder is further configured to perform the method of any of the preceding aspects.

[0030] In one embodiment, the present disclosure includes an encoder including: determining means for determining that a current picture includes multiple VCL NAL units that do not have the same NAL unit type; and determining that subpicA in the current picture is associated with an IRAP NAL unit type; generating means for generating an active entry of a reference picture list for a slice located in subpicA in a subsequent picture after the current picture in decoding order, where the active entry does not include a reference to any reference picture that precedes the current picture in decoding order when subpicA in the current picture is associated with the IRAP NAL unit type; encoding means for encoding the subsequent picture into a bitstream based on the reference picture list; and storing means for storing the bitstream for communication to a decoder.

[0031] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the encoder is further configured to perform the method of any of the preceding aspects.

[0032] For clarity, any one of the above-described embodiments may be combined with any one or more of the other above-described embodiments to create new embodiments within the scope of the present disclosure.

[0033] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. [Brief explanation of the drawings]

[0034] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in conjunction with the accompanying drawings and detailed description, in which like reference numerals represent like parts.

[0035] [Figure 1] 1 is a flowchart of an exemplary method for coding a video signal.

[0036] [Figure 2] 1 is a schematic diagram of an exemplary coding and decoding (codec) system for video coding.

[0037] [Figure 3] 1 is a schematic diagram illustrating an example video encoder;

[0038] [Figure 4] 1 is a schematic diagram illustrating an exemplary video decoder.

[0039] [Figure 5] FIG. 1 is a schematic diagram illustrating multiple sub-picture video streams split from a virtual reality (VR) picture video stream.

[0040] [Figure 6] FIG. 10 is a schematic diagram illustrating the constraints used when the current picture contains mixed Network Abstraction Layer (NAL) unit types.

[0041] [Figure 7] 1 is a schematic diagram illustrating an exemplary reference picture list structure including a reference picture list.

[0042] [Figure 8] FIG. 1 is a schematic diagram illustrating an exemplary bitstream containing pictures with mixed NAL unit types.

[0043] [Figure 9] 1 is a schematic diagram of an exemplary video coding device.

[0044] [Figure 10] 1 is a flowchart of an exemplary method for encoding a video sequence including pictures with mixed NAL unit types into a bitstream.

[0045] [Figure 11] 10 is a flowchart of an exemplary method for decoding a video sequence including pictures with mixed NAL unit types from a bitstream.

[0046] [Figure 12] 1 is a schematic diagram of an example system for coding a video sequence including pictures with mixed NAL unit types into a bitstream. DETAILED DESCRIPTION OF THE INVENTION

[0047] Initially, while exemplary implementations of one or more embodiments are provided below, it should be understood that the disclosed systems and / or methods may be implemented using any number of technologies, whether currently known or in existence. The present disclosure is in no way limited to the exemplary implementations, drawings, and technologies shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the appended claims, along with their full range of equivalents.

[0048] The following terms are defined as follows, unless used herein in a contrary context. Specifically, the following definitions are intended to provide further clarity to the present disclosure. However, in different contexts, terms may be explained differently. Therefore, the following definitions should be considered supplemental and not limiting of any other definitions provided for such terms herein.

[0049] A bitstream is a sequence of bits containing video data to be compressed for transmission between an encoder and a decoder. An encoder is a device configured to use an encoding process to compress video data into a bitstream. A decoder is a device configured to use a decoding process to reconstruct video data from the bitstream for display. A picture is an array of luma samples and / or chroma samples that generate a frame or its fields. The picture being coded or decoded can be referred to as the current picture for clarity of discussion, and any picture following the current picture can be referred to as a subsequent picture. A subpicture is a rectangular region of one or more slices in a sequence of pictures. Note that a square is a type of rectangle, and therefore, a subpicture can contain a square region. A slice is an integer number of complete tiles contained exclusively in a single Network Abstraction Layer (NAL) unit or an integer number of consecutive complete coding tree unit (CTU) rows within a tile of a picture. A NAL unit is a syntactic structure containing bytes of data and an indication of the type of data contained therein. NAL units include video coding layer (VCL) NAL units, which contain video data, and non-VCL NAL units, which contain supporting syntax data. A NAL unit type is the type of data structure contained in a NAL unit. An intra random access point (IRAP) NAL unit type is a data structure containing data from an IRAP picture or subpicture. An IRAP picture / subpicture is a picture / subpicture coded according to intra prediction, which indicates that a decoder can begin decoding a video sequence at the corresponding picture / subpicture without referencing pictures preceding the IRAP picture / subpicture. A clean random access (CRA) NAL unit type is a data structure containing data from a CRA picture or subpicture.A CRA picture / subpicture is an IRAP picture / subpicture that does not refresh the decoded picture buffer (DPB). The Instantaneous Decode Refresh (IDR) NAL unit type is a data structure containing data from an IDR picture or subpicture. An IDR picture / subpicture is an IRAP picture / subpicture that refreshes the DPB. A reference picture is a picture containing reference samples that can be used when coding other pictures by reference according to intra prediction. A reference picture list is a list of reference pictures used for intra prediction and / or inter-layer prediction. Some video coding systems refer to two picture lists, which can be denoted as Reference Picture List 1 and Reference Picture List 0. A reference picture list structure is an addressable syntax structure that contains multiple reference picture lists. An active entry is an entry in a reference picture list that references a reference picture that can be used by the current picture when performing inter prediction. A flag is a data structure containing a sequence of bits that can be set to indicate the corresponding data. A picture parameter set (PPS) is a parameter set containing picture-level data pertaining to one or more pictures. Decoding order is the order in which syntax elements are processed by the decoding process. A decoded video sequence is a sequence of pictures reconstructed by a decoder in preparation for presentation to the user.

[0050] Hereafter, the acronyms Coded Video Sequence (CVS), Decoded Picture Buffer (DPB), Instantaneous Decoding Refresh (IDR), Intra Random Access Point (IRAP), Least Significant Bit (LSB), Most Significant Bit (MSB), Network Abstraction Layer (NAL), Picture Order Count (POC), Raw Byte Sequence Payload (RBSP), Sequence Parameter Set (SPS), and Working Draft (WD) are used in this specification.

[0051] Many video compression techniques may be used to reduce the size of video files with minimal data loss. For example, video compression techniques may include performing spatial (e.g., intra-picture) prediction and / or temporal (e.g., inter-picture) prediction to reduce or remove data redundancy in a video sequence. In block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may be referred to as tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are coded using spatial prediction with respect to reference samples in neighboring blocks within the same picture. Video blocks in an inter-coded unidirectionally predicted (P) or bidirectionally predicted (B) slice of a picture may be coded using spatial prediction with respect to reference samples in neighboring blocks within the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may be referred to as a frame and / or an image, and a reference picture may be referred to as a reference frame and / or a reference image. Spatial or temporal prediction results in a prediction block that represents an image block. Residual data represents pixel differences between the original image block and the prediction block. Thus, inter-coded blocks are coded according to motion vectors that point to blocks of reference samples that form the prediction block, and the residual data indicates the differences between the coded block and the prediction block. Intra-coded blocks are coded according to an intra-coding mode and the residual data. For further compression, the residual data may be transformed from the pixel domain to a transform domain. These result in residual transform coefficients, which may be quantized. The quantized transform coefficients may be initially arranged in a two-dimensional array. The quantized transform coefficients may then be scanned to generate a one-dimensional vector of transform coefficients.Entropy coding may be applied to achieve even greater compression. Such video compression techniques are discussed in more detail below.

[0052] To ensure that the coded video can be decoded accurately, the video is coded and decoded according to corresponding video coding standards, including International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, Advanced Video Coding (AVC), also known as ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC) and Multiview Video Coding plus Depth (MVC+D), and three-dimensional (3D) AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The ITU-T and ISO / IEC Joint Video Experts Team (JVET) has begun development of a video coding standard called Versatile Video Coding. VVC is included in working drafts (WDs) including JVET-N1001-v10.

[0053] A video coding system may encode video by using IRAP pictures and non-IRAP pictures. An IRAP picture is a picture coded according to intra prediction that serves as a random access point in a video sequence. In intra prediction, blocks of a picture are coded by referencing other blocks within the same picture. This contrasts with non-IRAP pictures, which use inter prediction. In inter prediction, blocks of a current picture are coded by referencing other blocks in a reference picture different from the current picture. Because IRAP pictures are coded without reference to other pictures, they can be decoded without first decoding any other pictures. Therefore, a decoder can begin decoding a video sequence at any IRAP picture. In contrast, because non-IRAP pictures are coded with reference to other pictures, a decoder generally cannot begin decoding a video sequence at a non-IRAP picture. IRAP pictures may also refresh the DPB. This is because an IRAP picture acts as the starting point of a CVS, and pictures in a CVS do not reference pictures in previous CVSs. In this way, IRAP pictures can also stop coding errors in inter-prediction relationships because such errors cannot propagate through IRAP pictures. However, IRAP pictures are significantly larger than non-IRAP pictures in terms of data size. Thus, video sequences typically contain a small number of interspersed IRAP pictures and many non-IRAP pictures to balance coding efficiency and functionality. For example, a 60-frame CVS might contain one IRAP picture and 59 non-IRAP pictures.

[0054] In some cases, video coding systems may be used to code virtual reality (VR) video, sometimes referred to as 360-degree video. VR video may include a sphere of video content displayed as if the user were at the center of the sphere. Only a portion of the sphere, called a viewport, is visible to the user. For example, a user may use a head-mounted display (HMD) that selects and displays a viewport of the sphere based on the user's head movement. This provides the impression of being physically present in the virtual space depicted by the video. To achieve this result, each picture in a video sequence contains video data for the entire sphere at the corresponding instant in time. However, only a small portion of the picture (e.g., a single viewport) is displayed to the user; the remainder of the picture is discarded without being rendered. The entire picture is typically transmitted so that different viewports can be dynamically selected and displayed in response to the user's head movement. This approach can result in very large video file sizes.

[0055] To improve coding efficiency, some systems divide pictures into subpictures. A subpicture is a defined spatial region of a picture. Each subpicture contains a corresponding viewport of the picture. Video can be coded at two or more resolutions, with each resolution coded into a different sub-bitstream. When a user streams VR video, the coding system can merge the sub-bitstreams into a bitstream for transmission based on the current viewport used by the user. Specifically, the current viewport is derived from the high-resolution sub-bitstream, and the viewports not being displayed are derived from the low-resolution sub-bitstream. In this way, the highest quality video is displayed to the user, and the lower quality video is discarded. If the user selects a new viewport, the lower resolution video is presented to the user. The decoder can request that the new viewport receive higher resolution video. The encoder can then modify the merging process accordingly. When the IRAP picture is reached, the decoder can begin decoding the higher resolution video sequence in the new viewport. This approach significantly improves video compression without adversely affecting the user's viewing experience.

[0056] One concern with the above approach is that the length of time required to change resolution is based on the length of time it takes to reach an IRAP picture. This is because the decoder cannot begin decoding a different video sequence at a non-IRAP picture, as described above. One approach to reducing this latency is to include more IRAP pictures. However, this results in an increase in file size. To balance functionality and coding efficiency, different viewports / subpictures may include IRAP pictures at different frequencies. For example, viewports that are more likely to be viewed may have more IRAP pictures than other viewports. For example, in a basketball context, viewports related to the basket and / or center court may include IRAP pictures more frequently than viewports that view the stands or ceiling, because these are less likely to be viewed by a user.

[0057] This approach leads to other problems. Specifically, pictures after an IRAP picture are constrained not to reference pictures preceding the IRAP picture. However, this constraint is enforced at the picture level. A picture containing a mixed NAL unit, which includes both IRAP and non-IRAP subpictures, may not be considered an IRAP picture at the picture level. Therefore, such picture-level constraints may not apply. This could lead to parts of pictures following the IRAP subpicture improperly referencing pictures preceding the IRAP picture. In this case, the IRAP subpicture would not function properly as an access point because the reference picture / subpicture might not be available, preventing subpictures following the IRAP subpicture from being decodable. Furthermore, the IRAP subpicture should not prevent non-IRAP subpictures from referencing each other in a way that would defeat the purpose of having mixed NAL units (e.g., inter-coded sequences of different lengths depend on the subpicture position).

[0058] This specification discloses a mechanism for mitigating coding errors when a picture contains both IRAP NAL units and non-IRAP NAL units. Specifically, a subpicture of a current picture may contain an IRAP NAL unit. When this occurs, slices in pictures after the current picture that are also included in the subpicture are restricted from referencing reference pictures preceding the current picture. This ensures that the IRAP NAL unit breaks inter prediction at the subpicture level (e.g., stops the inter prediction reference chain). Thus, a decoder can begin decoding with the IRAP subpicture. Slices associated with subpictures in later pictures can always be decoded because such slices do not reference any data preceding the (undecoded) IRAP subpicture. This restriction does not apply to non-IRAP NAL units. Therefore, inter prediction is not broken for subpictures that contain non-IRAP data. In this way, the disclosed mechanism enables the implementation of additional functionality. For example, the disclosed mechanism supports dynamic resolution changes at the subpicture level when using subpicture bitstreams. Therefore, the disclosed mechanism allows lower resolution sub-picture bitstreams to be transmitted when streaming VR video without significantly impairing the user experience, thus increasing coding efficiency and therefore reducing network, memory, and / or processing resource usage at the encoder and decoder.

[0059] FIG. 1 is a flowchart of an exemplary method 100 for coding a video signal. Specifically, a video signal is encoded by an encoder. The encoding process compresses the video signal by using various mechanisms to reduce the video file size. The smaller file size allows the compressed video file to be transmitted to a user while reducing the associated bandwidth overhead. A decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally mirrors the encoding process, allowing the decoder to consistently reconstruct the video signal.

[0060] In step 101, a video signal is input to an encoder. For example, the video signal may be an uncompressed video file stored in memory. As another example, the video file may be captured by a video capture device, such as a video camera, and encoded to support live streaming of the video. The video file may include both an audio component and a video component. The video component includes a series of image frames that, when viewed in sequence, create the visual impression of motion. A frame includes pixels that are represented in terms of light, referred to herein as luma components (or luma samples), and color, referred to herein as chroma components (or color samples). In some examples, the frame may also include depth values ​​to support three-dimensional viewing.

[0061] In step 103, the video is partitioned into blocks. Partitioning involves subdividing pixels within each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame may first be divided into coding tree units (CTUs), which are blocks of a predetermined size (e.g., 64 pixels by 64 pixels). CTUs contain both luma samples and chroma samples. A coding tree may be used to divide the CTUs into blocks, and then the blocks may be recursively subdivided until a configuration that supports further encoding is achieved. For example, the luma component of a frame may be subdivided until each block contains relatively uniform light values. Furthermore, the chroma component of a frame may be subdivided until each block contains relatively uniform color values. Thus, the partitioning mechanism varies depending on the content of the video frame.

[0062] In step 105, various compression mechanisms are employed to compress the image blocks partitioned in step 103. For example, inter-prediction and / or intra-prediction may be used. Inter-prediction is designed to take advantage of the fact that objects in a common scene tend to appear in consecutive frames. Thus, a block representing an object in a reference frame need not be repeatedly described in adjacent frames. Specifically, an object such as a table may remain in a constant position across multiple frames. Therefore, the table may be described once, and adjacent frames may reference back to the reference frame. A pattern matching mechanism may be used to match objects across multiple frames. Furthermore, moving objects may be represented across multiple frames, for example, due to object motion or camera motion. As a specific example, a video may show a car moving across the screen over multiple frames. Such motion can be described using motion vectors. A motion vector is a two-dimensional vector that provides an offset from the object's coordinates in a frame to the object's coordinates in a reference frame. In this manner, inter-prediction can encode an image block in a current frame as a set of motion vectors that indicate its offset from a corresponding block in a reference frame.

[0063] Intra prediction encodes blocks within a common frame. It takes advantage of the fact that luma and chroma components tend to cluster together in a frame. For example, a green patch in a tree section tends to be located adjacent to similar green patches. Intra prediction uses multi-directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. Directional mode indicates that the current block is similar to the samples of neighboring blocks in the corresponding direction. Planar mode indicates that a series of blocks (e.g., a plane) along a row / column can be interpolated based on neighboring blocks at the end of the row. Planar mode effectively indicates a smooth transition of light / color across a row / column by using a relatively constant slope in changing values. DC mode is used for boundary smoothing, indicating that the block is similar to the average value associated with the samples of all neighboring blocks associated with the angular direction of the directional prediction mode. Therefore, intra-predicted blocks can be represented as various related prediction mode values ​​instead of their actual values. Additionally, inter-predicted blocks can represent image blocks as motion vector values ​​instead of actual values. In either case, the predicted block may not accurately represent the image block in some cases. Any differences are stored in the residual block. Transforms may be applied to the residual block to further compress the file.

[0064] Various filtering techniques may be applied in step 107. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction discussed above can result in the generation of blocky images at the decoder. Furthermore, block-based prediction schemes may encode blocks and then reconstruct the encoded blocks for later use as reference blocks. In-loop filtering schemes sequentially apply noise suppression filters, deblocking filters, adaptive loop filters, and sample adaptive offset (SAO) filters to blocks / frames. These filters mitigate such blocking artifacts so that the encoded file can be accurately reconstructed. Furthermore, these filters mitigate artifacts in the reconstructed reference blocks, making the artifacts less likely to generate additional artifacts in subsequent blocks that are coded based on the reconstructed reference blocks.

[0065] Once the video signal has been partitioned, compressed, and filtered, the resulting data is coded into a bitstream in step 109. The bitstream includes the data discussed above, as well as any signal data desired to support proper video signal reconstruction at the decoder. For example, such data may include partition data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream may be stored in memory for transmission to the decoder upon request. The bitstream may also be broadcast and / or multicast to multiple decoders. Generating the bitstream is an iterative process. Thus, steps 101, 103, 105, 107, and 109 may occur sequentially and / or simultaneously across many frames and blocks. The order depicted in FIG. 1 is presented for clarity and ease of discussion and is not intended to limit the video coding process to any particular order.

[0066] The decoder receives the bitstream and begins the decoding process in step 111. Specifically, the decoder uses an entropy decoding scheme to convert the bitstream into corresponding syntax data and video data. The decoder uses the syntax data from the bitstream to determine the frame partitions in step 111. The partitioning matches the results of the block partitioning in step 103. The entropy encoding / decoding used in step 111 is now described. The encoder makes many choices during the compression process, such as selecting a block partitioning scheme from several possible choices based on the spatial location of values ​​in the input image. Signaling the exact selection may use multiple bins. As used herein, a bin is a binary value (e.g., a bit value that can change depending on the context) that is treated as a variable. Entropy coding allows the encoder to retain a set of acceptable options and discard any options that are clearly infeasible for a particular case. Each acceptable option is assigned a codeword. The length of the code word is based on the number of allowable options (e.g., one bin for two options, two bins for three to four options, etc.). The encoder then encodes the code word for the selected option. This scheme reduces the size of the code word because it is desirable for the code word to be large enough to uniquely indicate a selection from a small subset of allowable options, as opposed to uniquely indicating a selection from a larger possible set of all possible options. The decoder then decodes the selection by determining the set of allowable options in a similar manner to the encoder. By determining the set of allowable options, the decoder can read the code word and determine the selection made by the encoder.

[0067] In step 113, the decoder performs block decoding. Specifically, the decoder uses an inverse transform to generate a residual block. The decoder then uses the residual block and the corresponding prediction block to reconstruct an image block according to the partitioning. The prediction block may include both intra-predicted and inter-predicted blocks, as generated by the encoder in step 105. The reconstructed image block is then positioned within a frame of the reconstructed video signal according to the partitioning data determined in step 111. The syntax for step 113 may also be signaled in the bitstream by entropy coding, as discussed above.

[0068] At step 115, filtering is performed on the frames of the reconstructed video signal in a manner similar to step 107 in the encoder. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and an SAO filter may be applied to the frames to remove blocking artifacts. Once the frames have been filtered, the video signal may be output to a display at step 117 for viewing by an end user.

[0069] FIG. 2 is a schematic diagram of an exemplary coding and decoding (codec) system 200 for video coding. Specifically, codec system 200 provides functionality to support the implementation of operational method 100. Codec system 200 is generalized to illustrate components used in both an encoder and a decoder. Codec system 200 receives and partitions a video signal, as discussed with respect to steps 101 and 103 in operational method 100, resulting in partitioned video signal 201. When acting as an encoder, codec system 200 then compresses partitioned video signal 201 into a coded bitstream, as discussed with respect to steps 105, 107, and 109 in method 100. When acting as a decoder, codec system 200 generates an output video signal from the bitstream, as discussed with respect to steps 111, 113, 115, and 117 in operational method 100. Codec system 200 includes a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, an intra-picture prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header formatting and context-adaptive binary arithmetic coding (CABAC) component 231. Such components are coupled as shown. In FIG. 2, black lines indicate the movement of data to be coded / decoded, and dashed lines indicate the movement of control data that controls the operation of other components. All of the components of codec system 200 may reside within an encoder. A decoder may include a subset of the components of codec system 200.For example, the decoder may include an intra-picture prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded picture buffer component 223. These components are described herein.

[0070] The partitioned video signal 201 is a captured video sequence that has been divided into blocks of pixels by a coding tree. The coding tree uses various partitioning modes to subdivide the blocks of pixels into smaller blocks of pixels. These blocks can then be further subdivided into smaller blocks. Blocks are sometimes referred to as nodes on the coding tree. Larger parent nodes are divided into smaller child nodes. The number of times a node is subdivided is called the depth of the node / coding tree. In some cases, the partitioned blocks may be included in a coding unit (CU). For example, a CU may be a subpart of a CTU, which includes a luma block, a red-difference chroma (Cr) block, and a blue-difference chroma (Cb) block, along with the corresponding syntax instructions for the CU. Partitioning modes may include binary tree (BT), triple tree (TT), and quad tree (QT), which are used to partition a node into two, three, or four child nodes of varying shapes, depending on the partitioning mode used. The partitioned video signal 201 is forwarded to a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, a filter control analysis component 227, and a motion estimation component 221 for compression.

[0071] The generic coder control component 211 is configured to make decisions related to coding images of a video sequence into a bitstream according to application constraints. For example, the generic coder control component 211 manages the optimization of bitrate / bitstream size versus reconstruction quality. Such decisions may be made based on storage space / bandwidth availability and image resolution requirements. The generic coder control component 211 also manages buffer utilization in relation to transmission rate to mitigate buffer underrun and overrun issues. To manage these issues, the generic coder control component 211 manages partitioning, prediction, and filtering by other components. For example, the generic coder control component 211 can dynamically increase compression complexity to increase resolution and bandwidth usage, or decrease compression complexity to decrease resolution and bandwidth usage. Therefore, the generic coder control component 211 controls other components of the codec system 200 to balance bitrate concerns with video signal reconstruction quality. The generic coder control component 211 generates control data that controls the operation of other components. The control data is also forwarded to the Header Formatting and CABAC component 231 to be encoded in the bitstream into signal parameters for decoding by the decoder.

[0072] The partitioned video signal 201 is also sent to a motion estimation component 221 and a motion compensation component 219 for inter-prediction. A frame or slice of the partitioned video signal 201 may be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-predictive coding of the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 200 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0073] The motion estimation component 221 and the motion compensation component 219 may be highly integrated but are shown separately for conceptual purposes. Motion estimation, performed by the motion estimation component 221, is the process of generating motion vectors, which estimate the motion of video blocks. A motion vector may indicate, for example, the displacement of a coded object relative to a predictive block. A predictive block is a block that is found to closely match a coded block in terms of pixel differences. A predictive block is sometimes referred to as a reference block. Such pixel differences may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. HEVC uses several coded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be divided into CTBs, which are then divided into CBs for inclusion in CUs. A CU can be coded as a prediction unit (PU), which contains prediction data, and / or a transform unit (TU), which contains the transform residual data of the CU. The motion estimation component 221 generates the motion vectors, PUs, and TUs by using rate-distortion analysis as part of a rate-distortion optimization process. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for a current block / frame and select the reference block, motion vector, etc. with the best rate-distortion characteristics. The best rate-distortion characteristics balance both the quality of the video reconstruction (e.g., the amount of data lost due to compression) and the coding efficiency (e.g., the size of the final encoding).

[0074] In some examples, the codec system 200 may calculate values ​​for sub-integer pixel positions of reference pictures stored in the decoded picture buffer component 223. For example, the video codec system 200 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference picture. Accordingly, the motion estimation component 221 may perform motion search for whole-pixel and fractional pixel positions and output fractional-pixel precision motion vectors. The motion estimation component 221 calculates motion vectors for PUs of video blocks in inter-coded slices by comparing the positions of the PUs with the positions of the predictive blocks in the reference pictures. The motion estimation component 221 outputs the calculated motion vectors as motion data to the header formatting and CABAC component 231 for encoding and the motion compensation component 219.

[0075] Motion compensation performed by motion compensation component 219 may involve fetching or generating a predictive block based on a motion vector determined by motion estimation component 221. Again, in some examples, motion estimation component 221 and motion compensation component 219 may be functionally integrated. Upon receiving a motion vector for the PU of the current video block, motion compensation component 219 may locate the predictive block to which the motion vector points. A residual video block is then formed by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded to form pixel difference values. Generally, motion estimation component 221 performs motion estimation on the luma component, and motion compensation component 219 uses the motion vector calculated based on the luma component for both the chroma and luma components. The predictive block and residual block are forwarded to scaling and quantization component 213 for transformation.

[0076] The partitioned video signal 201 is also sent to an intra-picture estimation component 215 and an intra-picture prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-picture estimation component 215 and the intra-picture prediction component 217 may be highly integrated but are shown separately for conceptual purposes. The intra-picture estimation component 215 and the intra-picture prediction component 217 intra-predict the current block relative to blocks within the current frame as an alternative to the inter-prediction performed by the inter-frame motion estimation component 221 and the motion compensation component 219, as described above. In particular, the intra-picture estimation component 215 determines the intra-prediction mode to use to encode the current block. In some examples, the intra-picture estimation component 215 selects an appropriate intra-prediction mode to encode the current block from multiple tested intra-picture prediction modes. The selected intra-prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0077] For example, the intra picture estimation component 215 may use a rate-distortion analysis to calculate rate-distortion values ​​for various tested intra picture prediction modes and select the intra prediction mode with the best rate-distortion characteristics from the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block coded to generate the coded block, as well as the bitrate (e.g., number of bits) used to generate the coded block. The intra picture estimation component 215 may calculate a ratio from the distortion and rate for various coded blocks to determine which intra prediction mode exhibits the best rate-distortion value for the block. Additionally, the intra picture estimation component 215 may be configured to code depth blocks of a depth map using a rate-distortion optimization (RDO)-based depth modeling mode (DMM).

[0078] The intra-picture prediction component 217 may generate a residual block from the prediction block based on the selected intra-picture prediction mode determined by the intra-picture estimation component 215 when implemented in an encoder, or may read the residual block from the bitstream when implemented in a decoder. The residual block contains the value differences between the prediction block and the original block, represented as a matrix. The residual block is then forwarded to the transform scaling and quantization component 213. The intra-picture estimation component 215 and the intra-picture prediction component 217 may operate on both the luma component and the chroma component.

[0079] The transform scaling and quantization component 213 is configured to further compress the residual block. The transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform, to the residual block to generate a video block containing residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. The transform may convert the residual information from the pixel value domain to a transform domain, such as the frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling involves applying a scale factor to the residual information so that different frequency information is quantized with different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also configured to quantize the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the transform scaling and quantization component 213 may then perform a scan of the matrix containing the quantized transform coefficients, which are forwarded to the header formatting and CABAC component 231 to be encoded in the bitstream.

[0080] The scaling and inverse transform component 229 applies the inverse operations of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, transform, and / or quantization to reconstruct a residual block in the pixel domain, for example, for use as a reference block that may later become a predictive block for another current block. The motion estimation component 221 and / or motion compensation component 219 may calculate a reference block by adding the residual block to the corresponding predictive block for use in motion estimation of a later block / frame. A filter is applied to the reconstructed reference block to mitigate artifacts generated during scaling, quantization, and transform. Otherwise, such artifacts may cause inaccurate predictions (and create additional artifacts) when subsequent blocks are predicted.

[0081] The filter control analysis component 227 and the in-loop filter component 225 apply filters to residual blocks and / or reconstructed image blocks. For example, a transformed residual block from the scaling and inverse transform component 229 may be combined with a corresponding prediction block from the intra-picture prediction component 217 and / or the motion compensation component 219 to reconstruct an original image block. The filter may then be applied to the reconstructed image block. In some examples, the filter may instead be applied to the residual block. Like the other components in FIG. 2, the filter control analysis component 227 and the in-loop filter component 225 are highly integrated and may be implemented together, but are shown separately for conceptual purposes. The filters applied to reconstructed reference blocks are applied to specific spatial regions and include multiple parameters to adjust how such filters are applied. The filter control analysis component 227 analyzes the reconstructed reference blocks to determine where such filters should be applied and sets the corresponding parameters. Such data is forwarded to the header formatting and CABAC component 231 as filter control data for encoding. The in-loop filter component 225 applies such filters based on the filter control data. The filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Such filters may be applied in the spatial / pixel domain (e.g., reconstructed pixel blocks) or the frequency domain, depending on the example.

[0082] When operating as an encoder, the filtered reconstructed picture blocks, residual blocks, and / or prediction blocks are stored in the decoded picture buffer component 223 for later use in motion estimation as discussed above. When operating as a decoder, the decoded picture buffer component 223 stores the reconstructed and filtered blocks and forwards them towards the display as part of the output video signal. The decoded picture buffer component 223 may be any memory device capable of storing prediction blocks, residual blocks, and / or reconstructed image blocks.

[0083] The Header Formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes such data into a coded bitstream for transmission to the decoder. Specifically, the Header Formatting and CABAC component 231 generates various headers for encoding control data, such as general control data and filter control data. Additionally, prediction data, including intra-prediction and motion data, as well as residual data in the form of quantized transform coefficient data, are all encoded in the bitstream. The final bitstream contains all information required by the decoder to reconstruct the original partitioned video signal 201. Such information may also include an intra-prediction mode index table (also known as a codeword mapping table), definitions of coding contexts for various blocks, indications of the most likely intra-prediction modes, indications of partition information, etc. Such data may be encoded using entropy coding. For example, the information may be encoded using context-adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. After entropy coding, the coded bitstream may be transmitted to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0084] 3 is a block diagram illustrating an exemplary video encoder 300. Video encoder 300 may be used to implement the encoding functionality of codec system 200 and / or to implement steps 101, 103, 105, 107, and / or 109 of operating method 100. Encoder 300 partitions an input video signal, resulting in a partitioned video signal 301 that is substantially similar to partitioned video signal 201. Partitioned video signal 301 is then compressed and encoded into a bitstream by components of encoder 300.

[0085] Specifically, the partitioned video signal 301 is forwarded to an intra-picture prediction component 317 for intra prediction. The intra-picture prediction component 317 may be substantially similar to the intra-picture estimation component 215 and the intra-picture prediction component 217. The partitioned video signal 301 is also forwarded to a motion compensation component 321 for inter prediction based on reference blocks in a decoded picture buffer component 323. The motion compensation component 321 may be substantially similar to the motion estimation component 221 and the motion compensation component 219. The prediction blocks and residual blocks from the intra-picture prediction component 317 and the motion compensation component 321 are forwarded to a transform and quantization component 313 for transforming and quantizing the residual blocks. The transform and quantization component 313 may be substantially similar to the transform scaling and quantization component 213. The transformed and quantized residual block and the corresponding prediction block (along with associated control data) are forwarded to an entropy coding component 331 for coding into a bitstream. The entropy coding component 331 may be substantially similar to the header formatting and CABAC component 231.

[0086] The transformed and quantized residual block and / or the corresponding prediction block are also forwarded from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into a reference block for use by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially similar to the scaling and inverse transform component 229. An in-loop filter in the in-loop filter component 325 is also applied to the residual block and / or the reconstructed reference block, depending on the example. The in-loop filter component 325 may be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may include multiple filters, as discussed with respect to the in-loop filter component 225. The filtered block is then stored in the decoded picture buffer component 323 for use as a reference block by the motion compensation component 321. The decoded picture buffer component 323 may be substantially similar to the decoded picture buffer component 223.

[0087] 4 is a block diagram illustrating an exemplary video decoder 400. Video decoder 400 may be used to implement the decoding functionality of codec system 200 and / or to implement steps 111, 113, 115, and / or 117 of operating method 100. Decoder 400 receives a bitstream, for example, from encoder 300, and generates a reconstructed output video signal based on the bitstream for display to an end user.

[0088] The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 can use header information to provide context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding the video signal, such as general control data, filter control data, partition information, motion data, prediction data, and quantized transform coefficients from the residual block. The quantized transform coefficients are forwarded to the inverse transform and quantization component 429 for reconstruction into the residual block. The inverse transform and quantization component 429 can be similar to the inverse transform and quantization component 329.

[0089] The reconstructed residual block and / or predictive block are forwarded to the intra-picture prediction component 417 for reconstructing into an image block based on an intra prediction operation. The intra-picture prediction component 417 may be substantially similar to the intra-picture estimation component 215 and the intra-picture prediction component 217. Specifically, the intra-picture prediction component 417 uses a prediction mode to locate a reference block within a frame and applies the residual block to the result to reconstruct an intra-predicted image block. The reconstructed intra-predicted image block and / or residual block and corresponding inter-predicted data are forwarded to the decoded picture buffer component 423 via the in-loop filter component 425, which may be substantially similar to the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block, and / or predictive block, and such information is stored in the decoded picture buffer component 423. The reconstructed picture blocks from the decoded picture buffer component 423 are forwarded to the motion compensation component 421 for inter-prediction. The motion compensation component 421 may be substantially similar to the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 generates a prediction block using a motion vector from a reference block and applies a residual block to the result to reconstruct an image block. The resulting reconstructed block may be forwarded to the decoded picture buffer component 423 via an in-loop filter component 425. The decoded picture buffer component 423 continues to store additional reconstructed image blocks, which may be reconstructed into frames via partition information. Such frames may be arranged in a sequence. The sequence is output to a display as a reconstructed output video signal.

[0090] 5 is a schematic diagram illustrating multiple sub-picture video streams 501, 502, and 503 split from a VR picture video stream 500. For example, the sub-picture video streams 501-503 and / or the VR picture video stream 500 may be encoded by an encoder, such as the codec system 200 and / or the encoder 300, in accordance with the method 100. Furthermore, the sub-picture video streams 501-503 and / or the VR picture video stream 500 may be decoded by a decoder, such as the codec system 200 and / or the decoder 400.

[0091] The VR picture video stream 500 contains multiple pictures presented over time. Specifically, VR works by coding a sphere of video content that can be viewed as if the user were at the center of the sphere. Each picture contains the entire sphere; however, only a portion of the picture, known as a viewport, is displayed to the user. For example, a user may use a head-mounted display (HMD) that selects and displays a viewport of the sphere based on the user's head movement. This provides the impression of being physically present in the virtual space depicted by the video. To achieve this result, each picture in the video sequence contains video data for the entire sphere at the corresponding instant in time. However, only a small portion of the picture (e.g., a single viewport) is displayed to the user; the remainder of the picture is discarded without being rendered. The entire picture is typically transmitted so that different viewports can be dynamically selected and displayed in response to the user's head movement.

[0092] In the illustrated example, the pictures in the VR picture video stream 500 can each be subdivided into subpictures based on the available viewports. Thus, each picture and corresponding subpicture includes a picture order count as part of its presentation. The subpicture video streams 501-503 are created when the subdivision is applied consistently over time. Such consistent subdivision creates the subpicture video streams 501-503, each of which includes a set of subpictures of a predetermined size, shape, and spatial location relative to the corresponding picture in the VR picture video stream 500. Furthermore, the subpictures in the subpicture video streams 501-503 vary in picture order count over presentation time. In this manner, the subpictures in the subpicture video streams 501-503 can be aligned based on their picture order relative to presentation time. The sub-pictures from the sub-picture video streams 501-503 at each picture order count value can then be merged in the spatial domain based on predefined spatial positions to reconstruct the VR picture video stream 500 for display. Specifically, the sub-picture video streams 501-503 can each be encoded into a separate sub-bitstream. When such sub-bitstreams are merged together, they result in a bitstream containing the entire set of pictures over presentation time. The resulting bitstream can be sent to a decoder for decoding and display based on the user's currently selected viewport.

[0093] One issue with VR video is that all of the subpicture video streams 501-503 can be transmitted to the user at high quality (e.g., high resolution). This allows the decoder to dynamically select the user's current viewport and display the subpictures from the corresponding subpicture video streams 501-503 in real time. However, the user may only view a single viewport, for example, from the subpicture video stream 501, while the subpicture video streams 502-503 are discarded. Thus, transmitting the subpicture video streams 502-503 at high quality can waste a significant amount of bandwidth. To improve coding efficiency, VR video may be encoded into multiple video streams 500, with each video stream 500 encoded at a different quality / resolution. In this way, the decoder can transmit a request for the current subpicture video stream 501. In response, the encoder (or intermediate slicer or other content server) can select higher-quality sub-picture video stream 501 from higher-quality video stream 500 and lower-quality sub-picture video streams 502-503 from lower-quality video stream 500. The encoder can then merge such sub-bitstreams together into a fully encoded bitstream for transmission to the decoder. In this way, the decoder receives a series of pictures in which the current viewport is at a higher quality and the other viewports are at lower, lower quality. Furthermore, the highest-quality sub-pictures are generally displayed to the user (without head movement), and the lower-quality sub-pictures are generally discarded, which balances functionality and coding efficiency.

[0094] In the case where a user switches their viewing orientation from sub-picture video stream 501 to sub-picture video stream 502, the decoder requires that the new sub-picture video stream 502 be transmitted at a higher quality. The encoder can then modify the merging mechanism accordingly. The decoder can begin decoding the new CVS only at the IRAP picture. This is because IRAP pictures are coded according to intra-prediction, which does not refer to other pictures. Therefore, an IRAP picture can be decoded even if the pictures preceding it are unavailable. Non-IRAP pictures are coded according to inter-prediction. Thus, a non-IRAP picture cannot be decoded without first decoding the corresponding set of reference pictures based on the reference picture list. Therefore, a decoder generally cannot begin decoding a video sequence at a non-IRAP picture. Due to these constraints, the sub-picture video stream 502 is displayed at a lower quality until the IRAP picture / sub-picture is reached. The IRAP picture is then decoded at a higher quality, allowing decoding of the higher quality version of the sub-picture video stream 502 to begin. This approach significantly improves video compression without adversely affecting the user's viewing experience.

[0095] One concern with the above approach is that the amount of time required to change resolution is based on the amount of time it takes to reach an IRAP picture in the video stream. This is because the decoder cannot begin decoding a different version of the subpicture video stream 502 at a non-IRAP picture. One approach to reducing this latency is to include more IRAP pictures, but this results in an increase in file size. To balance functionality and coding efficiency, different viewport / subpicture video streams 501-503 may include IRAP pictures at different frequencies. For example, viewport / subpicture video streams 501-503 that are more likely to be viewed may have more IRAP pictures than other viewport / subpicture video streams 501-503. For example, in a basketball context, viewport / subpicture video streams 501-503 relating to the basket and / or center court may contain IRAP pictures more frequently than viewport / subpicture video streams 501-503 relating to the stands or ceiling, as these are less likely to be visible to the user.

[0096] This approach leads to additional problems. Specifically, subpictures from subpicture video streams 501-503 that share a POC are part of a single picture. As noted above, slices from a picture are included in NAL units based on picture type. In some video coding systems, all NAL units related to a single picture are constrained to contain the same NAL unit type. When different subpicture video streams 501-503 have IRAP pictures with different frequencies, some of the pictures contain both IRAP and non-IRAP subpictures. This violates the constraint that each single picture should use only the same type of NAL unit.

[0097] This issue can be addressed by removing the constraint that all NAL units for slices in a picture use the same NAL unit type. For example, a picture is contained in an access unit. By removing this constraint, an access unit can contain both IRAP and non-IRAP NAL unit types. Furthermore, a flag can be coded to indicate when a picture / access unit contains a mix of IRAP and non-IRAP NAL unit types. In some examples, the flag is the mixed NAL unit types in picture flag (mixed_nalu_types_in_pic_flag). Additionally, a constraint may be applied to require that a single mixed picture / access unit can contain only one type of IRAP NAL unit and one type of non-IRAP NAL unit. This prevents unintended mixing of NAL unit types from occurring. If such mixing is allowed, the decoder must be designed to manage such mixing. This would unnecessarily increase the required hardware complexity without providing any additional benefit to the coding process. For example, a mixed picture may include IRAP NAL units of one type selected from IDR_W_RADL, IDR_N_LP, or CRA_NUT. Additionally, a mixed picture may include non-IRAP NAL units of one type selected from TRAIL_NUT, RADL_NUT, and RASL_NUT.

[0098] 6 is a schematic diagram illustrating a constraint 600 used when the current picture contains mixed NAL unit types. The constraint 600 may be applied when coding the VR picture video stream 500. In this manner, the constraint 600 may be used by an encoder, such as the codec system 200 and / or the encoder 300, that applies the method 100. Furthermore, the constraint 600 may be employed by a decoder, such as the codec system 200 and / or the decoder 400, that applies the method 100. The constraint 600 is a requirement imposed on the video data, on supporting parameters, and / or on processes involved in coding and / or decoding the video data.

[0099] Specifically, Figure 6 shows a sequence of pictures divided into subpictures. The subpictures are shown as subpicture A (subpicA) 601 and subpicture B (subpicB) 603. Subpictures, hereafter subpicA 601 and subpicB 603, are rectangular and / or square regions of one or more slices in a sequence of pictures. SubpicA 601 and subpicB 603 may be included in a subpicture video stream, such as any of subpicture video streams 501-503 of Figure 5. For clarity of explanation, only two subpictures are shown, but any number of subpictures may be used.

[0100] A picture is an array of luma samples and / or chroma samples that generate a frame or a field thereof. Pictures are coded and / or decoded in decoding order 650. Decoding order is the order in which syntax elements are processed by the decoding process. As the decoding order progresses, the decoding process progresses through pictures. For clarity, the picture currently being coded / decoded at a particular moment is referred to as the current picture 620. A picture that has already been coded / decoded is the preceding picture 610. A picture that has not yet been decoded is the following picture 630 and / or 640. As shown in Figure 6, pictures are consistently partitioned into subpicA 601 and subpicB 603. Therefore, each of the preceding picture 610, the current picture 620, and the following pictures 630 and 640 is partitioned into and / or contains subpicA 601 and subpicB 603.

[0101] As mentioned above, in some examples, IRAP subpictures may be applied with different frequencies. In the illustrated example, the current picture 620 includes multiple VCL NAL units that do not have the same NAL unit type. Specifically, subpicA in the current picture 620 includes an IRAP NAL unit 621, while subpicB 603 in the current picture 620 includes a non-IRAP NAL unit 623. The IRAP NAL unit 621 is a data structure that includes data from an IRAP picture or subpicture. An IRAP picture / subpicture is a picture / subpicture coded according to intra prediction, which indicates that a decoder can begin decoding a video sequence at the corresponding picture / subpicture without referencing pictures preceding the IRAP picture / subpicture. The IRAP NAL unit 621 may include CRA NAL units and / or IDR NAL units. A CRA picture / subpicture is an IRAP picture / subpicture that does not refresh the DPB, and an IDR picture / subpicture is an IRAP picture / subpicture that refreshes the DPB. A non-IRAP NAL unit 623 is any VCL NAL unit that does not contain an IRAP picture / subpicture. For example, a non-IRAP NAL unit 623 may include a leading picture, such as a random access skip leading (RASL) picture, a random access decodable leading (RADL) picture, or a trailing picture. A leading picture precedes an IRAP picture in presentation order and occurs after the IRAP picture in decoding order 650. A trailing picture occurs after an IRAP picture in both presentation order and decoding order. A non-IRAP NAL unit 623 may be coded according to inter prediction. Thus, the current picture 620 is coded according to both inter-picture prediction in subpicB 603 and intra-picture prediction in subpicA 601.

[0102] Previous picture 610 includes non-IRAP NAL unit 611 and non-IRAP NAL unit 613, subsequent picture 630 includes non-IRAP NAL unit 631 and non-IRAP NAL unit 633, and subsequent picture 640 includes non-IRAP NAL unit 641 and non-IRAP NAL unit 643. Non-IRAP NAL units 611, 613, 631, 633, 641, and 643 may be similar to non-IRAP NAL unit 623 (e.g., may be coded according to inter prediction) but may contain different video data.

[0103] The current picture 620 is a mixed NAL unit picture because it contains VCL NAL units that do not have the same NAL unit type. The presence of a mixed NAL unit picture can be signaled by a flag in a parameter set in the bitstream. The constraint 600 applies when a mixed NAL unit picture such as the current picture 620 is encountered.

[0104] During encoding, the encoder can determine that subpic A 601 in current picture 620 is associated with an IRAP NAL unit type and therefore contains IRAP NAL unit 621. This indicates that subpic A 601 in current picture 620 should act as a boundary, preventing inter-prediction in subpic A 601 from referencing across the current picture 620. Therefore, non-IRAP NAL unit 631 cannot reference non-IRAP NAL unit 611 or any other NAL unit preceding IRAP NAL unit 621. The encoder can encode current picture 620 by ensuring that all slices of the current picture mapped to subpic A 601 are associated with the same NAL unit type. For example, when subpic A contains at least one CRA slice (or IDR slice), all slices in subpic A 601 must also be CRA (or IDR). The slices of the current picture 620 (including IRAP NAL unit 621 and non-IRAP NAL unit 623) are then coded based on the NAL unit type (e.g., intra-prediction and inter-prediction, respectively). Subsequent pictures 630 and 640 that follow the current picture 620 in decoding order 650 are also coded. To ensure that IRAP NAL unit 621 prevents inter-prediction propagation, non-IRAP NAL units 631 and 641 are prevented from referencing (632) the preceding picture 610. Referencing 632 is controlled by a reference picture list, whose active entries indicate the reference pictures available for the picture currently being coded.In this way, constraint 600 ensures that active entries associated with slices located in subpicA in subsequent pictures 630 and 640 (e.g., non-IRAP NAL units 631 and 641) do not / do not reference 632 any reference pictures preceding current picture 620 in decoding order 650 when subpicA 601 in current picture 620 is associated with IRP NAL unit 621 having an IRP NAL unit type. Non-IRAP NAL unit 631 can still reference current picture 620 and / or subsequent picture 640. Furthermore, non-IRAP NAL unit 641 can still reference current picture 620 and / or subsequent picture 630. This stops inter prediction propagation for non-IRAP NAL units 631 and 641 because they follow IRP NAL unit 621 in subpicA. Subsequent pictures 630 and / or 640 can then be encoded based on the NAL unit type and according to constraint 600.

[0105] Such constraint 600 does not apply to non-IRAP NAL units 633 and 643, which are located in subpicB 603 and therefore do not follow the IRAP NAL units in subpicB 603. Accordingly, non-IRAP NAL units 633 and 643 may reference 634 the preceding picture 610. Accordingly, the active entries in the reference picture lists associated with non-IRAP NAL units 633 and 643 may reference the preceding picture 610, the current picture 620, and / or the subsequent picture 640 or 630, respectively. In this way, inter prediction is broken for subpicA by IRAP NAL unit 621, but IRAP NAL unit 621 does not stop inter prediction propagation for subpicB 603. Therefore, by employing constraint 600, the inter prediction chain can be stopped (or not) on a subpicture-by-subpicture basis. As noted above, only two subpictures are shown. However, constraint 600 can be applied to any number of sub-pictures to stop the propagation of inter cross-reference chains on a sub-picture-by-sub-picture basis.

[0106] 7 is a schematic diagram illustrating an exemplary reference picture list (RPL) structure 700 including a reference picture list. The RPL structure 700 can be used to store representations of reference pictures used in unidirectional inter-prediction and / or bidirectional inter-prediction. Therefore, the RPL structure 700 can be used by the codec system 200, the encoder 300, and / or the decoder 400 when performing the method 100. Furthermore, the RPL structure 700 can be used when coding the VR picture video stream 500, in which case the RPL structure 700 can be encoded according to the constraints 600.

[0107] The RPL structure 700 is an addressable syntax structure that includes multiple reference picture lists, such as RPL0 711 and RPL1 712. The RPL structure 700 may be coded and / or derived for use when coding a corresponding slice. The RPL structure 700 may be stored in the SPS and / or slice header of a bitstream, depending on the example. The reference picture lists, such as RPL0 711 and RPL1 712, are lists that indicate reference pictures used for inter-prediction. RPL0 711 and RPL1 712 may each include multiple entries 715. The RPL structure entries 715 are addressable locations in the RPL structure 700 that indicate the reference pictures associated with the reference picture lists, such as RPL0 711 and / or RPL1 712. Each entry 715 may include a picture order count (POC) value (or other pointer value) that references a picture used for inter-prediction. Specifically, references to pictures used by unidirectional inter prediction are stored in RPL0 711, and references to pictures used by bidirectional inter prediction are stored in both RPL0 711 and RPL1 712. For example, bidirectional inter prediction may use one reference picture indicated by RPL0 711 and one reference picture indicated by RPL1 712.

[0108] In a particular example, the RPL structure 700 may be denoted as ref_pic_list_struct(listIdx, rplsIdx), where the list index (listIdx) 721 identifies the reference picture list RPL0 711 and / or RPL1 712, and the reference picture list structure index (rplsIdx) 725 identifies an entry 715 in the reference picture list. Thus, ref_pic_list_struct is a syntax structure that returns an entry 715 based on listIdx 721 and rplsIdx 725. An encoder may encode a portion of the RPL structure 700 for each non-intra-coded slice in a video sequence. A decoder may then resolve the corresponding portion of the RPL structure 700 before decoding each non-intra-coded slice in the coded video sequence. The entry 715 that indicates a reference picture that is available for use when coding the current picture according to inter prediction is referred to as an active entry. Entries 715 that cannot be used for the current picture are called inactive entries.

[0109] 8 is a schematic diagram illustrating an example bitstream 800 including pictures with mixed NAL unit types. For example, bitstream 800 may be generated by codec system 200 and / or encoder 300 for decoding by codec system 200 and / or decoder 400 according to method 100. Furthermore, bitstream 800 may include VR picture video stream 500 merged from multiple sub-picture video streams 501-503 at multiple video resolutions. Furthermore, bitstream 800 may include RPL structure 700 coded according to constraint 600.

[0110] The bitstream 800 includes a sequence parameter set (SPS) 810, multiple picture parameter sets (PPSs) 811, multiple slice headers 815, and image data 820. The SPS 810 includes sequence data common to all pictures in the video sequence included in the bitstream 800. Such data may include picture size, bit depth, coding tool parameters, bitrate limits, etc. The PPS 811 includes parameters that apply to the entire picture. Therefore, each picture in the video sequence may reference a PPS 811. Note that while each picture references a PPS 811, a single PPS 811 can include data for multiple pictures in some instances. For example, multiple similar pictures may be coded according to similar parameters. In such a case, a single PPS 811 may include data for such similar pictures. The PPS 811 may indicate the coding tool available for slices in the corresponding picture, quantization parameters, offsets, etc. The slice header 815 contains parameters specific to each slice in a picture. Therefore, there may be one slice header 815 per slice in a video sequence. The slice header 815 may include slice type information, a picture order count (POC), a reference picture list, prediction weights, tile entry points, deblocking parameters, etc. Note that the slice header 815 may also be referred to as a tile group header in some contexts.

[0111] The image data 820 includes video data coded according to inter-prediction and / or intra-prediction, as well as corresponding transformed and quantized residual data. For example, a video sequence includes multiple pictures 821 coded as image data 820. A picture 821 is a single frame of a video sequence and is therefore generally displayed as a single unit when displaying the video sequence. However, subpictures 823 may be displayed to implement certain technologies, such as virtual reality. Each picture 821 references a PPS 811. A picture 821 may be divided into subpictures 823, tiles, and / or slices. A subpicture 823 is a spatial region of a picture 821 that is consistently applied throughout the coded video sequence. Thus, the subpicture 823 may be displayed by an HMD in a VR context. Furthermore, subpictures 823 with a specified POC may be obtained from the subpicture video streams 501-503 at the corresponding resolution. The subpictures 823 may reference an SPS 810. In some systems, a slice 825 is referred to as a tile group containing tiles. A slice 825 and / or a tile group of tiles references a slice header 815. A slice 825 may be defined as an integer number of complete tiles contained exclusively in a single NAL unit or an integer number of consecutive complete coding tree unit (CTU) rows within a tile of a picture 821. Therefore, a slice 825 is further divided into CTUs and / or coding tree blocks (CTBs). The CTUs / CTBs are further divided into coding blocks based on the code tree. The coding blocks may then be coded / decoded according to a prediction mechanism.

[0112] Parameter sets and / or slices 825 are coded in NAL units. A NAL unit may be defined as a syntactic structure that includes an indication of the type of data to follow and bytes containing that data in the form of RBSPs, interspersed with emulation prevention bytes, if necessary. More specifically, a NAL unit is a storage unit that includes a parameter set or slice 825 for a picture 821 and a corresponding slice header 815. Specifically, a VCL NAL unit 840 is a NAL unit that includes a slice 825 for a picture 821 and a corresponding slice header 815. Furthermore, non-VCL NAL units 830 include parameter sets, such as SPS 810 and PPS 811. Several types of NAL units may be used. For example, SPS 810 and PPS 811 may be included in the SPS NAL unit type (SPS_NUT) 831 and PPS NAL unit type (PPS_NUT) 832, respectively, both of which are non-VCL NAL units 830. In this way, a decoder can read SPS_NUT 831 from bitstream 800 to obtain SPS 810 coded by the encoder. Similarly, a decoder can read PPS_NUT 832 from bitstream 800 to obtain PPS 811 coded by the encoder.

[0113] IRAP picture / subpicture slices 825 may be included in IRAP NAL units 845. Non-IRAP picture / subpicture slices 825, e.g., leading and trailing pictures, may be included in non-IRAP NAL units 849. For example, slices 825 may be included in a single VCL NAL unit 840. The VCL NAL unit 840 may then be assigned a type identifier based on the type of picture 821 and / or subpicture 823 that contains the slice 825. For example, a slice 825 taken from a subpicture 823 that is a CRA subpicture is included in CRA_NUT 843. The bitstream 800 includes several types of IRAP NAL units 845, and therefore includes picture / subpicture types that include IDRs without leading pictures, IDRs with random-access decodable leading (RADL) pictures, and CRA pictures. The bitstream 800 also includes several types of non-IRAP NAL units 849, and therefore picture / subpicture types, including random access skip leading (RASL) pictures, RADL pictures, and trailing pictures.

[0114] A leading picture is a picture coded after an IRAP picture in decoding order and before that picture in presentation order. An IRAP NAL unit 845 is any NAL unit containing a slice 825 taken from an IRAP picture or subpicture. A non-IRAP NAL unit 849 is any NAL unit containing a slice 825 taken from any picture that is not an IRAP picture or subpicture (e.g., a leading or trailing picture). Because both IRAP NAL unit 845 and non-IRAP NAL unit 849 contain slice data, they are both VCL NAL units 840. In an exemplary embodiment, an IRAP NAL unit 845 may contain a slice 825 from an IDR associated with an IDR picture or RADL picture without a leading picture in an IDR_N_LP NAL unit 841 or an IDR_w_RADL NAL unit 842, respectively. Additionally, IRAP NAL units 845 may include slices 825 from CRA pictures in CRA_NUT 843. In an exemplary embodiment, non-IRAP NAL units 849 may include slices 825 from RASL pictures, RADL pictures, or trailing pictures in RASL_NUT 846, RADL_NUT 847, or TRAIL_NUT 848, respectively. In an exemplary embodiment, a complete list of possible NAL units is shown below, sorted by NAL unit type: [Table 1-1] [Table 1-2]

[0115] As described above, a VR video stream can contain subpictures 823 with IRAP pictures at different frequencies. This allows fewer IRAP pictures to be used for spatial regions that a user is less likely to see, and more IRAP pictures to be used for spatial regions that a user is more likely to view. In this way, spatial regions to which a user is more likely to switch back periodically can be quickly adjusted to a higher resolution. When this approach results in a picture 821 that contains both IRAP NAL units 845 and non-IRAP NAL units 849, the picture 821 is called a mixed picture. This state can be signaled by the mixed NAL unit type in the picture flag (mixed_nalu_types_in_pic_flag) 827. The mixed_nalu_types_in_pic_flag 827 may be set in the PPS 811. Additionally, mixed_nalu_types_in_pic_flag 827 may be set equal to 1 to specify that each picture 821 that references PPS 811 has multiple VCL NAL units 840 and that the VCL NAL units 840 do not have the same value of NAL unit type (nal_unit_type). Additionally, mixed_nalu_types_in_pic_flag 827 may be set equal to 0 when each picture 821 that references PPS 811 has multiple VCL NAL units 840 and that the VCL NAL units 840 of each picture 821 all have the same value of nal_unit_type.

[0116] Furthermore, various constraints may be used such that when mixed_nalu_types_in_pic_flag 827 is set, one or more VCL NAL units 840 in subpictures 823 of picture 821 all have a first particular value of NAL unit type, and all other VCL NAL units 840 in picture 821 have a second, different particular value of NAL unit type. For example, a constraint may require that mixed picture 821 contain a single type of IRAP NAL unit 845 and a single type of non-IRAP NAL unit 849. For example, picture 821 may contain one or more IDR_N_LP NAL units 841, one or more IDR_w_RADL NAL units 842, or one or more CRA_NUT 843, but not any combination of such IRAP NAL units 845. Furthermore, a picture 821 may contain one or more RASL_NUTs 846, one or more RADL_NUTs 847, or one or more TRAIL_NUTs 848, but not any combination of such IRAP NAL units 845. Additionally, a subpicture 823 may be constrained to only one type of VCL NAL unit 840. Furthermore, constraints 600 may be applied to a subpicture 823 based on the type of VCL NAL unit 840 used in the bitstream 800.

[0117] The preceding information is described in more detail below in this specification. In a video codec specification, picture types may be identified to define the decoding process, which may include for the derivation of picture identification (e.g., POC), for marking reference picture status in the DPB, for output of pictures from the DPB, etc.

[0118] In AVC and HEVC, picture types can be identified from the NAL unit type that contains the coded picture. AVC picture types include IDR and non-IDR pictures. HEVC picture types include trailing pictures, temporal sub-layer access pictures (TSA), gradual temporal sub-layer access pictures (STSA), random access decodable leading pictures (RADL), random access skip leading pictures (RASL), broken link access pictures (BLA), IDR, and CRA. Each of these picture types in HEVC can be further distinguished as a sub-layer reference picture or a sub-layer non-reference picture. BLA pictures include BLA with a leading picture, BLA with a RADL picture, and BLA without a leading picture. IDR pictures can include IDR with a RADL picture and IDR without a leading picture.

[0119] In HEVC, IDR, BLA, and CRA pictures are IRAP pictures. VVC uses IDR and CRA pictures as IRAP pictures. IRAP pictures provide the following features / advantages: The presence of an IRAP picture indicates that the decoding process can start from that picture. This functionality supports a random access function that allows the decoding process to start at a position in the bitstream as long as an IRAP picture exists at that position. The position may not be at the beginning of the bitstream. The presence of an IRAP picture 402 refreshes the decoding process so that pictures coded after the IRAP picture, except for RASL pictures, are coded without any reference to the preceding IRAP picture. Therefore, an IRAP picture prevents errors that occur before the IRAP picture in decoding order from propagating to pictures after the IRAP picture.

[0120] While providing the above-mentioned functionality, IRAP pictures incur a penalty in compression efficiency. The presence of IRAP pictures also causes a surge in bitrate. This penalty in compression efficiency has two causes. First, IRAP pictures are intra-predicted pictures, and therefore, IRAP pictures are represented with more bits than inter-predicted pictures. Second, the presence of IRAP pictures can disrupt temporal prediction by refreshing the decoding process when reference pictures are removed from the DPB. This can result in less efficient coding of pictures after the IRAP picture as fewer reference pictures are available for inter-prediction.

[0121] IDR pictures in HEVC may be derived and signaled differently than other picture types. Some of the differences are as follows: When signaling and deriving the POC value for an IDR picture, the most significant bit (MSB) of the POC may be set equal to 0 rather than being derived from the previous key picture. Furthermore, the slice header of an IDR picture may not contain information to assist reference picture management. For other picture types, such as CRA and trailing, a reference picture set (RPS) or reference picture list may be used for the reference picture marking process. This process is used to determine the status of a reference picture in the DPB as used for reference or not used for reference. For an IDR picture, such information does not need to be signaled, because the presence of IDR indicates that the decoding process should mark all reference pictures in the DPB as not used for reference.

[0122] In addition to picture type, picture identification is also used for multiple purposes. This includes picture identification for reference pictures in inter prediction, picture identification for output from the DPB, picture identification for motion vector scaling, and picture identification for weighted prediction. In AVC and HEVC, pictures can be identified by POC. In AVC and HEVC, pictures in the DPB can be marked as used for short-term reference, long-term reference, or unused for reference. When a picture is marked as unused for reference, it can no longer be used for prediction. When a picture is no longer needed for output, it can be removed from the DPB. AVC uses short-term and long-term reference pictures. A reference picture may be marked as unused for reference when it is no longer needed for prediction reference. The conversion between short-term, long-term, and unused for reference is controlled by the decoded reference picture marking process. An implicit sliding window process and an explicit memory management control operation (MMCO) process can be used as decoded reference picture marking mechanisms. The sliding window process marks a short-term reference picture as unused for reference when the number of reference frames equals the maximum number specified in the SPS. Short-term reference pictures are stored in a first-in, first-out manner, so that the most recently decoded short-term picture is kept in the DPB. An explicit MMCO process can include multiple MMCO commands. An MMCO command may mark one or more short-term or long-term reference pictures as unused for reference, or may mark all pictures as unused for reference. An MMCO command may also mark the current reference picture or an existing short-term reference picture as long-term and assign a long-term picture index to the long-term reference picture. In AVC, the reference picture marking operation, as well as the process of outputting and removing pictures from the DPB, are performed after a picture is decoded.

[0123] HEVC uses the RPS for reference picture management. For each slice, the RPS may contain the complete set of reference pictures used by the current picture or any subsequent pictures. Thus, the RPS signals the complete set of all pictures to be kept in the DPB for use by the current or subsequent pictures. This differs from the AVC scheme, in which only relative changes to the DPB are signaled. The RPS may not retain information from previous pictures during decoding to maintain the exact state of reference pictures in the DPB. The picture decoding order and DPB operation in HEVC take advantage of the RPS and improve error resilience. In AVC, picture marking and buffering operations may be applied after the current picture is decoded. In HEVC, the RPS is first decoded from the slice header of the current picture. Then, picture marking and buffering operations are applied before decoding the current picture.

[0124] VVC may directly signal and derive reference picture list 0 and reference picture list 1. Reference picture lists are not based on the RPS, sliding window, or MMCO process, as in HEVC and AVC. Reference picture marking is performed directly based on reference picture lists 0 and 1 by utilizing both active and inactive entries in the reference picture lists. Only active entries may be used as reference indices for inter prediction of CTUs of the current picture. Information for the derivation of the two reference picture lists is signaled by syntax elements and structures in the SPS, PPS, and slice header. A predefined RPL structure is signaled in the SPS for use by referencing in the slice header. Two reference picture lists are generated for all types of slices, including bidirectional inter prediction (B), unidirectional inter prediction (P), and intra prediction (I) slices. The two reference picture lists are constructed without using the reference picture list initialization process or the reference picture list modification process. The long-term reference picture (LTRP) is identified by the POC LSB. A delta POC MSB cycle may be signaled for the LTRP as desired on a per-picture basis.

[0125] HEVC can use regular slices, dependent slices, tiles, and wavefront parallel processing (WPP) as partitioning schemes. These partitioning schemes may be applied to maximum transmission unit (MTU) size matching, parallel processing, and end-to-end delay reduction. Each regular slice may be encapsulated in a separate NAL unit. Entropy coding dependencies and intra-picture prediction, including intra-sample prediction, motion information prediction, and coding mode prediction, may be disabled across slice boundaries. Therefore, regular slices can be reconstructed independently from other regular slices within the same picture. However, slices may still have some interdependencies due to loop filtering operations.

[0126] Regular slice-based parallelization may not require significant inter-processor or inter-core communication. One exception is that inter-processor and / or inter-core data sharing can be important for motion compensation when decoding predictively coded pictures. Such a process may involve more processing resources than inter-processor or inter-core data sharing due to intra-picture prediction. However, for the same reasons, the use of regular slices can incur substantial coding overhead due to the bit cost of slice headers and the lack of prediction across slice boundaries. Furthermore, regular slices serve as a mechanism for bitstream partitioning to match MTU size requirements due to their intra-picture independence and the fact that each regular slice is encapsulated in a separate NAL unit. In many cases, the goals of parallelization and MTU size matching place conflicting demands on the slice layout within a picture.

[0127] Dependent slices have short slice headers and allow bitstream partitioning at treeblock boundaries without breaking intra-picture prediction. Dependent slices provide fragmentation of regular slices into multiple NAL units. This reduces end-to-end delay by allowing transmission of part of a regular slice before the entire regular slice has finished encoding.

[0128] In WPP, a picture is partitioned into single rows of CTBs. Entropy decoding and prediction may use data from CTBs in other partitions. Parallel processing is possible through parallel decoding of CTB rows. The start of decoding of a CTB row may be delayed by one or two CTBs, depending on the example, ensuring that data related to the CTB above and to the right of the target CTB is available before the symmetric CTB is decoded. This staggered start creates the appearance of a wavefront. This process supports parallelization for up to as many processors / cores as there are CTB rows in a picture. Because intra-picture prediction between neighboring tree block rows within a picture is allowed, inter-processor / inter-core communication to enable intra-picture prediction can be substantial. WPP partitioning does not result in the generation of additional NAL units. Therefore, WPP does not need to be used for MTU size matching. However, if MTU size matching is required, regular slices can be used in WPP with certain coding overhead.

[0129] Tiles define horizontal and vertical boundaries that partition a picture into tile columns and rows. The scan order of CTBs may be local within a tile, in the order of the tile's CTB raster scan. Thus, a tile may be completely decoded before decoding the top-left CTB of the next tile in the picture's tile raster scan order. Similar to regular slices, tiles break not only entropy decoding dependencies but also intra-picture prediction dependencies. However, tiles may not be contained in individual NAL units. Therefore, tiles may not be used for MTU size matching. Each tile may be processed by one processor / core. Inter-processor / inter-core communication used for intra-picture prediction between processing units decoding neighboring tiles may be limited to carrying a shared slice header when a slice contains multiple tiles and sharing of loop filtering relationships of reconstructed samples and metadata. When multiple tiles or WPP segments are included in a slice, the entry point byte offset for each tile or WPP segment other than the first tile or WPP segment in the slice may be signaled in the slice header.

[0130] For simplicity, HEVC uses specific restrictions on the application of the four different picture partitioning schemes. A coded video sequence may not contain both tiles and wavefronts for most of the profiles specified in HEVC. Furthermore, for each slice and / or tile, one or both of the following conditions must be met: All coded treeblocks within a slice are contained in the same tile. Furthermore, all coded treeblocks within a tile are contained in the same tile. Additionally, a wavefront segment contains exactly one CTB row. When using WPP, a slice that starts within a CTB row should also end on the same CTB row.

[0131] HEVC may include the Temporal Motion Constrained Tile Set (MCTS) Supplemental Enhancement Information (SEI) message, the MCTS Extraction Information Set SEI message, and the MCTS Extraction Information Nesting SEI message. The temporal MCTS SEI message indicates the presence of an MCTS in the bitstream and signals the MCTS. For each MCTS, motion vectors are restricted to point to full sample positions within the MCTS and fractional sample positions that use only full sample positions within the MCTS for interpolation. The use of motion vector candidates derived from blocks outside the MCTS for temporal motion vector prediction is not permitted. In this way, each MCTS can be independently decoded without the presence of tiles not included in the MCTS. The MCTS Extraction Information Set SEI message provides supplemental information used in MCTS sub-bitstream extraction to generate a compliant bitstream for the MCTS set. The information includes a number of extraction information sets, each defining a number of MCTS sets, including the RBSP bytes of the replacement video parameter set (VPS), SPS, and PPS to be used during the MCTS sub-bitstream extraction process. When extracting a sub-bitstream according to the MCTS sub-bitstream extraction process, parameter sets such as VPS, SPS, and PPS may be rewritten or replaced. The slice header may also be updated because one or more slice address related syntax elements may have different values ​​after sub-bitstream extraction.

[0132] VVC may partition a picture as described below. A picture may be divided into tile groups and tiles. A tile may be a sequence of CTUs that cover a rectangular area of ​​the picture. A tile group, also known as a slice, may contain a certain number of tiles of the picture. Slices / tile groups may be organized according to raster scan mode and rectangular mode. In raster scan mode, a tile group / slice contains a sequence of tiles in raster scan order relative to the picture boundary. In rectangular mode, a tile group / slice contains a certain number of tiles that collectively form a rectangular area of ​​the picture. The tiles in a rectangular tile group are in raster scan order relative to the tile group / slice.

[0133] 360-degree video applications (e.g., VR) may display only a portion of the content sphere, resulting in only a subset of the entire picture. Viewport-dependent 360-degree delivery can be employed to reduce the bitrate for VR video delivery over DASH. Viewport-dependent coding may split the entire sphere / projected picture (e.g., using cube-map projection) into multiple MCTSs. Two or more bitstreams may then be coded at different spatial resolutions or qualities. The MCTS from the higher-resolution / quality bitstream is sent to the decoder for the viewport being displayed (e.g., the front viewport). The remaining viewports use MCTSs from the lower-resolution / quality bitstream. These MCTSs are packed in a specific manner and then sent to the receiver for decoding. The viewport that the user typically views is represented by the higher-resolution / quality MCTS, providing a good viewing experience. When the user turns to look at a different viewport (e.g., the left or right viewport), the displayed content comes from the lower-resolution / quality viewport. This continues for a short period of time until the system fetches the higher resolution / quality MCTS for that viewport.

[0134] There is a delay between the time when the user turns and the time when the higher resolution / quality representation of the viewport is displayed. This delay is based on how quickly the system can fetch the higher resolution / quality MCTS for that viewport. It depends on the IRAP period, which is the interval between two IRAPs. This is because the MCTS for a new viewport can only be decoded starting from the IRAP picture. If the IRAP period is coded every second, the following applies: The best-case scenario for delay is the network round-trip delay when the user turns to view the new viewport, just before the system starts fetching the new segment / IRAP period. In this scenario, the system can immediately request the higher resolution / quality MCTS for the new viewport. Therefore, the only delay is the network round-trip delay, which includes the fetch request delay plus the transmission time of the requested MCTS. This assumes that the minimum buffer delay can be set to zero or some other negligible value. The network round-trip delay can be, for example, about 200 milliseconds. The worst-case scenario for latency is the IRAP period plus the network round-trip delay when the user turns to view the new viewport immediately after the system makes a request for the next segment. To improve this worst-case scenario, the bitstream can be coded with more frequent IRAP pictures, resulting in shorter IRAP periods. This reduces overall latency. However, more IRAP pictures increases bandwidth and therefore reduces compression efficiency.

[0135] Mixed NAL unit types may be used within a picture by adding a PPS flag that specifies whether all VCL NAL units of the picture have the same NAL unit type. Constraints may be added to require that, for a particular picture, all VCL NAL units are of the same NAL unit type, or that some VCL NAL units have a specific IRAP NAL unit type and the rest have a specific non-IRAP VCL NAL unit type. An example description of the mechanism follows:

[0136] An IRAP picture is a coded picture with mixed_nalu_types_in_pic_flag equal to zero and each VCL NAL unit has a NalUnitType in the range from IDR_W_RADL to CRA_NUT, inclusive.

[0137] The syntax of an exemplary PPS RBSP is as follows: [Table 2]

[0138] The semantics of an exemplary NAL unit header are as follows: An IDR picture with a NalUnitType equal to IDR_N_LP has no associated leading picture in the bitstream. An IDR picture with a NalUnitType equal to IDR_W_RADL has no associated RADL picture present in the bitstream, but may have associated RADL pictures in the bitstream. For the VCL NAL units of any particular picture, the following applies: If mixed_nalu_types_in_pic_flag is equal to zero, all VCL NAL units SHOULD have the same value of nal_unit_type. Otherwise, some of the VCL NAL units SHOULD have a specific IRAP NAL unit type value (e.g., a value of nal_unit_type in the range from IDR_W_RADL to CRA_NUT, inclusive), while all other VCL NAL units SHOULD have a specific non-IRAP VCL NAL unit type (e.g., a value of nal_unit_type in the range from TRAIL_NUT to RSV_VCL_15, inclusive, or a value equal to GRA_NUT).

[0139] The semantics of an exemplary PPS RBSP are as follows: pps_seq_parameter_set_id specifies the value of sps_seq_parameter_set_id for the active SPS. The value of pps_seq_parameter_set_id shall be in the range of 0 to 5 (inclusive). mixed_nalu_types_in_pic_flag may be set equal to 1 to specify that each picture referencing the PPS has multiple VCL NAL units, and that these NAL units do not have the same value of nal_unit_type. mixed_nalu_types_in_pic_flag may be set equal to 0 to specify that the VCL NAL units of each picture referencing the PPS have the same value of nal_unit_type. When sps_idr_rpl_present_flag is equal to 0, the value of mixed_nalu_types_in_pic_flag shall be equal to 0.

[0140] A picture may be divided into sub-pictures. An indication of the presence of a sub-picture may be indicated in the SPS along with other sequence-level information for the sub-picture. Whether a sub-picture boundary is treated as a picture boundary in the decoding process (except for in-loop filtering operations) may be controlled by the bitstream. Whether in-loop filtering across sub-picture boundaries is disabled may be controlled by the bitstream for each sub-picture. The deblocking filter (DBF), SAO, and adaptive loop filter (ALF) processes are updated to control in-loop filtering operations across sub-picture boundaries. The sub-picture width, height, horizontal offset, and vertical offset may be signaled in units of luma samples in the SPS. The sub-picture boundary may be constrained to be a slice boundary. Treating subpictures as pictures in the decoding process (except for the in-loop filtering process) is specified by updating the coding_tree_unit() syntax, the derivation process for advanced temporal luma motion vector prediction, the luma sample bilinear interpolation process, the luma sample 8-tap interpolation filtering process, and the chroma sample interpolation process. Subpicture identifiers (IDs) are explicitly specified in the SPS and included in the tile group header to enable extraction of subpicture sequences without having to modify the VCL NAL units. An output subpicture set (OSPS) may specify canonical extraction and fit points for subpictures and their sets.

[0141] Prior systems have a particular problem: a bitstream may contain pictures with both IRAP slices and non-IRAP slices. If a slice in a picture that follows such a picture in decoding order and covers the same picture area as the IRAP slice in the picture references an earlier picture for inter prediction, an error occurs.

[0142] This disclosure includes improved techniques for supporting subpicture or MCTS-based random access in video coding. More specifically, this disclosure describes a method for imposing constraints on IRAP slices within a picture that contains both IRAP slices and non-IRAP slices. However, this technique may also be applied to other video / media codec specifications.

[0143] For example, a constraint is added to ensure that slices of a picture that (1) are after the mixed NAL unit picture in decoding order and (2) cover the same picture region of an IRAP slice within a picture do not reference reference pictures that are earlier in decoding order than the mixed NAL unit picture for inter prediction.

[0144] An IRAP picture may be defined as a coded picture with mixed_nalu_types_in_pic_flag equal to zero and each VCL NAL unit having a NalUnitType in the range from IDR_W_RADL to CRA_NUT, inclusive.

[0145] The syntax of an exemplary PPS RBSP is as follows: [Table 3]

[0146] The semantics of an exemplary NAL unit header are as follows: An IDR picture with a NalUnitType equal to IDR_N_LP has no associated leading picture in the bitstream. An IDR picture with a NalUnitType equal to IDR_W_RADL has no associated RADL picture present in the bitstream, but may have associated RADL pictures in the bitstream. For the VCL NAL units of any particular picture, the following applies: If mixed_nalu_types_in_pic_flag is equal to zero, all VCL NAL units shall have the same value of nal_unit_type. Otherwise, some VCL NAL units shall have a specific IRAP NAL unit type value (e.g., a value of nal_unit_type in the range from IDR_W_RADL to CRA_NUT, inclusive), while all other VCL NAL units shall have a specific non-IRAP VCL NAL unit type (e.g., a value of nal_unit_type in the range from TRAIL_NUT to RSV_VCL_15, inclusive, or a value equal to GRA_NUT).

[0147] The semantics of an exemplary PPS RBSP are as follows: pps_seq_parameter_set_id specifies the value of sps_seq_parameter_set_id for the active SPS. The value of pps_seq_parameter_set_id shall be in the range of 0 to 15 (inclusive). mixed_nalu_types_in_pic_flag may be set equal to 1 to specify that each picture referencing the PPS has multiple VCL NAL units and that these NAL units do not have the same value of nal_unit_type. mixed_nalu_types_in_pic_flag may be set equal to 0 to specify that the VCL NAL units of each picture referencing the PPS have the same value of nal_unit_type. When sps_idr_rpl_present_flag is equal to 0, the value of mixed_nalu_types_in_pic_flag shall be equal to 0. For each IRAP slice in picture picA that also has at least one non-IRAP slice, the following applies: An IRAP slice shall belong to subpicture subpicA, and subpicture boundaries shall be treated as picture boundaries in the decoding process (except for in-loop filtering operations). For example, the value of sub_pic_treated_as_pic_flag[i] for subpicA shall be equal to 1. An IRAP slice shall not belong to a subpicture in the same picture that contains one or more non-IRAP slices. For all layer access units (AUs) below, in decoding order, neither RefPicList[0] nor RefPicList[1] of a slice in subpicA shall contain, in active entries, any picture preceding picA in decoding order.

[0148] FIG. 9 is a schematic diagram of an exemplary video coding device 900. The video coding device 900 is suitable for implementing the disclosed examples / embodiments described herein. The video coding device 900 includes a downstream port 920, an upstream port 950, and / or a transceiver unit 910 including a transmitter and / or receiver for communicating data upstream and / or downstream over a network. The video coding device 900 also includes a processor 930 including a logic unit and / or central processing unit (CPU) for processing data and a memory 932 for storing data. The video coding device 900 may also include electrical components, optical-to-electrical (OE) components, electrical-to-optical (EO) components, and / or wireless communication components coupled to the upstream port 950 and / or downstream port 920 for communicating data over an electrical, optical, or wireless communication network. The video coding device 900 may also include input and / or output (I / O) devices 960 for communicating data to and from a user. The I / O devices 960 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices 960 may also include input devices such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0149] The processor 930 is implemented in hardware and software. The processor 930 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 930 is in communication with the downstream port 920, the Tx / Rx 910, the upstream port 950, and the memory 932. The processor 930 includes a coding module 914. The coding module 914 implements the disclosed embodiments described herein, such as the methods 100, 1000, and 1100, which may use the VR picture video stream 500, the bitstream 800, and / or the RPL structure 700 coded according to the constraints 600. The coding module 914 may also implement any other method / mechanism described herein. Furthermore, coding module 914 may implement codec system 200, encoder 300, and / or decoder 400. For example, coding module 914 may code a VR video including a current picture that has both IRAP NAL units and non-IRAP NAL units. For example, an IRAP NAL unit may be included in a sub-picture. When this occurs, coding module 914 may constrain slices in pictures after the current picture that are also included in the sub-picture. Such slices may be prevented from referencing reference pictures that precede the current picture. Therefore, coding module 914 allows video coding device 900 to provide additional functionality and / or coding efficiency when coding video data. In this way, coding module 914 improves the functionality of video coding device 900 while addressing problems inherent in video coding techniques. Furthermore, coding module 914 facilitates transformation of video coding device 900 into different states.Alternatively, the coding module 914 may be implemented as instructions stored in memory 932 and executed by the processor 930 (e.g., as a computer program product stored on a non-transitory medium).

[0150] Memory 932 may include one or more memory types such as a disk, tape drive, solid state drive, read-only memory (ROM), random access memory (RAM), flash memory, ternary content addressable memory (TCAM), static random access memory (SRAM), etc. Memory 932 is used as an overflow data storage device to store programs when such programs are selected for execution and to store instructions and data read during program execution.

[0151] 10 is a flowchart of an example method 1000 for encoding a video sequence including pictures with mixed NAL unit types into a bitstream, such as bitstream 800 including VR picture video stream 500 with RPL structure 700. Method 1000 can encode such a bitstream according to constraint 600. Method 1000 may be used by an encoder, such as codec system 200, encoder 300, and / or video coding device 900, when performing method 100.

[0152] Method 1000 may begin when an encoder receives a video sequence including multiple pictures, such as VR pictures, and determines, based on user input, for example, to encode the video sequence into a bitstream. The bitstream may include VR video data. The VR video data may include pictures, each representing a sphere of content at a corresponding instant in the video sequence. A picture may be partitioned into a set of subpictures. For example, each subpicture may include video data corresponding to a viewport of the VR video. Furthermore, various subpictures may include IRAP NAL units and non-IRAP NAL units with varying frequencies. In step 1001, the encoder determines that the current picture includes multiple VCL NAL units that do not have the same NAL unit type. For example, a VCL NAL unit may include an IRAP NAL unit having an IRAP NAL unit type and a non-IRAP NAL unit having a non-IRAP NAL unit type. For example, IRAP NAL unit types may include an IDR NAL unit type or a CRA NAL unit type. Additionally, non-IRAP NAL unit types may include a tail NAL unit type, a RASL NAL unit type, and / or a RADL NAL unit type.

[0153] In step 1003, the encoder encodes a flag into the bitstream. The flag indicates that a first NAL unit type value for a VCL NAL unit of the current picture is different from a second NAL unit type value for a VCL NAL unit of the current picture. In one example, the flag can be encoded into the PPS in the bitstream. As a specific example, the flag can be mixed_nalu_types_in_pic_flag 827. mixed_nalu_types_in_pic_flag can be set equal to 1 to specify that each picture that references the PPS has multiple VCL NAL units and that no VCL NAL units have the same value of nal_unit_type.

[0154] In step 1005, the encoder determines that subpicA in the current picture is associated with an IRAP NAL unit type. In step 1007, the encoder encodes the current picture into a bitstream. For example, the encoder ensures that all slices of the current picture mapped to subpicA are associated with the same NAL unit type (e.g., an IRAP NAL unit type). The encoder may encode subpicA according to intra prediction. The encoder may also determine that subpicB contains slices of a non-IRAP NAL unit type. Therefore, the encoder may encode subpicB of the current picture according to inter prediction.

[0155] In step 1009, the encoder may prepare to encode a subsequent picture after the current picture in decoding order. For example, the encoder may generate an active entry in a reference picture list for a slice located in a subpicture in the subsequent picture. The active entry for a specified subsequent picture indicates a picture that can be used as a reference picture when performing inter-prediction encoding on the specified subsequent picture. Specifically, the active entry for subpicA in the subsequent picture is constrained so that such an active entry does not reference any reference picture that precedes the current picture in decoding order. This constraint is used when subpicA in the current picture is associated with an IRAP NAL unit type. This constraint ensures that a slice in a subpicture after an IRAP subpicture does not reference a picture before the IRAP subpicture; otherwise, a coding error would occur if the IRAP subpicture were used as a random access point. Slices of subsequent pictures that do not follow an IRAP subpicture (eg, those located in subpicB and that follow a non-IRAP NAL unit) may continue to reference pictures that precede the current picture.

[0156] In step 1011, the encoder encodes the subsequent picture into a bitstream based on the reference picture list. For example, the encoder can encode the subsequent picture based on inter prediction and / or intra prediction depending on the NAL type associated with the slice in the corresponding subpicture. The inter prediction process uses a reference picture list. The reference picture list may include Reference Picture List 0 and Reference Picture List 1. Furthermore, the reference picture list may be coded in a reference picture list structure. The reference picture list structure may be coded into the bitstream. The encoder can also store the bitstream for communication to the decoder.

[0157] 11 is a flowchart of an example method 1100 for decoding a video sequence including pictures with mixed NAL unit types from a bitstream, such as bitstream 800 including VR picture video stream 500 with RPL structure 700. Method 1100 may decode such a bitstream according to constraint 600. Method 1100 may be used by a decoder, such as codec system 200, decoder 400, and / or video coding device 900, when performing method 100.

[0158] Method 1100 may begin when a decoder begins receiving a bitstream of coded data representing a video sequence, e.g., as a result of method 1000. The bitstream may include a VR video sequence including multiple pictures, such as VR pictures. The bitstream may include VR video data. The VR video data may include pictures, each representing a sphere of content at a corresponding instant in the video sequence. A picture may be partitioned into a set of subpictures. For example, each subpicture may include video data corresponding to a viewport of the VR video. Furthermore, various subpictures may include IRAP NAL units and non-IRAP NAL units with varying frequencies. In step 1101, the decoder receives a bitstream. The bitstream includes a current picture that includes multiple VCL NAL units that do not have the same NAL unit type. For example, the VCL NAL units may include IRAP NAL units having an IRAP NAL unit type and non-IRAP NAL units having a non-IRAP NAL unit type. For example, IRAP NAL unit types may include an IDR NAL unit type or a CRA NAL unit type. Additionally, non-IRAP NAL unit types may include a tail NAL unit type, a RASL NAL unit type, and / or a RADL NAL unit type.

[0159] In step 1103, the decoder determines, based on the flag, that the first NAL unit type value for the VCL NAL unit of the current picture is different from the second NAL unit type value for the VCL NAL unit of the current picture. In one example, the bitstream may include a PPS associated with the current picture. The flag may be obtained from the PPS. As a specific example, the flag may be mixed_nalu_types_in_pic_flag 827. mixed_nalu_types_in_pic_flag may be set equal to 1 to specify that each picture referencing the PPS has multiple VCL NAL units and that no VCL NAL units have the same value of nal_unit_type.

[0160] In step 1105, the decoder may determine that all slices of the current picture mapped to subpicA are associated with the same NAL unit type. The decoder may also decode subpicA and / or the current picture based on the slice's NAL unit type. For example, subpicA in the current picture may include an IRAP NAL unit. In such a case, subpicA may be decoded according to intra prediction. The decoder may also determine that subpicB includes a slice of a non-IRAP NAL unit type. Therefore, the decoder may decode subpicB of the current picture according to inter prediction.

[0161] In step 1107, the decoder may obtain an active entry of a reference picture list for a slice located at subpicA in a subsequent picture after the current picture in decoding order. The active entry for the specified subsequent picture indicates a picture that can be used as a reference picture when performing inter-prediction decoding processing on the specified subsequent picture.

[0162] In step 1109, the decoder may determine that the active entry does not reference any reference picture preceding the current picture in decoding order when subpicA in the current picture is associated with an intra random access point (IRAP) NAL unit type. This constraint ensures that slices in subpictures after an IRAP subpicture do not reference pictures preceding the IRAP subpicture, which would otherwise cause a coding error when the IRAP subpicture is used as a random access point. Slices of subsequent pictures that do not follow the IRAP subpicture (e.g., those located in subpicB and following non-IRAP NAL units) may continue to reference pictures preceding the current picture.

[0163] In step 1111, the decoder can decode the subsequent picture based on the reference picture list active entries. For example, the decoder can decode the subsequent picture based on inter prediction and / or intra prediction, depending on the NAL type associated with the slice in the corresponding subpicture. The inter prediction process uses reference picture lists. The reference picture lists may include Reference Picture List 0 and Reference Picture List 1. Furthermore, the reference picture lists may be obtained from a reference picture list structure coded in the bitstream. The decoder can forward the current picture, the subsequent picture, and / or their subpictures (e.g., subpicA or subpicB) for display as part of the decoded video sequence.

[0164] 12 is a schematic diagram of an example system 1200 for coding a video sequence including pictures with mixed NAL unit types into a bitstream, such as bitstream 800 including VR picture video stream 500 having RPL structure 700 and coded according to constraint 600. System 1200 may be implemented by an encoder and decoder, such as codec system 200, encoder 300, decoder 400, and / or video coding device 900. Furthermore, system 1200 may be used when implementing methods 100, 1000, and / or 1100.

[0165] The system 1200 includes a video encoder 1202. The video encoder 1202 includes a determination module 1201 for determining that a current picture includes multiple VCL NAL units that do not have the same NAL unit type. The determination module 1201 is further for determining that subpicA in the current picture is associated with an IRAP NAL unit type. The video encoder 1202 also includes a generation module 1203 for generating an active entry of a reference picture list for a slice located in subpicA in a subsequent picture after the current picture, where the active entry does not reference any reference picture preceding the current picture in decoding order when subpicA in the current picture is associated with the IRAP NAL unit type. The video encoder 1202 also includes an encoding module 1205 for encoding the subsequent picture into a bitstream based on the reference picture list. The video encoder 1202 further includes a storage module 1207 for storing the bitstream for communication to a decoder. The video encoder 1202 further includes a transmission module 1209 for transmitting the bitstream to a video decoder 1210. The video encoder 1202 may be further configured to perform any of the steps of the method 1000.

[0166] System 1200 also includes a video decoder 1210. The video decoder 1210 includes a receiving module 1211 for receiving a bitstream including a current picture that includes multiple VCL NAL units that do not have the same NAL unit type. The video decoder 1210 further includes an obtaining module 1213 for obtaining an active entry of a reference picture list for a slice located in subpicA in a subsequent picture after the current picture. The video decoder 1210 further includes a determining module 1215 for determining that the active entry does not reference any reference picture that precedes the current picture in decoding order when subpicA in the current picture is associated with an IRAP NAL unit type. The video decoder 1210 further includes a decoding module 1217 for decoding the subsequent picture based on the reference picture list active entry. The video decoder 1210 further includes a forwarding module 1219 for forwarding the subsequent picture for display as part of a decoded video sequence. Video decoder 1210 may be further configured to perform any of the steps of method 1100.

[0167] A first component is directly coupled to a second component when there are no intervening components other than a line, trace, or other medium between the first and second components. A first component is indirectly coupled to a second component when there are intervening components other than a line, trace, or other medium between the first and second components. The term "coupled" and variations thereof include both directly coupled and indirectly coupled. The use of the term "about," unless otherwise specified, means a range that includes ±10% of the subsequent number.

[0168] It should also be understood that the steps of the exemplary methods described herein do not necessarily have to be performed in the order described, and that the order of steps of such methods is merely exemplary. Similarly, additional steps may be included in such methods, and certain steps may be omitted or combined, in methods consistent with various embodiments of the present disclosure.

[0169] While multiple embodiments are provided in this disclosure, it will be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the disclosure. The examples are considered to be illustrative and not limiting, and the intention is not to be limited to the details provided herein. For example, various elements or components may be combined or integrated into another system, or certain features may be omitted, or not implemented.

[0170] Additionally, the techniques, systems, subsystems, and methods described and illustrated individually or separately in various embodiments may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

1. 1. A method for decoding a bitstream, comprising: receiving the bitstream, the bitstream having coded data of a current picture and a subsequent picture after the current picture in decoding order, both of the current picture and the subsequent picture including a first subpicture and a second subpicture, the bitstream including a Picture Parameter Set (PPS), the PPS including a mixed_nalu_types_in_pic_flag, the mixed_nalu_types_in_pic_flag equal to 1 specifying that each picture referencing the PPS has multiple Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); Obtaining an active entry of a reference picture list for the first subpicture in the subsequent picture, wherein the active entry does not include a reference to any reference picture preceding the current picture in decoding order when the first subpicture in the current picture is associated with an Intra Random Access Point (IRAP) NAL unit type; and and decoding the first sub-picture in the subsequent picture based on the reference picture list active entry.

2. The method of claim 1 , wherein the IRAP NAL unit type is a clean random access (CRA) NAL unit type.

3. The method of claim 1 , wherein the IRAP NAL unit type is an Instantaneous Decoder Refresh (IDR) NAL unit type.

4. The method according to any one of claims 1 to 3, wherein all slices of the current picture located in the first sub-picture are associated with the same NAL unit type.

5. 5. The method of claim 1, wherein when the mixed_nalu_types_in_pic_flag is equal to 1, all VCL NAL units of one or more of the subpictures of the current picture have a specific value of IRAP NAL unit type, and all other VCL NAL units of the current picture have a specific value of non-IRAP NAL unit type.

6. The method of any one of claims 1 to 5, wherein when the mixed_nalu_types_in_pic_flag is equal to 0, all VCL NAL units of the current picture have the same NAL unit type.

7. The method of any one of claims 1 to 6, wherein the second sub-picture of the current picture is associated with a non-IRAP NAL unit type.

8. 1. A method for encoding a bitstream, comprising: encoding a current picture and a subsequent picture after the current picture into the bitstream, wherein the current picture and the subsequent picture both include a first sub-picture and a second sub-picture; encoding a Picture Parameter Set (PPS) into the bitstream, the PPS including a mixed_nalu_types_in_pic_flag, the mixed_nalu_types_in_pic_flag equal to 1 specifying that each picture referencing the PPS has multiple Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); The method of claim 1, wherein an active entry does not include a reference to any reference picture that precedes the current picture in decoding order when the first subpicture in the current picture is associated with an Intra Random Access Point (IRAP) NAL unit type.

9. The method of claim 8 , wherein the IRAP NAL unit type is a clean random access (CRA) NAL unit type.

10. 9. The method of claim 8, wherein the IRAP NAL unit type is an instantaneous decoder refresh (IDR) NAL unit type.

11. The method according to any one of claims 8 to 10, wherein all slices of the current picture located in the first sub-picture are associated with the same NAL unit type.

12. 12. The method of claim 8, wherein when the mixed_nalu_types_in_pic_flag is equal to 1, all VCL NAL units of one or more of the subpictures of the current picture have a specific value of IRAP NAL unit type, and all other VCL NAL units of the current picture have a specific value of non-IRAP NAL unit type.

13. The method of any one of claims 8 to 12, wherein when the mixed_nalu_types_in_pic_flag is equal to 0, all VCL NAL units of the current picture have the same NAL unit type.

14. The method of any one of claims 8 to 13, wherein the second sub-picture of the current picture is associated with a non-IRAP NAL unit type.

15. 1. A video coding device, comprising:

8. A video coding device comprising: a processor; a receiver coupled to the processor; a memory coupled to the processor; and a transmitter coupled to the processor, wherein the processor, the receiver, the memory, and the transmitter are configured to perform the method of any one of claims 1 to 7.

16. 1. A video coding device, comprising:

15. A video coding device comprising: a processor; a receiver coupled to the processor; a memory coupled to the processor; and a transmitter coupled to the processor, wherein the processor, the receiver, the memory, and the transmitter are configured to perform the method of any one of claims 8 to 14.

17. 10. A non-transitory computer readable medium comprising a computer program for use by a video coding device, the computer program comprising computer executable instructions stored on the non-transitory computer readable medium such that, when executed by a processor, the computer program causes the video coding device to perform the method of any one of claims 1 to 7.

18. 15. A non-transitory computer readable medium comprising a computer program for use by a video coding device, the computer program comprising computer executable instructions stored on the non-transitory computer readable medium such that, when executed by a processor, the computer program causes the video coding device to perform the method of any one of claims 8 to 14.

19. 1. A decoder comprising: receiving means for receiving a bitstream, the bitstream having coded data of a current picture and a subsequent picture after the current picture in decoding order, both of the current picture and the subsequent picture including a first subpicture and a second subpicture, the bitstream including a Picture Parameter Set (PPS), the PPS including a mixed_nalu_types_in_pic_flag, the mixed_nalu_types_in_pic_flag equal to 1 specifying that each picture referencing the PPS has multiple Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); obtaining means for obtaining an active entry of a reference picture list for the first subpicture in the subsequent picture, wherein the active entry does not include a reference to any reference picture preceding the current picture in decoding order when the first subpicture in the current picture is associated with an Intra Random Access Point (IRAP) NAL unit type; decoding means for decoding the first sub-picture in the subsequent picture based on the reference picture list active entry.

20. The decoder of claim 19, wherein the decoder is further configured to perform the method of any one of claims 2 to 7.

21. 1. An encoder comprising: encoding means for encoding a current picture and a subsequent picture after the current picture into a bitstream, wherein the current picture and the subsequent picture both include a first sub-picture and a second sub-picture; the encoding means encodes a Picture Parameter Set (PPS) into the bitstream, the PPS including a mixed_nalu_types_in_pic_flag, the mixed_nalu_types_in_pic_flag equal to 1 specifying that each picture referencing the PPS has multiple Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); an active entry does not include a reference to any reference picture that precedes the current picture in decoding order when the first subpicture in the current picture is associated with an intra random access point (IRAP) NAL unit type.

22. Encoder according to claim 21, wherein the encoder is further configured to perform a method according to any one of claims 9 to 14.

23. 1. A decoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming being configured to, when executed by the one or more processors, cause the decoder to perform the method of any one of claims 1 to 7.

24. 1. An encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming being configured to, when executed by the one or more processors, cause the encoder to perform the method of any one of claims 8 to 14.

25. A device for storing and transmitting a bitstream, comprising: a receiver, a processor, a transmitter, and a storage medium, wherein the receiver is configured to receive a bitstream, the storage medium is configured to store the bitstream, and the transmitter is configured to transmit the bitstream; the bitstream has coded data of a current picture and a subsequent picture after the current picture in decoding order, both of the current picture and the subsequent picture including a first subpicture and a second subpicture; the bitstream includes a Picture Parameter Set (PPS), the PPS includes a mixed_nalu_types_in_pic_flag, the mixed_nalu_types_in_pic_flag equal to 1 specifying that each picture referencing the PPS has multiple Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); The device, wherein the processor obtains an active entry of a reference picture list for the first subpicture in the subsequent picture, the active entry not including a reference to any reference picture preceding the current picture in decoding order when the first subpicture in the current picture is associated with an intra random access point (IRAP) NAL unit type.

26. 1. A system for a bitstream, comprising: an encoding device; a decoding device; and one or more storage media; the encoding device is configured to obtain a video signal and to encode the video signal to obtain one or more bitstreams; each of the one or more bitstreams has coded data of a current picture and a subsequent picture after the current picture in decoding order, both of the current picture and the subsequent picture including a first subpicture and a second subpicture; the bitstream includes a Picture Parameter Set (PPS), the PPS includes a mixed_nalu_types_in_pic_flag, the mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture referencing the PPS has multiple Video Coding Layer (VCL) Network Abstraction Layer (NAL) units, and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type); an active entry in a reference picture list for the first subpicture in the subsequent picture does not include a reference to any reference picture preceding the current picture in decoding order when the first subpicture in the current picture is associated with an Intra Random Access Point (IRAP) NAL unit type; the one or more storage media are used to store the one or more bitstreams; The decoding device is used to decode the one or more bitstreams.

Images