Sub-picture signaling in video coding

Abstract

Methods, devices and systems for video bitstream generation and parsing are described. An exemplary video bitstream processing method includes parsing a header for a first field at a video unit level in a video bitstream, the first field being used to indicate whether the video unit includes a plurality of network abstraction layer bit units of a plurality of types, upon determining that the video unit includes a plurality of network abstraction layer bit units of a plurality of types, parsing a second field in the header, the second field indicating whether the video unit includes an intra-frame random access point, and decoding the video bitstream based on the first field and the second field to generate a video.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. Patent Application No. 16 / 709,907 filed on December 10, 2019. The entire contents of the aforementioned patent application are incorporated by reference as part of the disclosure of this application. Technical Field

[0003] The patent document is generally directed to video and image encoding and decoding. Background Art

[0004] Over the past three decades, many different video coding standards have been published for representing digital video in a compressed format by removing some of the visual redundancy and information in the digital video. During this time, the pixel resolution of the encoded video content has steadily increased from the source input format (SIF) resolution in the early MPEG-1 video standard to the 4K or 8K video currently being considered. As a result, newer video coding standards have adopted more efficient and flexible coding techniques to accommodate the increased video resolution. Summary of the invention

[0005] This patent document describes, among other things, techniques for encoding and decoding digital video using techniques that can be used to signal sub-pictures in an encoded video bitstream and correspondingly decode the encoded video bitstream into unencoded video.

[0006] In one example aspect, a bitstream processing method is disclosed. The method includes parsing a header for a first field at a video unit level in a video bitstream, the first field indicating whether the video unit includes a plurality of network abstraction layer bit units of a plurality of types; parsing a second field in the header due to determining that the video unit includes a plurality of network abstraction layer bit units of a plurality of types, the second field indicating whether the video unit includes an intra-frame random access point; and decoding the video code stream based on the first field and the second field to generate a video.

[0007] In another example aspect, a method for parsing a video bitstream is disclosed. The method includes parsing a syntax structure to obtain type information about one or more network abstraction layer (NAL) units constituting the video picture and whether the video picture is an intra random access picture, and reconstructing the one or more sub-pictures based on the parsing.

[0008] In another example aspect, a method for encoding video is disclosed. The method includes including a first field in a header at a video unit level in a video bitstream, the first field indicating whether the video unit includes a plurality of network abstraction layer bit units of a plurality of types; due to determining that the video unit includes a plurality of network abstraction layer bit units of a plurality of types, further including a second field in the header, the second field indicating whether the video unit includes an intra-frame random access point; and encoding the video to generate a video bitstream.

[0009] In yet another exemplary aspect, a video processing device including a processor is disclosed. The processor is configured to implement the encoding or decoding method described herein.

[0010] In another example aspect, a computer program storage medium is disclosed. The computer program storage medium includes code stored thereon. The code, when executed by a processor, causes the processor to implement the described method.

[0011] These and other aspects are described in this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example of a video picture partitioned into multiple sub-pictures is shown.

[0013] FIG. 2A to FIG. 2B An example processing order is shown when encoding or decoding a video picture including multiple sub-pictures.

[0014] Figure 3A Examples of video pictures partitioned into coding tree units (CTUs), tiles, slices, and sub-pictures are shown.

[0015] Figure 3B A video picture with 24 sub-pictures and 24 slices is shown.

[0016] Figure 4 A block diagram of an example video encoder is shown.

[0017] Figure 5 A block diagram of an example video decoder is shown.

[0018] Figure 6 An example of a video system is shown.

[0019] Figure 7 An example of a video processing hardware platform is shown.

[0020] Figure 8 is a flow chart of an example method for video processing.

[0021] Fig. 9 is a flow chart of an example method for video processing.

[0022] Fig.10 is a flow chart of an example method for parsing a bitstream.

[0023] Fig.11 is a flow chart of an example method for video encoding. DETAILED DESCRIPTION

[0024] The section headings used in this document are only for readability and do not limit the scope of the embodiments and techniques disclosed in each section to only that section. Certain features are described using examples of the H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), and H.266 Versatile Video Coding (VVC) standards. However, the applicability of the disclosed techniques is not limited to H.264 / AVC or H.265 / HEVC or H.266 / WC systems.

[0025] The present disclosure relates to video processing and communication, and in particular to a method and apparatus for encoding a digital video or picture to generate a bitstream, a method and apparatus for decoding a bitstream to reconstruct a digital video or picture, and a method and apparatus for extracting a bitstream to form a sub-bitstream.

[0026] Brief Discussion

[0027] The technology used to compress digital video and pictures uses the correlation characteristics between pixel samples to remove redundancy in videos and pictures. The encoder can partition a picture into one or more regions containing multiple units. Such regions break the prediction dependencies within the picture so that the region can be decoded or at least the syntax elements corresponding to the region can be correctly parsed without referring to the data of another region in the same picture. Such regions are introduced in the video coding standard to facilitate resynchronization after data loss, parallel processing, areas of interest for encoding and streaming, packetized transmission, viewport-dependent streaming, etc. For example, in the H.264 / AVC standard, examples of such regions may be slices and slice groups. In the H.265 / HEVC standard, examples of such regions may be slices and tiles.

[0028] In the development of the next generation video coding standard, MPEG (Moving Picture Experts Group) requires that the next generation video codec should be able to efficiently extract the desired viewport from the compressed bitstream. In addition, in applications involving 4K, 8K or even higher resolution video, viewers can be allowed to select the viewport for presentation on the terminal device, so it is useful to be able to efficiently extract the sub-bitstream from the bitstream corresponding to the original resolution.

[0029] Since the H.265 / HEVC standard provides the highest coding efficiency among the currently released video coding standards, H.265 / HEVC is selected as the codec for deploying the above video services in the current stage. The H.265 / HEVC standard defines slices and tiles. A picture can be partitioned into one or more tiles and slices. According to the H.265 standard, the consistency requirement is that one or both of the following conditions are met for each slice and tile:

[0030] - All CTUs in a slice belong to the same tile.

[0031] -All CTUs in a tile belong to the same slice.

[0032] By limiting the loop filtering operation at tile boundaries (e.g., loop_filter_across_tiles_enabled_flag is set to 0 in the PPS), tiles in a picture can be decoded independently without reference to other areas of the picture outside the tile. It is assumed that the same tile partitioning is applied to all pictures in a CVS (coded video sequence). In addition, if the encoder limits the search range of an area including one or more tiles to a co-located area of ​​tiles in one or more reference pictures for inter-frame prediction coding, the decoder will decode the area without referencing any other parts outside the area both spatially in the current decoded picture containing the area and temporally in the reference pictures. Such an area can be called a motion constrained tile set (MCTS).

[0033] In the HEVC standard and the Versatile Video Coding (VVC) standard (note that the VVC version 1 standardization process will be completed in July 2020), in addition to the concepts of tiles and slices, another picture partition called sub-picture is included. Typically, sub-pictures, tiles, and slices partition video pictures into non-overlapping areas that cover the entire picture. Since sub-pictures, tiles, and slices are fairly independent concepts, overlaps may occur between the three of them. For example, a sub-picture can be formed by multiple rectangular slices; a tile can also contain multiple rectangular slices; and a slice can also contain multiple tiles, etc.

[0034] Since the VVC standard is needed to support virtual reality (VR) applications, it may be necessary to support and encode some large-size 360-degree videos. For such large-size videos, it is desirable to use partitions in 360 video pictures in video encoding. One potential technique is "sub-picture" partitioning. Various sub-pictures (all belonging to a single picture) can be encoded into different NAL unit types. In addition, sub-pictures can be grouped together to support various random access operations.

[0035] 1. An Example Embodiment

[0036] JVET-P2001vD (November 14, 2019), the current Versatile Video Coding (VVC) specification (spec) supports the concepts of sub-pictures, slices, and tiles in normal video picture partitioning. In Section 6.3.1 "Partitioning a Picture into Sub-Pictures, Slices, and Tiles", it is stated that "a sub-picture contains one or more slices that together cover a rectangular area of ​​the picture."

[0037] In the current VVC specification, the structure of a sub-picture is described in the sequence parameter set (SPS) as:

[0038] Table 1

[0039]

[0040]

[0041] The italic text in the SPS above indicates how sub-pictures are defined: Generally speaking, all partitions are based on CTUs (basic units), and sub-pictures can be efficiently defined by specifying the upper left X, Y position plus the width and height information of the sub-picture.

[0042] Using the current sub-image design, Figure 1 Partitioning of sub-pictures in (each color block is associated with one sub-picture) can be supported. Figure 1 The processing order of all sub-pictures in can be in various ways, the key point is that when a sub-picture is encoded, the entire left and top borders of this sub-picture should be available (for example, either the border pixels belong to other sub-pictures that have been processed (decoded), or the border is a picture boundary).

[0043] The two possible processing orders for these ten sub-pictures are Figure 2A and Figure 2B Given in.

[0044] 2. An Example Embodiment

[0045] In the current VVC specification, the sub-picture structure can be defined in a sequence parameter set (SPS). In the standard text VVCDraft 7 (November 2019), it is also clearly stated that one or more slices can be located inside each sub-picture. For each slice, its bitstream will form a NAL unit and will carry the NAL unit type (name and value). In order to better support the random access function of the VVC standard, a new syntax element called "mixed_irap_nalu_types_in_pic_flag" can be added to the picture parameter set (PPS) as follows (emphasized in italics in Table 2):

[0046] Table 2

[0047]

[0048] And the semantics will be:

[0049] mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture of the referenced PPS has more than one VCL NAL unit, and the VCL NAL units do not have the same nal_unit_type value, and the picture is not an IRAP (Intra Random Access Picture). mixed_nalu_types_in_pic_flag equal to 0 specifies that each picture of the referenced PPS has one or more VCL NAL units, and the VCL NAL units of each picture of the referenced PPS have the same nal_unit_type value.

[0050] The above text (in VVC Draft 7JVET-P2001) is not changed. But the following is added:

[0051] mixed_irap_nalu_types_in_pic_flag equal to 1 specifies that each picture of the referenced PPS has more than one VCL NAL unit and the VCL NAL units do not have the same nal_unit_type value, and the picture has a mixed nal_unit_type in the range of IDR_W_RADL to CRA_NUT (inclusive). mixed_irap_nalu_types_in_pic_flag equal to 0 specifies that each picture of the referenced PPS has one or more VCL NAL units and the VCL NAL units of each picture of the referenced PPS do not have the same nal_unit_type value, and the picture is not an IRAP picture.

[0052] The key point of adding this flag in PPS is to support a way to allow mixed IRAP nalu_type types in pictures in applications. It can be understood that by adding more standard text as bitstream constraints in the VVC specification, a similar goal of supporting mixed IRAP nalu_type in pictures can also be achieved.

[0053] It will be understood that the techniques described in this document can be combined within a video encoder device or a video decoder device to significantly improve the performance of operations for encoding video or decoding video. For example, some video applications (such as virtual reality experiences or games) require real-time (or faster than real-time) encoding or decoding of video to provide a satisfactory user experience. The disclosed techniques improve the performance of such applications by using the picture region-based encoding or decoding techniques described herein. For example, encoding or decoding of less than all portions of a video frame based on the user's point of view allows for selective encoding of only the video being viewed by the user. In addition, the reorganization of picture blocks to create picture regions in a rectangular video frame allows for the use of standard rectangular frame-based video encoding tools (such as motion search, transforms, and quantization).

[0054] Figure 3A An example of a video picture with CTUs, tiles and sub-pictures or slices is shown. In the depicted picture, there are 18 tiles, 24 sub-pictures and 24 slices (20x12=240 CTUs).

[0055] 3. Additional Information for Illustrating Embodiments Using Current VVC Technology

[0056] In addition to the syntax examples of Table 1 and Table 2 described previously, the following text may be added to the current version of the VVC codec to implement some embodiments.

[0057] In JVET-P2001 (VVC Draft 7), there is a design for indicating mixed NAL unit types for video pictures, which depends on a syntax element named "mixed_nalu_types_in_pic_flag" in the picture parameter set (PPS). The current proposal suggests adding another syntax element as "mixed_irap_nalu_types_in_pic_flag" to indicate mixed irap nalu types and only irap nalu types in video pictures. The added indication will support a wider range of applications, especially those involving sub-picture partitioning. The added syntax element will also serve the purpose of flexible random access point indication, which indicates fairly independently encoded sub-pictures in video pictures.

[0058] 4. Introduction and Problem Statement

[0059] 4.1 Related existing designs for NAL unit types in VVC specification JVET-P2001:

[0060] The following information is extracted from JVET-P2001 [1], with key information highlighted by underline:

[0061] Table 3

[0062] In Table 5 VVC-specific - NAL unit type code and NAL unit type class

[0063]

[0064]

[0065] Regarding random access conditions, some key terms are defined in the VVC specification. In the JVET-P2001 specification, under 3 "Definitions":

[0066] Clean Random Access (CRA) picture: An IRAP picture with nal_unit_type equal to CRA NUT for each of its VCL NAL units.

[0067] NOTE—A CRA picture does not reference any picture other than itself for inter prediction during its decoding process and may be the first picture in the bitstream in decoding order, or may appear later in the bitstream. A CRA picture may have associated RADL or RASL pictures. When a CRA picture has NoIncorrectPicOutputFlag equal to 1, the associated RASL pictures are not output by the decoder as they may not be decodable because they may contain references to pictures that are not present in the bitstream.

[0068] 3.54 Progressive Decoding Refresh (GDR) picture: A picture has nal_unit_type equal to GDR NUT for each of its VCL NAL units.

[0069] Instantaneous Decoding Refresh (IDR) picture: An IRAP picture has nal_unit_type equal to IDR W RADL or IDR N LP for each of its VCL NAL units.

[0070] NOTE—An IDR picture does not reference any picture other than itself for inter prediction during its decoding, and may be the first picture in the bitstream in decoding order, or may appear later in the bitstream. Each IDR picture is the first picture of a CVS in decoding order. When an IDR picture has nal_unit_type equal to IDR WRADL for each of its VCL NAL units, it may have associated RADL pictures. When an IDR picture has nal_unit_type equal to IDR N LP for each of its VCL NAL units, it does not have any associated leading pictures. An IDR picture has no associated RASL pictures.

[0071] Intra random access point (IRAP) picture: A coded picture for which all VCL NAL units have the same nal_unit_type value in the range IDR W RADL to CRA NUT, inclusive.

[0072] NOTE 1—An IRAP picture does not reference any picture other than itself for inter prediction during its decoding process and can be either a CRA picture or an IDR picture. The first picture in the bitstream in decoding order must be an IRAP or GDR picture. If the necessary parameter sets are available when referenced, an IRAP picture and all subsequent non-RASL pictures in a CVS can be correctly decoded in decoding order without performing the decoding process of any pictures that precede the IRAP picture in decoding order.

[0073] NOTE 2—The value of mixed_nalu_types_in_pic_flag for an IRAP picture is equal to 0. When mixed_nalu_types_in_pic_flag for a picture is equal to 0, and any slice of the picture has a nal_unit_type in the range IDR W RADL to CRANUT, inclusive, all other slices of the picture have the same nal_unit_type value, and the picture is known to be an IRAP picture.

[0074] For indication of mixed types of NAL_unit_type for a picture, a flag (syntax element) 'mixed_nalu_types_in_pic_flag' has been located in the PPS (Picture Parameter Set) as shown in Table 4.

[0075] Table 4

[0076]

[0077] In the VVC specification, under 7.4.3.4 Semantics:

[0078] mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture of the referenced PPS has more than one VCL NAL unit and the VCL NAL units do not have the same nal_unit_type value and the picture is not an IRAP picture. mixed_nalu_types_in_pic_flag equal to 0 specifies that each picture of the referenced PPS has one or more VCL NAL units and the VCL NAL units of each picture of the referenced PPS have the same nal_unit_type value.

[0079] When no_mixed_nalu_types_in_pic_constraint_flag is equal to 1, the value of mixed_nalu_types_in_pic_flag shall be equal to 0.

[0080] For each slice in picture picA having a nal_unit_type value nalUnitTypeA in the range IDR_W_RADL to CRA_NUT (inclusive), picture picA also contains one or more slices having another nal_unit_type value (i.e. the value of mixed_nalu_types_in_pic_flag for picture picA is equal to 1), the following applies:

[0081] - The slice shall belong to the sub-picture subpicA for which the value of subpic_treatment_as_pic_flag[i] is equal to 1.

[0082] - A slice shall not belong to a sub-picture of picA that contains VCL NAL units with nal_unit_type not equal to nalUnitTypeA.

[0083] - For all subsequent PUs in the CLVS in decoding order, neither RefPicList[0] nor RefPicList[1] of the slice in subpicA shall include in the active entry any picture that precedes picA in decoding order.

[0084] Logo no_mixed_nalu_types_in_pic_constraint_flag " is already in general_constraint_info(), as follows:

[0085] Table 5

[0086]

[0087] Related semantics:

[0088] no_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that it is a requirement for bitstream conformance that mixed_nalu_types_in_pic_flag shall be equal to 0. no_mixed_nalu_types_in_pic_constraint_flag equal to 0 does not impose such a constraint.

[0089] In the VVC spec under 7.4.2.2 "NAL unit header semantics" it states:

[0090] For any particular picture's VCL NAL unit, the following applies:

[0091] - If mixed_nalu_types_in_pic_flag is equal to 0, the nal_unit_type value shall be the same for all coded slice NAL units of a picture. A picture or PU is said to have the same NAL unit type as the coded slice NAL units of the picture or PU.

[0092] - Otherwise (mixed_nalu_types_in_pic_flag is equal to 1), one or more VCL NAL units shall have a specific_unit_type value in the range of IDR W RADL to CRA NUT (inclusive), and other VCL NAL units shall have a specific nal_unit_type value in the range of TRAIL_NUT to RSV_VCL_6 (inclusive) or equal to GRA_NUT.

[0093] Note that there is a typo in the previous paragraph. The word "GRA_NUT" is incorrect; e.g. in Table 5 above, there is no entry for "GRA_NUT". A fix has been provided in JVET-Q004 for the current VVC specification: replace GRA_NUT with GDR_NUT. The typo is highlighted in bold underline.

[0094] 4.2 Using VVC specification JVET-P2001 to process sub-images:

[0095] exist Figure 3B In the example video picture shown, there are 18 tiles, 24 sub-pictures and 24 slices (20x12=240 CTUs).

[0096] In one potential use case, we use the above " Figure 7 " to encode each picture, we only pick to display sub-pictures 5, 6, 9, 10 (rectangular area) to start. At a random access point on the timeline, we want to do a scene change (cut) to display sub-pictures 6, 7, 10, 11 instead of the original 5, 6, 9, 10. If we follow the exact constraints in the current VVC specification (JVET-P2001), at that random access point we would mark all sub-pictures from 1 to 24 in the above figure to have the exact same IRAP NAL unit type (e.g., IDR_N_LP for all 24 sub-pictures or CRA_NUT for all 24 sub-pictures). If we want the case of having mixed NAL unit types, then we cannot guarantee that we have only IRAP sub-pictures / pictures.

[0097] 5. Proposed Solution

[0098] It is recommended to add another flag "mixed_irap_nalu_types_in_pic_flag" to PPS, as shown in Table 2:

[0099] The unchanged semantics are:

[0100] mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture that references a PPS has more than one VCL NAL unit and the VCL NAL units do not have the same nal_unit_type value and the picture is not an IRAP picture. mixed_nalu_types_in_pic_flag equal to 0 specifies that each picture that references a PPS has one or more VCL NAL units and the VCL NAL units of each picture that references a PPS have the same nal_unit_type value.

[0101] The new semantics added are:

[0102] mixed_irap_nalu_types_in_pic_flag equal to 1 specifies that each picture of the referenced PPS has more than one VCL NAL unit and the VCL NAL units do not have the same nal_unit_type value, and the picture is in the range of IDR_W_RADL to CRA_NUT (inclusive). mixed_irap_nalu_types_in_pic_flag equal to 0 specifies that each picture of the referenced PPS has one or more VCL NAL units and the VCL NAL units of each picture of the referenced PPS do not have the same nal_unit_type value, and the picture is not an IRAP picture.

[0103] With the addition of “mixed_irap_nalu_types_in_pic_flag”, the VVC specification may provide improved support for mixed random access NAL unit type indication.

[0104] For application standards, such as the DVB standard, the added flag will provide a more flexible indication mechanism for random access indication support.

[0105] Figure 4 is a diagram illustrating a first example device including at least an example video encoder or a picture encoder.

[0106] The acquisition unit 1001 captures videos and pictures. The acquisition unit 1001 may be equipped with one or more cameras for taking videos or pictures of natural scenes. Optionally, the acquisition unit 1001 may be implemented using a camera to obtain a depth video or depth picture. Optionally, the acquisition unit 1001 may include components of an infrared camera. Optionally, the acquisition unit 1001 may be configured with a remote sensing camera. The acquisition unit 1001 may also be a device or apparatus that generates a video or picture by scanning an object using radiation.

[0107] Optionally, the acquisition unit 1001 can perform preprocessing on the video or picture, such as automatic white balance, automatic focus, automatic exposure, backlight compensation, sharpening, denoising, stitching, upsampling / downsampling, frame rate conversion, virtual view synthesis, etc.

[0108] The acquisition unit 1001 may also receive a video or picture from another device or processing unit. For example, the acquisition unit 1001 may be a component unit in a transcoder. The transcoder feeds one or more decoded (or partially decoded) pictures to the acquisition unit 1001. Another example is that the acquisition unit 1001 acquires a video or picture from another device via a data link to the device.

[0109] Note that the acquisition unit 1001 can be used to capture other media information besides videos and pictures, such as audio signals. The acquisition unit 1001 can also receive artificial information, such as characters, texts, videos or pictures generated by computers, and the like.

[0110] Encoder 1002 is an implementation of an example encoder. The input of encoder 1002 is the video or picture output by acquisition unit 1001. Encoder 1002 encodes the video or picture and outputs the generated video or picture bitstream.

[0111] The storage / transmission unit 1003 receives a video or picture bitstream from the encoder 1002 and performs system layer processing on the bitstream. For example, the storage / transmission unit 1003 encapsulates the bitstream according to the transmission standard and the media file format (e.g., MPEG-2TS, ISOBMFF, DASH, MMT, etc.). The storage / transmission unit 1003 stores the transport stream or media file obtained after encapsulation in the memory or disk of the first example device, or transmits the transport stream or media file via a wired or wireless network.

[0112] Note that in addition to the video or picture bitstream from the encoder 1002, the input of the storage / transmission unit 1003 may also include audio, text, images, graphics, etc. The storage / transmission unit 1003 generates a transport or media file by encapsulating these different types of media bitstreams.

[0113] The first example device described in this embodiment may be a device capable of generating or processing a video (or picture) bit stream in a video communication application, such as a mobile phone, a computer, a media server, a portable mobile terminal, a digital camera, a broadcasting device, a CDN (content distribution network) device, a surveillance camera, a video conferencing device, etc.

[0114] Figure 5 is a diagram illustrating a second example device including at least an example video decoder or a picture decoder.

[0115] The receiving unit 1101 receives a video or picture bit stream by obtaining the bit stream from a wired or wireless network, by reading a memory or a disk in an electronic device, or by acquiring data from other devices via a data link.

[0116] The input of the receiving unit 1101 may also include a transport stream or a media file containing a video or picture bitstream. The receiving unit 1101 extracts the video or picture bitstream from the transport stream or the media file according to the specification of the transport or media file format.

[0117] The receiving unit 1101 outputs the video or picture bitstream and passes it to the decoder 1102. Note that in addition to the video or picture bitstream, the output of the receiving unit 1101 may also include an audio bitstream, characters, text, images, graphics, etc. The receiving unit 1101 passes the output to the corresponding processing unit in the second example device. For example, the receiving unit 1101 passes the output audio bitstream to the audio decoder in the device.

[0118] The decoder 1102 is an implementation of an example decoder. The input of the encoder 1102 is the video or picture bit stream output by the receiving unit 1101. The decoder 1102 decodes the video or picture bit stream and outputs the decoded video or picture.

[0119] The rendering unit 1103 receives the decoded video or picture from the decoder 1102. The rendering unit 1103 presents the decoded video or picture to the viewer. The rendering unit 1103 may be a component of the second example device, such as a screen. The rendering unit 1103 may also be a device separated from the second example device and having a data link to the second example device, such as a projector, a monitor, a television, etc. Optionally, the rendering 1103 post-processes the decoded video or picture before presenting it to the viewer, such as automatic white balance, automatic focus, automatic exposure, backlight compensation, sharpening, denoising, splicing, upsampling / downsampling, frame rate conversion, virtual view synthesis, etc.

[0120] Note that in addition to the decoded video or picture, the input of the rendering unit 1103 can be other media data from one or more units of the second example device, such as audio, characters, text, images, graphics, etc. The input of the rendering unit 1103 can also include artificial data, such as lines and marks drawn on the slide by the local teacher to attract attention in the distance education application. The rendering unit 1103 combines different types of media and then presents the combination to the viewer.

[0121] The second example device described in this embodiment can be a device that can decode or process a video (or picture) bit stream in a video communication application, such as a mobile phone, a computer, a set-top box, a television, an HMD, a monitor, a media server, a portable mobile terminal, a digital camera, a broadcasting device, a CDN (content distribution network) device, monitoring, video conferencing equipment, etc.

[0122] Figure 6 The diagram contains Figure 5 The first example device and Figure 4 A diagram of an electronic system of a second example device in FIG.

[0123] Service device 1201 is Figure 4 The first example device in.

[0124] The storage medium / transmission network 1202 may include internal memory resources of a device or electronic system, external memory resources accessible via a data link, and a data transmission network including a wired and / or wireless network. The storage medium / transmission network 1202 provides storage resources or a data transmission network for the storage / transmission unit 1203 in the service device 1201.

[0125] The destination device 1203 is Figure 5 The receiving unit 1201 in the destination device 1203 receives a video or picture bitstream, a transport stream containing the video or picture bitstream, or a media file containing the video or picture bitstream from a storage medium / transmission network 1202 .

[0126] The electronic system described in this embodiment can be a device or system that can generate, store or transmit and decode video (or picture) bit streams in video communication applications, such as mobile phones, computers, IPTV systems, OTT systems, Internet multimedia systems, digital TV broadcasting systems, video surveillance systems, portable mobile terminals, digital cameras, video conferencing systems, etc.

[0127] Figure 7An example apparatus 1400 is shown that can be used to implement the encoder-side or decoder-side techniques described in this document. The apparatus 1400 includes a processor 1402, which can be configured to perform the encoder-side or decoder-side techniques or both. The apparatus 1400 may also include a memory (not shown) for storing processor-executable instructions and for storing video bitstreams and / or display data. The apparatus 1400 may include video processing circuitry (not shown), such as transform circuits, arithmetic encoding / decoding circuits, data encoding techniques based on lookup tables, and the like. The video processing circuitry may be partially included in the processor and / or partially included in other dedicated circuitry, such as a graphics processor, a field programmable gate array (FPGA), and the like.

[0128] Figure 8 800 is a flowchart of an example method 800 for video bitstream processing. The method 800 may be implemented by a video decoder described in this document. The method 800 (and 900 described below) may be implemented using a hardware platform (such as a reference Figure 7 described) to achieve this.

[0129] Method 800 includes parsing (802) a header at a video unit level in a video bitstream for a first field indicating whether the video unit includes a plurality of network abstraction layer bit units having a plurality of types. In some embodiments, the video unit may be a video picture (e.g., as shown in FIG. 3 ).

[0130] The method 800 includes further parsing (804) a second field in the header due to determining that the video unit includes a plurality of network abstraction layer bit units having a plurality of types, the second field indicating whether the video unit includes an intra random access point.

[0131] The method 800 includes decoding ( 806 ) a video bitstream based on the first field and the second field to generate a video.

[0132] In some embodiments, both the first field and the second field may be single bit fields. Such an embodiment has the advantage of maintaining backward compatibility with legacy standards while introducing a minimal amount of overhead (single bit) when introducing the additional information conveyed by the second bit.

[0133] In some embodiments, the first bit and the second bit may be included in different positions within the header. Alternatively, in some embodiments, the two bits may be continuous with each other. In some cases, whether the second field is included in the header may depend on the value of the first field. For example, a zero value in the first field may indicate that the second field is not included.

[0134] Fig. 9 A flow chart of another example method 900 for video bitstream parsing is shown.

[0135] The method 900 includes, at 902, for decoding a video picture including one or more sub-pictures, parsing a syntax structure to obtain type information about one or more network abstraction layer (NAL) units constituting the video picture and whether the video picture is an intra random access picture.

[0136] The method 900 includes, at 904, reconstructing one or more sub-pictures based on the parsing.

[0137] In some embodiments, the syntax structure may include a first field indicating whether the one or more NAL units include multiple NAL units.

[0138] In some embodiments, the syntax structure may include a first field and an optional second field, the presence of the second field depends on the value of the first field, and wherein the first field indicates whether a mixed type of NAL unit is present in one or more NAL units. For example, as previously described, the first field and the second field may be single bit fields positioned consecutively within the bitstream.

[0139] Fig.10 An example flow chart 1000 is shown that can be used to signal or infer the presence of a sub-picture in a video picture. For example, such a sub-picture can be used to convey 3D picture information.

[0140] At 1002, a check is made regarding the value of a first flag. The value may be set to a specific value, such as 0, to indicate that the picture includes all NAL slices of the same slice type (1004). For example, the flag may be used to indicate that the current picture is a "legacy" picture that does not include independently decodable sub-pictures. After determining that all NAL slices are of the same type (1004), the header may be further parsed (1006) for additional information.

[0141] If the check at 1002 shows that the first flag has a second value (e.g., 1), it is inferred that there are multiple NAL units and that these NAL units may have different values. At 1010, a check can be performed on the value of the second flag. If the value is a specific value (e.g., 0), it is determined that all NAL units have the same value (1012). After the determination, the header can be parsed for additional information (1006). If the check at 1010 shows that the second flag has another specific value, it is inferred that the NAL unit has a different type, but the picture is not an intra random access picture (1014).

[0142] from Fig.10As can be understood from the logic flow depicted in FIG, a single bit is introduced into the existing syntax structure of the picture header to realize the signaling of the existence of a sub-picture in a video picture. The signaling is compatible with the signaling of the currently existing slices and / or tiles and / or CTUs.

[0143] In some embodiments, a method of encoding video (e.g., Fig.11 The method 1100 described in the foregoing includes: including (1102) in a header at a video unit level in a video bitstream:

[0144] A first field indicating whether the video unit includes a plurality of network abstraction layer bit units having a plurality of types; selectively including (1104) a second field in the header due to determining that the video unit includes a plurality of network abstraction layer bit units having a plurality of types, the second field indicating whether the video unit includes an intra-frame random access point; and encoding (1106) the video to generate a video bitstream. For example, the encoding may use various intra-frame, inter-frame, and other techniques to represent the video data in the bitstream.

[0145] In some embodiments, the video unit is a video picture.

[0146] As previously mentioned, in some embodiments, the first field and the second field are single bit fields. However, in this document, other lengths of these fields are also possible.

[0147] The second field may be positioned in the video bitstream immediately following the first field.

[0148] In some embodiments, encoding 1106 includes encoding a plurality of sub-pictures into a video unit, and wherein each sub-picture has a separate network abstraction layer unit type.

[0149] In some embodiments, a video encoding method may include generating a syntax structure for encoding a video picture including one or more sub-pictures, the syntax structure including type information about one or more network abstraction layer (NAL) units constituting the video picture and whether the video picture is an intra-frame random access picture; and encoding the one or more sub-pictures based on the syntax structure.

[0150] In the encoding method disclosed above, in some embodiments, the syntax structure may include a first field indicating whether one or more NAL units may include multiple NAL units.

[0151] In some embodiments, the syntax structure includes a first field and an optional second field, the presence of the second field depends on the value of the first field, and wherein the first field indicates whether a NAL unit of a mixed type is present in one or more NAL units.

[0152] In some embodiments, the first field and the second field are single bit flags located next to each other in the video bitstream.

[0153] In some embodiments, for example, Figure 7 The video encoder described in can implement the above encoding method. In some embodiments, for example, Figure 7 The video decoder depicted in can implement the video bitstream parsing method described herein. The video decoder can be, for example, a transcoder that changes a video from one bitstream representation to another bitstream representation.

[0154] In some embodiments, a computer program product may include a computer readable medium including processor executable code for implementing the methods described herein and in the claims.

[0155] 6. Industrial Applicability

[0156] As can be seen from the above description, a technique for signaling a sub-picture that is not necessarily an intra-frame random access point (such as a sub-picture containing different view data for a 3D picture) is disclosed. Using the disclosed technique, some embodiments of a video encoder or video decoder can use a picture header with a minimum number of bits added to a syntax structure to support the transmission of sub-pictures while maintaining backward compatibility with signaling of slices, CTUs, etc.

[0157] The disclosed and other embodiments, modules and functional operations described in this document can be implemented in digital electronic circuit systems or computer software, firmware or hardware, including the structures disclosed in this document and their structural equivalents, or a combination of one or more thereof. The disclosed and other embodiments can be implemented as one or more computer program products, that is, one or more computer program instruction modules encoded on a computer-readable medium, for execution by a data processing device or controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagation signal, or one or more combinations thereof. The term "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagation signal is an artificially generated signal (e.g., a machine-generated electrical, optical or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device.

[0158] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0159] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0160] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to receive data from or transfer data to or both of one or more mass storage devices (e.g., magnetic, magneto-optical disks, or optical disks) for storing data. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in a dedicated logic circuit system.

[0161] Although this patent document contains many details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. In addition, although features may be described as working in certain combinations and even initially claimed as such, in some cases one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of a subcombination.

[0162] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed to achieve the desired results. In addition, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0163] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and shown in this patent document.

Claims

1. A method for processing a video bit stream, comprising: Parsing, at a video picture level in a video bitstream, a picture parameter set for a first field indicating whether the video picture includes a plurality of network abstraction layer units having a plurality of types; due to determining that the video picture includes a plurality of network abstraction layer units having a plurality of types, parsing a second field in the picture parameter set, the second field indicating whether the plurality of network abstraction layer units have mixed network abstraction layer unit types in a range of IDR_W_RADL to CRA_NUT; as well as Based on the first field and the second field, the video bitstream is decoded to generate a video. 2 . The method of claim 1 , wherein the first field and the second field are single bit fields. 3 . The method of claim 1 , wherein the second field is positioned immediately after the first field in the video bitstream.

4. The method of claim 1, wherein the decoding the video bitstream comprises: A plurality of sub-pictures included in the video picture are decoded, wherein each sub-picture has an independent network abstraction layer unit type.

5. A method for encoding a video, comprising: Including a first field in a picture parameter set at a video picture level in a video bitstream, the first field indicating whether the video picture includes a plurality of network abstraction layer units having a plurality of types; Due to determining that the video picture includes a plurality of network abstraction layer units having a plurality of types, further including a second field in the picture parameter set, the second field indicating whether the plurality of network abstraction layer units have mixed network abstraction layer unit types in the range of IDR_W_RADL to CRA_NUT; and The video is encoded to generate the video bitstream. The method of claim 5 , wherein the first field and the second field are single bit fields.

7. The method of claim 5, wherein the second field is positioned immediately after the first field in the video bitstream.

8. The method of claim 5, wherein the encoding comprises: A plurality of sub-pictures are encoded into the video picture, and each sub-picture has an independent network abstraction layer unit type.

9. A video decoding apparatus, comprising a processor, wherein the processor is configured to execute a method for processing a video bitstream, the method comprising: for decoding a video picture including one or more sub-pictures, parsing a syntax structure to obtain type information about one or more network abstraction layer units constituting the video picture and whether the video picture has a hybrid network abstraction layer unit type in the range of IDR_W_RADL to CRA_NUT; as well as Based on the parsing, the one or more sub-pictures are reconstructed.

10. The apparatus of claim 9, wherein the syntax structure comprises a first field indicating whether the one or more network abstraction layer units comprise a plurality of network abstraction layer units.

11. The apparatus of claim 9, wherein the syntax structure comprises a first field and an optional second field, the presence of the second field depending on the value of the first field, and wherein the second field indicates whether a mixed type of network abstraction layer unit exists when the first field indicates that the one or more network abstraction layer units include multiple network abstraction layer units.

12. The apparatus of claim 11, wherein the first field and the second field are single-bit flags located immediately adjacent to each other in the video bitstream.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the processor causes the processor to implement a video parsing method, wherein the video parsing method comprises: Parsing, at a video picture level in a video bitstream, a picture parameter set for a first field indicating whether the video picture includes a plurality of network abstraction layer units having a plurality of types; due to determining that the video picture includes a plurality of network abstraction layer units having a plurality of types, parsing a second field in the picture parameter set, the second field indicating whether the plurality of network abstraction layer units have mixed network abstraction layer unit types in a range of IDR_W_RADL to CRA_NUT; as well as Based on the first field and the second field, the video bitstream is decoded to generate a video.

14. The computer program product of claim 13, wherein the first field and the second field are single bit fields.

15. The computer program product of claim 13, wherein the second field is located immediately after the first field in the video bitstream.

16. The computer program product of claim 13, wherein the decoding the video bitstream comprises: A plurality of sub-pictures included in the video picture are decoded, wherein each sub-picture has an independent network abstraction layer unit type.

Images