Video encoder, video decoder, and corresponding method

Abstract

To provide a method of decoding a coded video bitstream implemented by a video decoder.SOLUTION: A method includes the steps of causing a video decoder receiver to receive a mixed intra random access point (IRAP) picture including a first sub-picture and a second sub-picture, in which the first sub-picture is an IRAP picture and the second sub-picture is a non-IRAP sub-picture, causing the receiver to receive a reference picture list (RPL) of mixed IRAP pictures, causing a video decoder processor to decode the second sub-picture using the RPL, and causing the processor to generate an image on the basis of the decoded second sub-picture.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] Generally, this disclosure describes techniques for supporting sub-picture based random access in video coding. More specifically, this disclosure describes techniques for allowing a picture to include one or more regions identified as intra random access points and, at the same time, remaining regions identified as non-intra random access points. [Background technology]

[0002] The amount of video data required to render even a relatively short video can be substantial, which can create difficulties when streaming or otherwise communicating the data over communication networks with limited bandwidth capacity. Thus, video data is typically compressed before being communicated over modern telecommunications networks. The size of the video can also be an issue when storing the video on a storage device, since memory resources may be limited. Video compression devices often use software and / or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data required to represent a digital video image. The compressed data is then received by a video decompressor, which decodes the video data at the destination. Due to limited network resources and an ever-increasing demand for higher video quality, improved compression and decompression techniques are desired that increase compression ratios with little or no sacrifice in image quality. Summary of the Invention

[0003] A first aspect relates to a method for decoding a coded video bitstream implemented by a video decoder, the method including: a receiver of the video decoder receiving a mixed Intra Random Access Point (IRAP) picture including a first sub-picture and a second sub-picture, where the first sub-picture is an IRAP picture and the second sub-picture is a non-IRAP sub-picture; the receiver receiving a Reference Picture List (RPL) of the mixed IRAP picture; a processor of the video decoder decoding the second sub-picture using the RPL; and the processor generating an image based on the decoded second sub-picture.

[0004] Conventional coding techniques do not allow IRAP pictures to refer to and utilize the RPL, but the techniques disclosed herein allow IRAP pictures, especially mixed IRAP pictures, to refer to and utilize the RPL. Thus, even if a mixed IRAP picture includes an IRAP sub-picture, the codec is allowed to refer to and utilize the RPL to code the non-IRAP sub-picture. This is particularly beneficial for VR coding applications. Thus, the video coding coder / decoder (also known as "codec") is improved compared to current codecs. In effect, the improved video coding process provides users with a better user experience when transmitting, receiving, and / or displaying video.

[0005] In a first implementation of the method according to the first aspect itself, the mixed IRAP picture is received in a split bitstream comprising a first sub-bitstream and a second sub-bitstream.

[0006] In a second embodiment of the method according to the first aspect itself, or any preceding embodiment of the first aspect, the first sub-picture is placed in the first sub-bitstream and the second sub-picture is placed in the second sub-bitstream.

[0007] In a third embodiment of the method according to the first aspect itself or any preceding embodiment of the first aspect, the IRAP picture is an Instantaneous Decoder Refresh (IDR) picture.

[0008] In a fourth embodiment of the method according to the first aspect itself or any preceding embodiment of the first aspect, the first sub-picture is an IRAP picture contained within a first Network Abstraction Layer (NAL) unit and the second sub-picture is a non-IRAP sub-picture contained within the second NAL unit.

[0009] In a fifth embodiment of the method according to the first aspect itself, or any preceding embodiment of the first aspect, the method further comprises a step of receiving a flag in the bitstream, the flag indicating whether the bitstream includes any mixed IRAP pictures.

[0010] In a sixth embodiment of the method according to the first aspect itself or any preceding embodiment of the first aspect, the flag is in a Sequence Parameter Set (SPS) of the bitstream.

[0011] In a seventh embodiment of the method according to the first aspect itself or any preceding embodiment of the first aspect, the flag is designated as sps_mixed_tile_groups_in_pic_flag.

[0012] A second aspect relates to a method for encoding a video bitstream implemented by a video encoder, the method including: a processor of the video encoder encoding a mixed intra random access point (IRAP) picture including a first sub-picture and a second sub-picture, where the first sub-picture is an IRAP picture and the second sub-picture is a non-IRAP sub-picture; the processor encoding a reference picture list (RPL) of the mixed IRAP picture; the processor generating a bitstream including the mixed IRAP picture and an RPL corresponding to the mixed IRAP picture; and storing the bitstream in a memory of the video encoder for transmission to a video decoder.

[0013] Conventional coding techniques do not allow IRAP pictures to refer to and utilize the RPL, but the techniques disclosed herein allow IRAP pictures, especially mixed IRAP pictures, to refer to and utilize the RPL. Thus, even if a mixed IRAP picture includes an IRAP sub-picture, the codec is allowed to refer to and utilize the RPL to code the non-IRAP sub-picture. This is particularly beneficial for VR coding applications. Thus, the video coding coder / decoder (also known as "codec") is improved compared to current codecs. In effect, the improved video coding process provides users with a better user experience when transmitting, receiving, and / or displaying video.

[0014] In a first implementation of the method according to the second aspect itself, a mixed IRAP picture is encoded into a split bitstream comprising a first sub-bitstream and a second sub-bitstream.

[0015] In a second embodiment of the method according to the second aspect itself, or any preceding embodiment of the second aspect, the first sub-picture is encoded into a first sub-bitstream and the second sub-picture is encoded into a second sub-bitstream.

[0016] In a third embodiment of the method according to the second aspect itself or any preceding embodiment of the second aspect, the IRAP picture is an Instantaneous Decoder Refresh (IDR) picture.

[0017] In a fourth embodiment of the method according to the second aspect itself, or in any preceding embodiment of the second aspect, the first sub-picture is an IRAP picture contained within a first Network Abstraction Layer (NAL) unit, and the second sub-picture is a non-IRAP sub-picture contained within the second NAL unit.

[0018] In a fifth embodiment of the method according to the second aspect itself, or any preceding embodiment of the second aspect, the method further comprises a step of encoding a flag in the bitstream, the flag indicating whether the bitstream includes any mixed IRAP pictures.

[0019] In a sixth embodiment of the method according to the second aspect itself or any preceding embodiment of the second aspect, the flag is in a Sequence Parameter Set (SPS) of the bitstream.

[0020] In a seventh embodiment of the method according to the second aspect itself or any preceding implementation of the second aspect, the flag is designated as sps_mixed_tile_groups_in_pic_flag.

[0021] A third aspect relates to a decoding device, the decoding device including: a receiver; a memory coupled to the receiver; and a processor coupled to the memory, the receiver configured to receive a coded video bitstream including a mixed Intra Random Access Point (IRAP) picture including a first sub-picture and a second sub-picture, and a Reference Picture List (RPL) of the mixed IRAP picture, the first sub-picture being an IRAP picture and the second sub-picture being a non-IRAP sub-picture, the memory storing instructions, and the processor configured to execute the instructions to cause the decoding device to decode the second sub-picture using the RPL and generate an image based on the decoded second sub-picture.

[0022] Conventional coding techniques do not allow IRAP pictures to refer to and use the RPL, but the decoding device disclosed in this specification allows IRAP pictures, especially mixed IRAP pictures, to refer to and use the RPL. Thus, even if a mixed IRAP picture includes an IRAP sub-picture, the codec is allowed to refer to and use the RPL to code the non-IRAP sub-picture. This is particularly beneficial in VR coding applications. Therefore, the video coding coder / decoder (also known as "codec") is improved compared to current codecs. In practice, the improved video coding process provides users with a better user experience when transmitting, receiving, and / or displaying video.

[0023] In a first implementation of the decoding device according to the third aspect itself, the decoding device further comprises a display configured to display the image.

[0024] In a second embodiment of the decoding device according to the third aspect itself, or any preceding embodiment of the third aspect, the receiver is configured to receive a mixed IRAP picture in a split bitstream comprising a first sub-bitstream and a second sub-bitstream.

[0025] In a third embodiment of the decoding device according to the third aspect itself, or any preceding embodiment of the third aspect, the first sub-picture is placed in the first sub-bitstream and the second sub-picture is placed in the second sub-bitstream.

[0026] In a fourth embodiment of the decoding device according to the third aspect itself or in any preceding embodiment of the third aspect, the IRAP picture is an Instantaneous Decoder Refresh (IDR) picture.

[0027] A fourth aspect relates to an encoding device including a memory including instructions and a processor coupled to the memory, the processor being configured to execute the instructions to cause the encoding device to: encode a mixed intra random access point (IRAP) picture including a first sub-picture and a second sub-picture, where the first sub-picture is an IRAP picture and the second sub-picture is a non-IRAP sub-picture; encode a reference picture list (RPL) of the mixed IRAP picture; generate a bitstream including the mixed IRAP picture and an RPL corresponding to the mixed IRAP picture; and store the bitstream in the memory for transmission to a video decoder.

[0028] Conventional coding techniques do not allow IRAP pictures to refer to and use the RPL, but the coding device disclosed in this specification allows IRAP pictures, especially mixed IRAP pictures, to refer to and use the RPL. Thus, even if a mixed IRAP picture includes an IRAP sub-picture, the codec is allowed to refer to and use the RPL to code the non-IRAP sub-picture. This is particularly beneficial for VR coding applications. Therefore, the video coding coder / decoder (also known as "codec") is improved compared to current codecs. In practice, the improved video coding process provides users with a better user experience when transmitting, receiving, and / or displaying video.

[0029] In a first implementation of the encoding device according to the fourth aspect itself, the encoding device further comprises a transmitter coupled to the processor, the transmitter configured to transmit the bitstream towards the video decoder.

[0030] In a second implementation of the encoding device according to the fourth aspect itself, the mixed IRAP picture is encoded into a split bitstream comprising a first sub-bitstream and a second sub-bitstream.

[0031] In a third implementation of the encoding device according to the fourth aspect itself, the first sub-picture is encoded with a first sub-bitstream and the second sub-picture is encoded with a second sub-bitstream.

[0032] In a fourth implementation of the encoding device according to the fourth aspect itself, the IRAP picture is an Instantaneous Decoder Refresh (IDR) picture.

[0033] A fifth aspect relates to a coding device, comprising: a receiver configured to receive pictures to be encoded or to receive a bitstream to be decoded; a transmitter coupled to the receiver, configured to transmit the bitstream to a decoder or to transmit the decoded images to a display; a memory coupled to at least one of the receiver or the transmitter, configured to store instructions; and a processor coupled to the memory, configured to execute the instructions stored in the memory to perform the methods described herein.

[0034] Conventional coding techniques do not allow IRAP pictures to refer to and utilize the RPL, but the coding device disclosed herein allows IRAP pictures, especially mixed IRAP pictures, to refer to and utilize the RPL. Thus, even if a mixed IRAP picture includes an IRAP sub-picture, the codec is allowed to refer to and utilize the RPL to code the non-IRAP sub-picture. This is particularly beneficial in VR coding applications. Thus, the video coding coder / decoder (also known as "codec") is improved compared to current codecs. In effect, the improved video coding process provides users with a better user experience when transmitting, receiving, and / or displaying video.

[0035] A sixth aspect relates to a system, the system including an encoder and a decoder in communication with the encoder, the encoder or decoder including a decoding device, an encoding device or a coding apparatus as disclosed herein.

[0036] Conventional coding techniques do not allow IRAP pictures to refer to and utilize the RPL, but the system disclosed herein allows IRAP pictures, especially mixed IRAP pictures, to refer to and utilize the RPL. Thus, even if a mixed IRAP picture includes an IRAP sub-picture, the codec is allowed to refer to and utilize the RPL to code the non-IRAP sub-picture. This is particularly beneficial for VR coding applications. Thus, the video coding coder / decoder (aka "codec") is improved compared to current codecs. In effect, the improved video coding process provides users with a better user experience when transmitting, receiving, and / or displaying video.

[0037] A seventh aspect relates to a means for coding, the means for coding including: receiving means configured to receive a bitstream to be decoded; transmitting means coupled to the receiving means, configured to transmit the decoded image to a display means; storage means coupled to at least one of the receiving means or the transmitting means, configured to store instructions; and processing means coupled to the storage means, configured to execute the instructions stored in the storage means to perform the methods disclosed herein.

[0038] Conventional coding techniques do not allow IRAP pictures to refer to and utilize the RPL, but the coding means disclosed herein allow IRAP pictures, especially mixed IRAP pictures, to refer to and utilize the RPL. Thus, even if a mixed IRAP picture includes an IRAP sub-picture, the codec is allowed to refer to and utilize the RPL to code the non-IRAP sub-picture. This is particularly beneficial for VR coding applications. Thus, the video coding coder / decoder (aka "codec") is improved compared to current codecs. In effect, the improved video coding process provides users with a better user experience when transmitting, receiving, and / or displaying video.

[0039] An eighth aspect relates to a decoding apparatus, the decoding apparatus including: a receiver configured to receive a coded video bitstream; and a processor coupled to the receiver, the processor configured to parse a flag in the coded video bitstream, where a value of the flag indicates that a picture in the coded video bitstream includes a mixed Network Abstraction Layer (NAL) unit type; decode the picture as a non-intra random access point (IRAP) picture based on the value of the flag; and generate an image based on the decoded picture.

[0040] The decoding device disclosed herein enables the decoding device to identify which pictures in a coded video bitstream contain mixed network abstraction layer (NAL) unit types. That is, the decoding device is configured to analyze a flag in the coded video bitstream to identify that a picture contains mixed NAL unit types. Even if a picture contains an IRAP NAL unit type, the picture is decoded as a non-IRAP picture based on the value of the flag.

[0041] In a first implementation of the encoding device according to the eighth aspect itself, if a picture contains an IRAP NAL unit type, the picture is decoded as a non-IRAP picture.

[0042] In a second implementation of the encoding device according to the eighth aspect itself, the value of the flag is one.

[0043] In effect, an improved video coding process provides users with a better user experience when transmitting, receiving and / or displaying video. [Brief description of the drawings]

[0044] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, in which like reference numerals represent like parts. [Figure 1] 1 is a block diagram illustrating an example coding system that can utilize bi-prediction techniques. [Diagram 2] 1 is a block diagram illustrating an example video encoder capable of implementing bi-prediction techniques. [Diagram 3] 1 is a block diagram illustrating an example of a video decoder capable of implementing bi-directional prediction techniques. [Figure 4] FIG. 2 is a schematic diagram illustrating a Reference Picture Set (RPS) with entries for the current picture in all subsets of the RPS. [Diagram 5]FIG. 1 is a schematic diagram of one embodiment of a picture suitable for use in a VR coding application. [Figure 6] FIG. 6 is a schematic diagram of one embodiment of a video bitstream corresponding to the picture of FIG. 5. [Figure 7] 1 is an embodiment of a method for decoding a coded video bitstream. [Figure 8] 1 is an embodiment of a method for encoding a video bitstream. [Figure 9] 1 is a schematic diagram of a video coding device; [Figure 10] FIG. 2 is a schematic diagram of an embodiment of a means for coding; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] The following are various abbreviations used in this specification: Coded Video Sequence (CVS), Decoded Picture Buffer (DPB), Instantaneous Decoding Refresh (IDR), Intra Random Access Point (IRAP), Least Significant Bit (LSB), Most Significant Bit (MSB), Network Abstraction Layer (NAL), Picture Order Count (POC), Raw Byte Sequence Payload (RBSP), Sequence Parameter Set (SPS), and Working Document (WD).

[0046] FIG. 1 is a block diagram illustrating an example coding system 10 that can utilize video coding techniques as described herein. As shown in FIG. 1, coding system 10 includes a source device 12 that provides encoded video data that is subsequently decoded by a destination device 14. In particular, source device 12 can provide video data to destination device 14 via a computer-readable medium 16. Source device 12 and destination device 14 can include any of a wide range of devices, including desktop computers, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" pads, televisions, cameras, displays, digital media players, video game consoles, video streaming devices, and the like. In some cases, source device 12 and destination device 14 can be equipped for wireless communication.

[0047] The destination device 14 may receive the encoded video data to be decoded via a computer-readable medium 16. The computer-readable medium 16 may include any type of medium or device capable of moving the encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable medium 16 may include a communication medium that enables the source device 12 to transmit the encoded video data directly to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14. The communication medium may include any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may help facilitate communication from the source device 12 to the destination device 14.

[0048] In some examples, the encoded data may be output from output interface 22 to a storage device. Similarly, the encoded data may be accessed from a storage device by an input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a digital video disc (DVD), a compact disc read only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, the storage device may correspond to a file server or another intermediate storage device that may store the encoded video generated by source device 12. Destination device 14 may access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing the encoded video data and transmitting the encoded video data to destination device 14. Examples of file servers include a web server (e.g., for a website), a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive. Destination device 14 may access the encoded video data via any standard data connection, including an Internet connection. This may include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., digital subscriber lines (DSL), cable modems, etc.), or a combination of both suitable for accessing the encoded video data stored on a file server. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0049] The techniques of this disclosure are not necessarily limited to wireless applications or settings. The techniques may be applied to video coding in support of any of a variety of multimedia applications, such as terrestrial television broadcast, cable television transmission, satellite television transmission, Internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, coding system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0050] In the example of FIG. 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. According to this disclosure, the video encoder 20 of source device 12 and / or the video decoder 30 of destination device 14 may be configured to apply techniques for video coding. In other examples, the source and destination devices may include other components or configurations. For example, source device 12 may receive video data from an external video source, such as an external camera. Similarly, destination device 14 may interface with an external display device rather than including an integrated display device.

[0051] The illustrated coding system 10 of FIG. 1 is merely an example. The techniques of video coding may be performed by any digital video encoding and / or decoding device. Although the techniques of this disclosure are generally performed by a video coding device, the techniques may also be performed by a video encoder / decoder, typically referred to as a "codec." Additionally, the techniques of this disclosure may also be performed by a video preprocessor. The video encoder and / or decoder may be a graphics processing unit (GPU) or similar device.

[0052] Source device 12 and destination device 14 are merely examples of such coding devices, where source device 12 generates and transmits coded video data to destination device 14. In some examples, source device 12 and destination device 14 may operate substantially symmetrically, such that each of source device and destination device 12, 14 includes video encoding and decoding components. Thus, coding system 10 may support one-way or two-way video transmission between video devices 12, 14, e.g., for video streaming, video playback, video broadcasting, or video telephony.

[0053] Video source 18 of source device 12 may include a video capture device such as a video camera, a video archive containing previously captured video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, video source 18 may generate computer graphics-based data as the source video or a combination of live, archived, and computer-generated video.

[0054] In some cases, where video source 18 is a video camera, source device 12 and destination device 14 may form a so-called camera phone or video telephone. However, as noted above, the techniques described in this disclosure may be applicable to video coding in general, and may be applied to wireless and / or wired applications. In either case, captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output to computer-readable medium 16 by output interface 22.

[0055] Computer-readable medium 16 may include a transitory medium, such as a wireless broadcast or wired network transmission, or a storage medium (i.e., a non-transitory storage medium), such as a hard disk, a flash drive, a compact disc, a digital video disc, a Blu-ray disc, or other computer-readable medium. In some examples, a network server (not shown) may receive the encoded video data from source device 12 and provide the encoded video data to destination device 14, for example, via a network transmission. Similarly, a computing device of a media production facility, such as a disc stamping facility, may receive the encoded video data from source device 12 and generate a disc including the encoded video data. Thus, computer-readable medium 16 may be understood to include one or more computer-readable media of various forms in various examples.

[0056] An input interface 28 of destination device 14 receives information from computer-readable medium 16. The information on computer-readable medium 16 may include syntax information defined by video encoder 20, including syntax elements that describe characteristics and / or processing of blocks and other coded units, such as groups of pictures (GOPs), which is also used by video decoder 30. Display device 32 displays the decoded video data to a user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0057] Video encoder 20 and video decoder 30 may operate according to a video coding standard, such as the High Efficiency Video Coding (HEVC) standard currently under development, and may conform to the HEVC Test Model (HM). Alternatively, video encoder 20 and video decoder 30 may operate according to the International Telecommunications Union Telecommunications Standardization Sector (ITU-T) H.264 standard, or other proprietary or industry standards such as those referred to as Moving Picture Expert Group (MPEG) 4, Part 10, Advanced Video Coding (AVC), H.265 / HEVC, or extensions of such standards. However, the techniques of this disclosure are not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. 1, in some aspects, video encoder 20 and video decoder 30 may be integrated with audio encoders and decoders, respectively, and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units, or other hardware and software, to handle the encoding of both audio and video in a common data stream or separate data streams. Where applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols, such as the User Datagram Protocol (UDP).

[0058] Each of the video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, the device may store instructions of the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (codec) in the respective device. The device including the video encoder 20 and / or the video decoder 30 may include an integrated circuit, a microprocessor, and / or a wireless communication device, such as a mobile phone.

[0059] 2 is a block diagram illustrating an example of a video encoder 20 capable of implementing video coding techniques. Video encoder 20 may perform intra- and inter-coding of video blocks within a video slice. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy of video within a given video frame or picture. Inter-coding relies on temporal prediction to reduce or remove temporal redundancy of video within adjacent frames or pictures of a video sequence. Intra-mode (I-mode) may refer to any of several spatial-based coding modes. Inter-modes, such as uni-directional (aka uni-predictive) prediction (P-mode) or bi-directional (aka bi-predictive) (B-mode), may refer to any of several temporal-based coding modes.

[0060] As shown in FIG. 2, video encoder 20 receives a current video block in a video frame to be encoded. In the example of FIG. 2, video encoder 20 includes a mode selection unit 40, a reference frame memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. Mode selection unit 40 in turn includes a motion compensation unit 44, a motion estimation unit 42, an intra prediction (aka intra prediction) unit 46, and a partition unit 48. For video block reconstruction, video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and a summer 62. A deblocking filter (not shown in FIG. 2) may also be included to filter block boundaries to remove blockiness artifacts from the reconstructed video. If necessary, the deblocking filter would typically filter the output of summer 62. In addition to the deblocking filter, additional filters (in loop or post loop) may be used. If desired, the output of summer 50 can be filtered (as an in-loop filter), although such a filter is not shown for simplicity.

[0061] During the encoding process, video encoder 20 receives a video frame or slice to be encoded. The frame or slice may be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-predictive coding of the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction. Alternatively, intra-prediction unit 46 may perform intra-predictive coding of the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial prediction. Video encoder 20 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0062] Further, partition unit 48 may partition blocks of video data into sub-blocks based on an evaluation of a previous partitioning scheme in a previous coding pass. For example, partition unit 48 may first partition a frame or slice into largest coding units (LCUs) and partition each of the LCUs into sub-coding units (sub-CUs) based on a rate-distortion analysis (e.g., rate-distortion optimization). Mode selection unit 40 may further generate a quadtree data structure indicating the partitioning of the LCUs into sub-CUs. A leaf-node CU of the quadtree may include one or more prediction units (PUs) and one or more transform units (TUs).

[0063] This disclosure uses the term "block" to refer to either a CU, PU, ​​or TU in the context of HEVC, or similar data structures in the context of other standards (e.g., macroblocks and subblocks thereof in H.264 / AVC). A CU includes a coding node, a PU, and a TU associated with the coding node. The size of a CU corresponds to the size of the coding node and is square in shape. The size of a CU can range from 8x8 pixels up to the size of a treeblock of 64x64 pixels or more. Each CU can include one or more PUs and one or more TUs. Syntax data associated with a CU may, for example, describe partitioning the CU into one or more PUs. The partitioning mode may vary depending on whether the CU is coded in skip or direct mode, intra-prediction mode, or inter-prediction (a.k.a. inter-prediction) mode. A PU may be partitioned to be non-square in shape. Syntax data associated with a CU may also describe partitioning the CU into one or more TUs according to, for example, a quadtree. The TUs can be square or non-square (eg, rectangular) in shape.

[0064] Mode select unit 40 may, for example, select one of the intra- or inter-coding modes based on the error result, provide the resulting intra- or inter-coded block to summer 50 to generate residual block data, and provide the resulting intra- or inter-coded block to summer 62 to reconstruct the coded block for use as a reference frame. Mode select unit 40 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy coding unit 56.

[0065] Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is a process of generating motion vectors that estimate the motion of video blocks. A motion vector may indicate, for example, the displacement of a PU of a video block in a current video frame or picture relative to a predictive block in a reference frame (or other coded unit) relative to a current block being coded in the current frame (or other coded unit). A predictive block is a block found to closely match a block to be coded in terms of pixel difference, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metric. In some examples, video encoder 20 may calculate values ​​of sub-integer pixel locations of a reference picture stored in reference frame memory 64. For example, video encoder 20 may interpolate ¼ pixel locations, ⅛ pixel locations, or other fractional pixel location values ​​of a reference picture. Thus, motion estimation unit 42 can perform motion searches for whole and fractional pixel locations and output motion vectors with fractional pixel accuracy.

[0066] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU with the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (list 0) or a second reference picture list (list 1), with each list identifying one or more reference pictures stored in reference frame memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

[0067] The motion compensation performed by motion compensation unit 44 may include fetching or generating a predictive block based on the motion vector determined by motion estimation unit 42. Again, in some examples, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. Upon receiving the motion vector of the PU of the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference picture lists. Adder 50 forms a residual video block by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded to form pixel difference values, as discussed below. In general, motion estimation unit 42 performs motion estimation on the luma component, and motion compensation unit 44 uses motion vectors calculated for both the chroma and luma components based on the luma component. Mode selection unit 40 may also generate syntax elements related to the video blocks and video slices used by video decoder 30 in decoding video blocks of the video slices.

[0068] Intra prediction unit 46 may intra predict the current block as an alternative to the inter prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. In particular, intra prediction unit 46 may determine an intra prediction mode to use to encode the current block. In some examples, intra prediction unit 46 may encode the current block using different intra prediction modes, e.g., during separate encoding passes, and intra prediction unit 46 (or, in some examples, mode selection unit 40) may select an appropriate intra prediction mode to use from the tested modes.

[0069] For example, intra prediction unit 46 may calculate a rate-distortion value using a rate-distortion analysis of the various intra prediction modes tested and select the intra prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block coded to generate the coded block, and the bit rate (i.e., number of bits) used to generate the coded block. Intra prediction unit 46 may calculate a ratio from the distortions and rates of the various coded blocks to determine which intra prediction mode exhibits the best rate-distortion value for the block.

[0070] Further, intra prediction unit 46 may be configured to code the depth blocks of the depth map using a depth modeling mode (DMM). Mode selection unit 40 may determine whether an available DMM mode produces better coding results than the intra prediction mode and other DMM modes, for example, using rate-distortion optimization (RDO). Data of the texture image corresponding to the depth map may be stored in reference frame memory 64. Motion estimation unit 42 and motion compensation unit 44 may also be configured to inter predict the depth blocks of the depth map.

[0071] After selecting an intra-prediction mode for a block (e.g., one of a conventional intra-prediction mode or a DMM mode), intra-prediction unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy coding unit 56. Entropy coding unit 56 may encode the information indicating the selected intra-prediction mode. Video encoder 20 may include in the transmitted bitstream configuration data, which may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables), a definition of the coding contexts of the various blocks, and an index to the most likely intra-prediction mode, intra-prediction mode index table, and modified intra-prediction mode index table to use for each context.

[0072] Video encoder 20 forms a residual video block by subtracting the prediction data from mode select unit 40 from the original video block being coded. Summer 50 represents one or more components that perform this subtraction operation.

[0073] Transform processing unit 52 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block, producing a video block that includes residual transform coefficient values. Transform processing unit 52 may perform other transforms that are conceptually similar to the DCT. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used.

[0074] Transform processing unit 52 applies a transform to the residual block, generating a block of residual transform coefficients. The transform may convert the residual information from a pixel value domain to a transform domain, such as the frequency domain. Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of a matrix including the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform the scan.

[0075] Following quantization, entropy coding unit 56 entropy codes the quantized transform coefficients. For example, entropy coding unit 56 may perform context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. In the case of context-based entropy coding, the context may be based on neighboring blocks. Following entropy coding by entropy coding unit 56, the encoded bitstream may be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

[0076] Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transformation, respectively, to reconstruct the residual block into the pixel domain, e.g., for later use as a reference block. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the frames in reference frame memory 64. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for use in motion estimation. Adder 62 adds the reconstructed residual block to the motion compensated predictive block generated by motion compensation unit 44 to generate a reconstructed video block for storage in reference frame memory 64. The reconstructed video block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block for intercoding blocks in subsequent video frames.

[0077] 3 is a block diagram illustrating an example of a video decoder 30 that may implement video coding techniques. In the example of FIG. 3, the video decoder 30 includes an entropy decoding unit 70, a motion compensation unit 72, an intra prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference frame memory 82, and an adder 80. In some examples, the video decoder 30 performs a decoding path that is generally inverse to the encoding path described with respect to the video encoder 20 (FIG. 2). The motion compensation unit 72 may generate prediction data based on the motion vectors received from the entropy decoding unit 70, while the intra prediction unit 74 may generate prediction data based on the intra prediction mode indicator received from the entropy decoding unit 70.

[0078] During the decoding process, video decoder 30 receives an encoded video bitstream from video encoder 20, which represents video blocks of encoded video slices and associated syntax elements. Entropy decoding unit 70 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Video decoder 30 may receive the syntax elements at the video slice level and / or the video block level.

[0079] If the video slice is coded as an intra-coded (I) slice, intra prediction unit 74 may generate prediction data for video blocks of the current video slice based on a signaled intra prediction mode and data from previously decoded blocks of the current frame or picture. If the video frame is coded as an inter-coded (e.g., B, P, or GPB) slice, motion compensation unit 72 generates prediction blocks for video blocks of the current video slice based on motion vectors and other syntax elements received from entropy decoding unit 70. The prediction blocks may be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct the reference frame lists, List 0 and List 1, using a default construction technique based on reference pictures stored in reference frame memory 82.

[0080] Motion compensation unit 72 determines prediction information for video blocks of the current video slice by analyzing the motion vectors and other syntax elements, and uses the prediction information to generate predictive blocks for the current video block being decoded. For example, motion compensation unit 72 uses some of the received syntax elements to determine a prediction mode (e.g., intra prediction or inter prediction) used to code the video blocks of the video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), configuration information of one or more reference picture lists of the slice, motion vectors for each inter-coded video block of the slice, inter-prediction status for each inter-coded video block of the slice, and other information for decoding video blocks in the current video slice.

[0081] Motion compensation unit 72 may also perform the interpolation based on an interpolation filter. Motion compensation unit 72 may use an interpolation filter used by video encoder 20 during encoding of the video block to calculate the interpolated values ​​of the sub-integer pixels of the reference block. In this case, motion compensation unit 72 may determine the interpolation filter used by video encoder 20 from the received syntax element and use the interpolation filter to generate the predictive block.

[0082] Data for texture images corresponding to the depth maps may be stored in reference frame memory 82. Motion compensation unit 72 may also be configured to inter-predict depth blocks of the depth maps.

[0083] Image and video compression has undergone rapid growth, leading to a variety of coding standards. Such video coding standards include Advanced Video Coding (AVC), also known as ITU-T H.261, ISO / IEC MPEG-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC) and Multiview Video Coding plus Depth (MVC+D), and 3D AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), 3D HEVC (3D-HEVC).

[0084] There is also a new video coding standard named Versatile Video Coding (VVC) being developed by the joint video experts team (JVET) of ITU-T and ISO / IEC. The latest working draft (WD) of VVC is included in JVET-L1001-v1, which is publicly available at http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 12_Macao / wg11 / JVET-L1001-v11.zip. The techniques disclosed herein are based on the incomplete development of VVC by the joint video experts team (JVET) of ITU-T and ISO / IEC. However, the techniques also apply to other video / media codec specifications.

[0085] Discuss the basics of video coding.

[0086] Video compression techniques perform spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (i.e., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be referred to as tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks of the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks of the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame.

[0087] Spatial or temporal prediction results in a prediction block of the block to be coded. The residual data represents pixel differences between the original block to be coded and the prediction block. Inter-coded blocks are coded according to a motion vector pointing to a block of reference samples forming the prediction block, and the residual data indicating the difference between the coded block and the prediction block. Intra-coded blocks are coded according to an intra-coding mode and the residual data. To further compress, the residual data can be transformed from the pixel domain to a transform domain to generate residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to generate a one-dimensional vector of transform coefficients, and entropy coding can be applied to achieve even more compression.

[0088] Discusses picture types in video coding.

[0089] Video codec specifications need to identify picture types in order to specify the decoding process, including deriving picture identification (eg, POC), marking reference picture status in the DPB, outputting pictures from the DPB, etc.

[0090] In AVC and HEVC, a picture type can be identified from the NAL unit type that contains the coded picture. AVC picture types include IDR and non-IDR pictures. Meanwhile, HEVC's main picture types include trailing picture, temporal sub-layer access picture (TSA), gradual temporal sub-layer access picture (STSA), random access decodable leading picture (RADL), random access skip leading picture (RASL), broken link access picture (BLA), instantaneous random access, and clean random access. For each of these main picture types in HEVC, a picture can be further differentiated as either a sub-layer reference picture or a sub-layer non-reference picture. A BLA picture is further differentiated as either a BLA with a leading picture, a BLA with a RADL picture, or a BLA without a leading picture. IDR pictures are further distinguished as IDR with RADL picture or IDR without leading picture.

[0091] Discuss Intra Random Access Point (IRAP) pictures.

[0092] In HEVC, IDR, BLA, and Clean Random Access (CRA) pictures are collectively considered as Intra Random Access Point (IRAP) pictures. For VVC, it was agreed at the 12th JVET meeting in October 2018 that both IDR and CRA pictures are included in IRAP pictures.

[0093] An IRAP picture provides two important functions or advantages: First, the presence of an IRAP picture indicates that the decoding process can start from that picture. This function enables a random access function, where the decoding process starts at a position in the bitstream, not necessarily at the beginning of the bitstream, as long as an IRAP picture is present at that position. Second, the presence of an IRAP picture refreshes the decoding process, such that coded pictures starting with the IRAP picture are coded without any reference to previous pictures, except for RASL pictures. The presence of an IRAP picture in the bitstream prevents any errors that may have occurred during the decoding of pictures coded before the IRAP picture from propagating to the IRAP picture and those pictures that follow it in the decoding order.

[0094] Although IRAP pictures provide important features, they may entail a penalty in compression efficiency. For example, their presence can cause bitrates to spike. The compression efficiency penalty has two sources. First, because an IRAP picture is an intra-predicted picture, the picture itself requires more bits to represent compared to other inter-predicted pictures. Second, an IRAP picture breaks temporal prediction. For example, when an IRAP picture is encountered during the decoding process, the DPB is refreshed and previous reference pictures are removed. In addition, an IRAP picture reduces the efficiency of coding pictures that follow the IRAP picture in decoding order. For example, pictures that follow the IRAP picture in decoding order require more bits to represent because they have fewer reference pictures for their inter-predictive coding.

[0095] Among the picture types considered to be IRAP pictures, IDR pictures in HEVC have different signaling and derivation when compared to other picture types. Some differences are:

[0096] For the signaling and derivation of the POC value for an IDR picture, the most significant bit (MSB) of the POC is not derived from the previous key picture. Rather, the MSB is simply set equal to 0.

[0097] The slice header of an IDR picture does not contain any information that needs to be signaled to assist in reference picture management. For other picture types (e.g., CRA, Trailing, TSA, etc.), information such as the Reference Picture Set (RPS) described below, or some other form of similar information (e.g., Reference Picture List) is required for the reference picture marking process (e.g., the process of determining the status of a reference picture in the DPB, whether it is used for reference or not used for reference). However, for an IDR picture, there is no need to signal such information, since the presence of the IDR indicates that the decoding process will only mark all reference pictures in the DPB as not used for reference.

[0098] This paper discusses reference picture management in video coding.

[0099] In addition to picture type, picture identification is also required for several purposes, including use as a reference picture in inter prediction, output of the picture from the Decoded Picture Buffer (DPB), scaling of motion vectors, weighted prediction, etc. In AVC and HEVC, pictures can be identified by their Picture Order Count (POC).

[0100] In AVC and HEVC, pictures in the DPB can be marked as "used for short-term reference", "used for long-term reference", or "unused for reference". When a picture is marked as "unused for reference", it can no longer be used for prediction. If the picture no longer needs to be output, it can be removed from the DPB.

[0101] In AVC, there are two types of reference pictures: short-term and long-term. A reference picture can be marked as "not used for reference" when the picture is no longer needed for prediction reference. The conversion between these three statuses (short-term, long-term, and not used for reference) is controlled by the decoded reference picture marking process. There are two alternative decoded reference picture marking mechanisms: an implicit sliding window process and an explicit memory management control operation (MMCO) process. The sliding window process marks a short-term reference picture as "not used for reference" when the number of reference frames is equal to a specified maximum number (e.g., max_num_ref_frames in the SPS). Short-term reference pictures are stored in a first-in, first-out manner, so the most recently decoded short-term picture is kept in the DPB.

[0102] An explicit MMCO process may include multiple MMCO commands that may mark one or more short-term or long-term reference pictures as "unused for reference", mark all pictures as "unused for reference", or mark the current reference picture or an existing short-term reference picture as long-term and assign a long-term picture index to the long-term reference picture.

[0103] In AVC, the reference picture marking operation, as well as the processes for outputting and removing pictures from the DPB, are performed after decoding the pictures.

[0104] HEVC introduces a different approach to reference picture management, called Reference Picture Set (RPS). The most fundamental difference with the RPS concept compared to AVC's MMCO / sliding window is that for each particular slice, a complete set of reference pictures is provided that will be used in the current picture or any subsequent picture. Thus, the complete set of all pictures kept in the DPB for use in the current or future pictures is signaled. This differs from the AVC scheme, where only relative changes to the DPB are signaled. With the RPS concept, no information from previous pictures in the decoding order is needed to maintain the correct status of reference pictures in the DPB.

[0105] The order of picture decoding and DPB operations in HVC is changed compared to AVC to take advantage of RPS and improve error resilience. In AVC, picture marking and buffer operations (both outputting and removing decoded pictures from DPB) are usually applied after the current picture is decoded. In HEVC, RPS is first decoded from the slice header of the current picture, and then picture marking and buffer operations are usually applied before decoding the current picture.

[0106] We discuss RPS signaling in HEVC.

[0107] Each slice header in HEVC contains parameters for signaling the RPS with respect to the picture that contains the slice. The only exception is that no RPS is signaled for an IDR slice. Instead, the RPS is inferred to be empty. In case of an I-slice that does not belong to an IDR picture, an RPS may be provided even if the I-slice belongs to an I-picture. This is because there may be pictures following the I-picture in decoding order that use inter prediction based on pictures preceding the I-picture in decoding order. The number of pictures in the RPS must not exceed the DPB size limit specified by the sps_max_dec_pic_buffering syntax element in the SPS.

[0108] Each picture is associated with a POC value that represents the output order. The slice header contains a fixed-length codeword pic_order_cnt_lsb, which represents the least significant bits of the complete POC value, also known as the POC LSB. The length of the codeword is signaled in the SPS and can be between 4 and 16 bits. The RPS concept uses the POC to identify reference pictures. In addition to the POC value in the slice header itself, each slice header either directly contains in the SPS or inherits from the SPS a coded representation of the POC values ​​(or LSBs) of each picture in the RPS.

[0109] The RPS of each picture consists of five different lists of reference pictures, also called five RPS subsets: RefPicSetStCurrBefore consists of all short-term reference pictures that precede the current picture in both decoding order and output order and can be used for inter-prediction of the current picture; RefPicSetStCurrAfter consists of all short-term reference pictures that precede the current picture in decoding order, follow the current picture in output order and can be used for inter-prediction of the current picture; RefPicSetStFoll consists of all short-term reference pictures that can be used for inter-prediction of one or more pictures that follow the current picture in decoding order and are not used for inter-prediction of the current picture; RefPicSetLtCurr consists of all long-term reference pictures that can be used for inter-prediction of one or more pictures that follow the current picture in decoding order and are not used for inter-prediction of the current picture; RefPicSetLtFoll consists of all long-term reference pictures that can be used for inter-prediction of one or more pictures that follow the current picture in decoding order and are not used for inter-prediction of the current picture.

[0110] The RPS is signaled using up to three loops that iterate through different types of reference pictures (short-term reference pictures with lower POC values ​​than the current picture, short-term reference pictures with higher POC values ​​than the current picture, and long-term reference pictures). Additionally, a flag (used_by_curr_pic_X_flag) is sent for each reference picture indicating whether the reference picture is used for reference by the current picture (included in either of the lists RefPicSetStCurrBefore, RefPicSetStCurrAfter, or RefPicSetLtCurr) or is not used for reference by the current picture (included in either of the lists RefPicSetStFoll or RefPicSetLtFoll).

[0111] Figure 4 shows an RPS 400 with a current picture B14 that contains entries (e.g., pictures) in all subsets 402 of the RPS 400. In the example of Figure 4, the current picture B14 contains exactly one picture in each of the five subsets 402 (also known as RPS subsets). P8 is a picture in a subset 402 called RefPicSetStCurrBefore because the picture is earlier in the output order and is used by B14. P12 is a picture in a subset 402 called RefPicSetStCurrAfter because the picture is later in the output order and is used by B14. P13 is a picture in a subset 402 called RefPicSetStFoll because the picture is a short-term reference picture that is not used by B14 (but must be kept in the DPB because it is used by B15). P4 is a picture in the subset 402 called RefPicSetLtCurr because the picture is a long-term reference picture used by B14. I0 is a picture in the subset 402 called RefPicSetLtFoll because the picture is a long-term reference picture that is not used by the current picture (but needs to be kept in the DPB because it is used by B15).

[0112] The short-term part of the RPS 400 may be included directly in the slice header. Alternatively, the slice header may only contain a syntax element representing an index, referring to a predefined list of RPSs transmitted in the active SPS. The short-term part of the RPS 402 may be signaled using either of two different schemes: Inter-RPS, described below, or Intra-RPS, described here. When using Intra-RPS, num_negative_pics and num_positive_pics are signaled, which represent the length of two different lists of reference pictures. These lists contain reference pictures that have a negative and a positive POC difference, respectively, compared to the current picture. Each element of these lists is coded with a variable length code, which represents the difference in the POC value compared to the previous element of the list minus one. For the first picture of each list, the signaling relates to the POC value of the current picture minus one.

[0113] When coding RPSs that rotate in a sequence parameter set, it is possible to code an element of one RPS (e.g., RPS 400) with reference to another RPS already coded in the sequence parameter set. This is called inter-RPS. Since all RPSs of a sequence parameter set are in the same Network Abstraction Layer (NAL) unit, there are no error robustness problems associated with this method. The inter-RPS syntax exploits the fact that the RPS of the current picture can be predicted from the RPSs of previously decoded pictures. This is because all reference pictures of the current picture must either be reference pictures of previous pictures or previously decoded pictures themselves. It is only necessary to indicate which of these pictures are reference pictures and should be used for the prediction of the current picture. Thus, the syntax consists of the following: an index pointing to the RPS to be used as a predictor, a delta_POC that is added to the delta_POC of the predictor to obtain the delta POC of the current RPS, and a set of indicators indicating which pictures are reference pictures and whether they are only used for the prediction of future pictures. In one embodiment, delta POC refers to the difference in POC value between a current reference picture and another (eg, an earlier) reference picture.

[0114] An encoder that wants to take advantage of the use of long-term reference pictures must set the SPS syntax element long_term_ref_pic_present_flag to 1. Long-term reference pictures are then signaled in the slice header by a fixed-length codeword poc_lsb_lt, which represents the least significant bits of the complete POC value of each long-term picture. Each poc_lsb_lt is a copy of the pic_order_cnt_lsb codeword signaled for a particular long-term picture. The set of long-term pictures in the SPS can also be signaled as a list of POC LSB values. The POC LSB of a long-term picture can then be signaled in the slice header as an index into this list.

[0115] The delta_POC_msb_cycle_lt_minus1 syntax element can further be signaled to enable calculation of the full POC distance of a long-term reference picture relative to the current picture. The codeword delta_POC_msb_cycle_lt_minus1 needs to be signaled for every long-term reference picture that has the same POC LSB value as any other reference picture in the RPS.

[0116] We discuss reference picture marking in HEVC.

[0117] Before picture decoding, there will typically be a number of pictures in the DPB. Some pictures may be marked as "used for reference" because they are available for prediction. Other pictures may be marked as "not used for reference" because they are not available for prediction but are waiting for output. When the slice header is parsed, a picture marking process is performed before decoding the slice data. Pictures that are present in the DPB and marked as "used for reference" but are not included in the RPS are marked as "not used for reference". Pictures that are not present in the DPB but are included in the reference picture set are ignored if used_by_curr_pic_X_flag is equal to zero. However, instead, if used_by_curr_pic_X_flag is equal to one, this reference picture was intended to be used for prediction in the current picture but is missing. An unintended picture loss is then inferred and the decoder needs to take appropriate action.

[0118] After decoding the current picture, the decoded picture is marked as "used for short-term reference".

[0119] We discuss the construction of reference picture lists in HEVC.

[0120] In HEVC, the term inter prediction is used to indicate prediction derived from data elements (e.g., sample values ​​or motion vectors) of reference pictures other than the currently decoded picture. As in AVC, a picture can be predicted from multiple reference pictures. The reference pictures used for inter prediction are organized into one or more reference picture lists. A reference index identifies which reference picture in the list should be used to create the prediction signal.

[0121] A single reference picture list, List 0, is used for P slices, and two reference picture lists, List 0 and List 1, are used for B slices. Similar to AVC, constructing reference picture lists in HEVC includes reference picture list initialization and reference picture list modification.

[0122] In AVC, the initialization process for list 0 is different for P slices (decoding order is used) and B slices (output order is used). In HEVC, output order is used in both cases.

[0123] The reference picture list initialization creates default list 0 and list 1 (if the slice is a B slice) based on three RPS subsets: RefPicSetStCurrBefore, RefPicSetStCurrAfter, and RefPicSetLtCurr. Short-term pictures with early (late) output order are inserted in list 0 (list 1) first in ascending order of POC distance to the current picture, then short-term pictures with late (early) output order are inserted in list 0 (list 1) in ascending order of POC distance to the current picture, and finally, long-term pictures are inserted last. In terms of RPS, in case of list 0, the entry for RefPicSetStCurrBefore is inserted in the first list, followed by the entry for RefPicSetStCurrAfter. Then, the entry for RefPicSetLtCurr is added if available.

[0124] In HEVC, if the number of entries in the list is less than the target number of active reference pictures (signaled in the picture parameter set or slice header), the above process is repeated (reference pictures already added to the reference picture list are added again). If the number of entries is more than the target number, the list is truncated.

[0125] After the reference picture list is initialized, the reference pictures may be modified such that the reference pictures of the current picture may be arranged in any order, including the case where one particular reference picture may appear at multiple positions in the list based on the reference picture list modification command. When the flag indicating the presence or absence of list modification is set to 1, a fixed number of commands (equal to the target number of entries in the reference picture list) are signaled, each command inserting one entry into the reference picture list. The reference picture is identified in the command by an index into the list of reference pictures of the current picture derived from RPS signaling. This is different from the reference picture list modification of H.264 / AVC, where the picture is identified by either the picture number (derived from the frame_num syntax element) or the long-term reference picture index, and fewer commands may be required, for example to swap the first two entries of the initial list, or to insert one entry at the beginning of the initial list and shift the other entries.

[0126] A reference picture list is not allowed to contain any reference picture with a TemporalID higher than the current picture. An HEVC bitstream may consist of several temporal sub-layers. Each NAL unit belongs to a particular sub-layer, indicated by its TemporalID (equal to temporal_id_plus1-1).

[0127] Reference picture management based directly on the reference picture list is discussed.

[0128] The JCT-VC document JCTVC-G643, published at http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 7_Geneva / wg11 / JCTVC-G643-v3.zip, includes an approach that directly uses three reference picture lists: reference picture list 0, reference picture list 1, and an idle reference picture list for management of reference pictures in the DPB. This approach eliminates the need to utilize either 1) the sliding window and MMCO process, and the reference picture list initialization and modification process in AVC, or 2) the reference picture set, and the reference picture list initialization and modification process in HEVC, which simplifies signaling and decoding.

[0129] JVET document JVET-L0112, published at http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 12_Macao / wg11 / JVET-L0112-v4.zip, describes another approach for reference picture management based directly on reference picture lists. Instead of using three reference picture lists as proposed in JCTVC-G643, the approach proposed in JVET-L0112 uses only two reference picture lists: reference picture list 0 and reference picture list 1. Each reference picture list contains information related to the reference pictures for constructing the associated final reference picture list (e.g., the reference pictures in reference picture list 0 are for constructing final reference picture list 0, and the reference pictures in reference picture list 1 are for constructing final reference picture list 1). Each reference picture list may include inactive reference pictures (eg, not needed for the current picture, but may be needed for a future picture).

[0130] We discuss picture partitioning schemes in HEVC.

[0131] HEVC includes four different picture partitioning schemes, namely, regular slices, dependent slices, tiles, and wavefront parallel processing (WPP), which can be applied for maximum transfer unit (MTU) size matching, parallel processing, and reduced end-to-end delay.

[0132] Regular slices are similar to those in H.264 / AVC: each regular slice is encapsulated in its own NAL unit, and in-picture prediction (intra-sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across slice boundaries are disabled. Thus, regular slices can be reconstructed independently from other regular slices in the same picture (although interdependencies may still remain due to loop filtering operations).

[0133] Regular slices are the only tool available for parallelization, and are also available in H.264 / AVC in a substantially identical form. Regular slice-based parallelization does not require much inter-processor or inter-core communication (except for inter-processor or inter-core data sharing for motion compensation when decoding predictively coded pictures, which is typically much heavier than inter-processor or inter-core data sharing by in-picture prediction). However, for the same reasons, the use of regular slices can incur significant coding overhead due to the bit cost of slice headers and the lack of prediction across slice boundaries. Furthermore, regular slices (in contrast to the other tools described below) also serve as the primary mechanism for bitstream partitioning to match MTU size requirements, due to their in-picture independence from regular slices, and because each regular slice is encapsulated in its own NAL unit. In many cases, the goal of parallelization and the goal of matching MTU size put conflicting demands on the slice layout of a picture. The realization of this situation led to the development of the parallelization tools described below.

[0134] Dependent slices have short slice headers and allow for bitstream partitioning at treeblock boundaries without upsetting any in-picture prediction. Essentially, dependent slices reduce end-to-end delay by fragmenting regular slices into multiple NAL units, allowing parts of a regular slice to be transmitted before the coding of the entire regular slice is completed.

[0135] In WPP, a picture is partitioned into single rows of coding tree blocks (CTBs). Entropy decoding and prediction can use data from CTBs of other partitions. Parallelism is possible by parallel decoding of CTB rows, where the start of decoding of a CTB row is delayed by two CTBs, so that data related to the CTB above and to the right of the target CTB is available before the target CTB is being decoded. This staggered start (which looks like a wave front when represented graphically) allows parallelization up to as many processors / cores as there are CTB rows in the picture. In-picture prediction between adjacent treeblock rows within a picture is allowed, so the amount of inter-processor / inter-core communication required to allow in-picture prediction can be significant. WPP partitioning does not generate additional NAL units compared to the case where it is not applied. Thus, WPP is not a tool for MTU size matching. However, if MTU size matching is required, regular slices can be used in WPP with certain coding overhead.

[0136] Tiles define horizontal and vertical boundaries that partition the picture into tile columns and rows. The scan order of the CTBs is changed to be local within the tile (in the tile CTB raster scan order) before decoding the top-left CTB of the next tile in the picture tile raster scan order. Like regular slices, tiles break in-picture prediction dependencies and entropy decoding dependencies. However, tiles are not required to be included in individual NAL units (as they are in this respect identical to WPP). For this reason, tiles cannot be used for MTU size matching. Each tile can be processed by one processor / core, and the inter-processor / inter-core communication required for in-picture prediction between processing units decoding adjacent tiles is limited to the communication of shared slice headers and loop filter processing related to sharing reconstructed samples and metadata when a slice spans multiple tiles. When multiple tiles or WPP segments are included in a slice, the entry point byte offset of each tile or WPP segment of the slice, except for the first one, is signaled in the slice header.

[0137] For the sake of brevity, restrictions for the application of four different picture partitioning schemes are specified in HEVC. A given coded video sequence cannot contain both tiles and wavefronts for most profiles specified in HEVC. For each slice and tile, one or both of the following conditions must be satisfied: 1) all coded treeblocks in a slice belong to the same tile; 2) all coded treeblocks in a tile belong to the same slice. Finally, a wavefront segment contains exactly one CTB row, and if WPP is used, if a slice starts in a CTB row, the slice must end in the same CTB row.

[0138] We discuss motion constrained tile sets (MCTS).

[0139] Recent amendments to HEVC are specified in JCT-VC output document JCTVC-AC1005, J Boyce, A. Ramasubramonian, R. Skupin, GJ Sullivan, A. Tourapis, Y.-K. Wang (editors), “HEVC Additional Supplemental Enhancement Information (Draft 4),” Oct. 24, 2017, and available at http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 29_Macau / wg11 / JCTVC-AC1005-v2.zip. With this amendment, HEVC specifies three MCTS-related supplemental enhancement information (SEI) messages: the temporary MCTS SEI message, the MCTS extraction information set SEI message, and the MCTS extraction information nested SEI message.

[0140] The temporal MCTS SEI message indicates the presence of an MCTS in the bitstream and signals the MCTS. For each MCTS, motion vectors are restricted to point to full sample positions within the MCTS and fractional sample positions that require only full sample positions within the MCTS for interpolation, and the use of motion vector candidates for temporal motion vector prediction derived from blocks outside the MCTS is not allowed. In this way, each MCTS can be decoded independently, even if there are no tiles not included in the MCTS.

[0141] The MCTS extraction information set SEI message provides supplemental information that can be used in MCTS sub-bitstream extraction (specified as part of the semantics of the SEI message) to generate a conforming bitstream of an MCTS set. This information includes several extraction information sets, each of which specifies several MCTS sets and includes the RBSP bytes of the replacement VPS, SPS, and PPS to be used during the MCTS sub-bitstream extraction process. When extracting a sub-bitstream according to the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, and PPS) need to be rewritten or replaced, and the slice header needs to be slightly updated because one or all of the syntax elements related to slice addresses (including first_slice_segment_in_pic_flag and slice_segment_address) typically need to have different values.

[0142] Discuss tile groups.

[0143] After the 12th JVET meeting held in Macao in October 2018, it was agreed to replace the concept of slices with tile groups. However, at the time of this disclosure, the latest draft of the VVC did not yet include the agreed concept of tile groups. Contribution JVET-L0686 is publicly available at http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 12_Macao / wg11 / JVET-L0686-v2.zip and contains the text of the agreed tile groups. In the tile groups agreed upon at the 12th JVET meeting, one or more tiles can be grouped into a tile group. The tiles that belong to a tile group are consecutive in the raster scan order of the picture. In the remainder of this disclosure, the tile groups described in JVET-L0686 are referred to as raster scan tile groups.

[0144] Contribution JVET-L0114, published at http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 12_Macao / wg11 / JVET-L0114-v1.zip, describes another approach to tile groups. The tile groups described herein are constrained such that the tiles grouped together in the tile group form a rectangular-shaped region in the picture. In the remainder of this disclosure, the tile groups described in JVET-L0114 are referred to as rectangular tile groups.

[0145] Discuss viewport-dependent use cases for 360° video applications.

[0146] A 360-degree (360°) video application only displays a portion of the entire sphere (resulting in only a subset of the entire picture). To reduce bitrate, 360° video is delivered over DASH using a use case scenario called 360° Delivery over DASH, which is viewport dependent. The use case scenario is as follows:

[0147] Divide the sphere / entire projected image into multiple MCTS (e.g., using cube map projection (CMP)).

[0148] Two or more bitstreams are encoded at different spatial resolutions or qualities.

[0149] When delivering to a decoder, the MCTS from the higher resolution / quality bitstream is used to display a viewport (e.g., the front viewport) and the MCTS from the lower resolution / quality bitstream is used to display the rest. These MCTS are packed in a specific way and then sent to the receiver for decoding.

[0150] To provide a good viewing experience, it is expected that the viewport seen by the user is represented by a high resolution / quality MCTS. When the user turns his / her head to look at another viewport (e.g., the left or right viewport), the displayed content will come from the lower resolution / quality viewport for a short period of time while the system fetches the high resolution / quality MCTS for that viewport.

[0151] When a user turns his / her head to look at a different viewport, there is a delay between when the user turns his / her head and when the higher resolution / quality representation of that viewport is seen. This delay depends on how fast the system can fetch the higher resolution / quality MCTS for that viewport, which in turn depends on the IRAP period (e.g., the interval between occurrences between two IRAPs) since the MCTS for the new viewport can only be decoded starting from the IRAP picture. If the IRAP period is coded per second, the following applies:

[0152] The best case scenario of the delay is equal to the network round-trip delay when the user turns his / her head to look at a new viewport just before the system starts fetching the new segment / IRAP period. In this scenario, the system could immediately request a higher resolution / quality MCTS for the new viewport. So the only delay is the network round-trip delay (the sum of the fetch request delay and the transmission time of the requested MCTS, assuming the minimum buffering delay can be set to zero (but typically streaming systems do not allow the minimum buffering delay to be set to zero), and the sensor delay is small and negligible). The network round-trip delay could be, for example, around 200 milliseconds (ms).

[0153] The worst case scenario for delay is the IRAP duration plus the network round-trip delay if the user turns his or her head to look at the new viewport just after the system has already made a request for the next segment.

[0154] To improve on the worst-case scenario above, the bitstream can be coded with more frequent IRAP pictures, resulting in a shorter IRAP period and thus a lower overall delay, although this results in increased bandwidth requirements due to reduced compression efficiency.

[0155] The problems with the existing IRAP concept are discussed.

[0156] Until the latest developments in HEVC and VVC, the IRAP concept is a picture-level concept. It means that the smallest object in a coded video bitstream associated with an intra-random access point is a picture. It would be beneficial if the IRAP concept could be applied at the sub-picture level. Using IRAP at the sub-picture level would help reduce the worst-case scenarios in 360° delivery over DASH, as mentioned above. However, it is difficult to improve the IRAP concept in this way in existing video coding specifications (e.g., HEVC, VVC, etc.) due to the following issues:

[0157] In general, the signaling, derivation and decoding processes of IRAP pictures are different from those of non-IRAP pictures. Furthermore, many aspects are specified at the picture level. Some differences are:

[0158] Regarding POC derivation, the POC MSB is always set to 0 for IDR pictures, while the POC MSB is derived from the previous key picture for other picture types.

[0159] Regarding reference picture management, when an IDR picture is received, no information needs to be signaled for the IDR picture. The decoder just marks all reference pictures in the DPB as "unused for reference". Meanwhile, for other picture types, information (e.g. RPS, RPL, etc.) needs to be signaled to assist in reference picture management.

[0160] Sub-pictures can be extracted from the original bitstream to form a new bitstream via a sub-bitstream extraction process. It is desirable that the signaling, derivation, and decoding processes of the same sub-picture before and after the sub-bitstream extraction process remain unchanged and produce the same decoding / reconstruction result.

[0161] One of the shortcomings of current coding techniques is the inclusion of IRAP pictures, which are intra-coded. Therefore, the coder / decoder (aka codec) codes IRAP pictures without the aid of a reference picture list (RPL). As a result, the syntax used in current coding techniques instructs the codec not to look up the RPL whenever it encounters an IRAP picture.

[0162] In virtual reality (VR) coding, it may be desirable to split a picture into sub-pictures, where one sub-picture is an IRAP sub-picture and the other is a non-IRAP sub-picture. When a picture is split in this way, it may be called a mixed IRAP picture. However, mixed IRAP pictures present problems in current coding techniques. In fact, current syntax requires that the codec ignore any RPL throughout a mixed IRAP picture because of the presence of IRAP sub-pictures. Because the RPL is ignored, the codec cannot code non-IRAP sub-pictures that rely on the RPL for proper coding as inter-coded pictures.

[0163] Disclosed herein is a video coding technique that allows IRAP pictures, particularly mixed IRAP pictures, to refer to and utilize the RPL. Thus, even if a mixed IRAP picture contains an IRAP sub-picture, the codec is allowed to refer to and utilize the RPL to code the non-IRAP sub-picture. This is particularly useful in VR coding applications, but the concept can be applied to other fields as well.

[0164] 5 is a schematic diagram of an embodiment of a picture 500 suitable for use in a VR coding application. As shown, the picture 500 is divided into a first sub-picture 502 and a second sub-picture 504. In one embodiment, the first sub-picture 502 corresponds to a viewport used in a VR application. The viewport is the portion of the picture that a user of the VR application or program is currently looking at. In one embodiment, the second sub-picture 504 comprises the remaining portion of the picture 500. That is, the second sub-picture 504 is the portion of the picture 500 outside the viewport. In one embodiment, the first sub-picture 502 is an IRAP picture and the second sub-picture 504 is a non-IRAP picture at a particular time instant t in the bitstream. As such, the picture 500 may be referred to as a mixed IRAP picture at time t.

[0165] Figure 6 is a schematic diagram of one embodiment of a video bitstream 600 corresponding to the picture 500 of Figure 5. As used herein, the video bitstream 600 may also be referred to as a coded video bitstream, a bitstream, or variations thereof. The picture 500 of Figure 6 may be contained within a single NAL unit (represented by a bold black rectangle) or may be contained within several NAL units.

[0166] As shown in FIG. 6, the bitstream 600 is divided into a first sub-bitstream 602 and a second sub-bitstream 604. The first sub-bitstream 602 corresponds to the first sub-picture 502, and the second sub-bitstream 604 corresponds to the second sub-picture 504. Because the first sub-bitstream 602 corresponds to the first sub-picture 502, which is the viewport in this example, the first sub-bitstream 602 contains more IRAP sub-pictures than the second sub-bitstream 604. The IRAP picture allows the codec to start decoding at that position in the bitstream 600. Because the first sub-stream 602 contains several IRAP pictures, the decoder can start decoding the first sub-picture 502 at various different positions. Examples of IRAP pictures include an Instantaneous Decoder Refresh (IDR) picture, a Clean Random Access (CRA) picture, and a Broken Link Access (BLA) picture.

[0167] According to the embodiments disclosed herein, when a decoder (e.g., video decoder 30) encounters picture 500 in bitstream 600 during the decoding process, the decoder is no longer instructed by the presence of an IRAP sub-picture (e.g., first sub-picture 502) to ignore any RPL throughout the mixed IRAP picture (e.g., picture 500). Thus, even if the mixed IRAP picture contains an IRAP sub-picture, the decoder is allowed to refer to and use the RPL to decode the non-IRAP sub-picture (e.g., second sub-picture 504). That is, IRAP pictures, especially mixed IRAP pictures, are allowed to refer to and use the RPL. This allows mixed IRAP pictures to be useful in VR applications.

[0168] In one embodiment, a flag may be signaled by an encoder (e.g., video encoder 20) to indicate to a decoder (e.g., video decoder 30) whether a bitstream (e.g., bitstream 600) includes any of the mixed IRAP pictures (e.g., picture 500). The flag may be signaled in the sequence parameter set (SPS), picture parameter set (PPS), or another parameter set of the bitstream. In one embodiment, the flag is designated as sps_mixed_tile_groups_in_pic_flag.

[0169] In one embodiment, the first sub-picture 502 and the second sub-picture 504 may be referred to as a tile group. In one embodiment, an RPL is constructed at the beginning of the decoding of each tile group, regardless of the NAL unit type of that tile group. The RPL may include, for example, RefPicList[0] and RefPicList[1] in the case of the RPL approach, or RefPicList0[] and RefPicList1[] in the case of the reference picture set (RPS) approach. Similar lists including reference pictures for inter prediction operations may also be utilized.

[0170] 7 is an embodiment of a method 700 for decoding a coded video bitstream (e.g., bitstream 600) implemented by a video decoder (e.g., video decoder 30). Method 700 may be executed after receiving a decoded bitstream directly or indirectly from a video encoder (e.g., video encoder 20). Method 700 improves the decoding process (e.g., makes the decoding process more efficient, faster, etc. than conventional decoding processes) because IRAP pictures, especially mixed IRAP pictures, are allowed to refer to and utilize the RPL. This allows for mixed IRAP pictures, which are beneficial in VR applications. Thus, in effect, the performance of the codec is improved, leading to a better user experience.

[0171] In block 702, a video decoder receives a mixed intra-random access point (IRAP) picture (e.g., picture 500) including a first sub-picture (e.g., sub-picture 502) and a second sub-picture (e.g., sub-picture 504). In one embodiment, the first sub-picture is an IRAP picture and the second sub-picture is a non-IRAP sub-picture. In one embodiment, the mixed IRAP picture is received in a split bitstream including a first sub-bitstream (e.g., first sub-bitstream 602) and a second sub-bitstream (e.g., second sub-bitstream 604). In one embodiment, the IRAP picture is an Instantaneous Decoder Refresh (IDR) picture. In one embodiment, the mixed IRAP picture is included in a single Network Access Layer (NAL) unit.

[0172] The video decoder receives a reference picture list (RPL) for the mixed IRAP picture in block 704. For mixed IRAP pictures that include at least one IRAP picture, the RPL is received or otherwise obtained by the video decoder.

[0173] At block 706, the video decoder decodes the second sub-picture using the RPL. At block 708, the video decoder generates an image based on the decoded second sub-picture. In one embodiment, the image may be displayed to a user on a display or screen of an electronic device (e.g., a smartphone, tablet, laptop, personal computer, etc.).

[0174] In one embodiment, the method 700 further includes receiving a flag in the bitstream, the flag indicating whether the bitstream includes a mixed IRAP picture, in one embodiment, the flag is signaled in the SPS, PPS, or another part of the bitstream.

[0175] 8 is an embodiment of a method 800 for encoding a video bitstream (e.g., bitstream 500) implemented by a video encoder (e.g., video encoder 20). Method 800 may be executed when pictures (e.g., from a video) are encoded into a video bitstream and then transmitted towards a video decoder (e.g., video decoder 30). Method 800 improves the encoding process (e.g., makes the encoding process more efficient, faster, etc. than conventional encoding processes) because IRAP pictures, especially mixed IRAP pictures, are allowed to refer to and utilize the RPL. This allows mixed IRAP pictures, which are beneficial in VR applications. Thus, in effect, the performance of the codec is improved, leading to a better user experience.

[0176] In block 802, a video encoder encodes a mixed intra-random access point (IRAP) picture including a first sub-picture and a second sub-picture, where the first sub-picture is an IRAP picture and the second sub-picture is a non-IRAP sub-picture. In one embodiment, the mixed IRAP picture is encoded into a split bitstream including a first sub-bitstream and a second sub-bitstream. In one embodiment, the first sub-picture is encoded into the first sub-bitstream and the second sub-picture is encoded into the second sub-bitstream. In one embodiment, the IRAP picture is an Instantaneous Decoder Refresh (IDR) picture. In one embodiment, the mixed IRAP picture is encoded into a single Network Access Layer (NAL) unit.

[0177] In block 804, the video encoder encodes a reference picture list (RPL) for the mixed IRAP picture.

[0178] In block 806, the video encoder generates a bitstream including the mixed IRAP picture and an RPL corresponding to the mixed IRAP picture. The video encoder encodes the RPL in the bitstream regardless of whether the mixed IRAP picture includes at least one IRAP picture.

[0179] At block 808, the video encoder stores the bitstream for transmission to the video decoder. The bitstream may be at least temporarily stored in memory until the video encoder transmits the video bitstream (e.g., bitstream 600) to the video decoder. Once received by the video decoder, the encoded video bitstream is decoded (e.g., as described above) to generate or create images for display to a user on a display or screen of an electronic device (e.g., a smartphone, tablet, laptop, personal computer, etc.).

[0180] In one embodiment, the method 800 further includes encoding a flag in the bitstream that indicates whether the bitstream contains mixed IRAP pictures, in one embodiment, the flag is signaled in the SPS, PPS, or another part of the bitstream.

[0181] The description of the technology disclosed herein is provided with reference to the state-of-the-art approach in JVET-L0686-v2 and JVET-L0112-v2. Changes related to the approaches in JVET-L0686-v2 and JVET-L0112-v2 are shown in italics (in the following, instead of italics, {}in show( { } The contents of are in italics), and deletions are in bold (below, instead of bold [] Shown in ( [ ] On the other hand, the text of the approaches in JVET-L0686-v2 and JVET-L0112-v2 not mentioned below are applied as is.

[0182] The following provisions are provided:

[0183] { Instantaneous Decoding Refresh (IDR) Tile Group: A tile group contained in a VCL NAL unit with nal_unit_type equal to IDR_NUT. }

[0184] { Intra-random access point (IRAP) tile group: A tile group contained in a VCL NAL unit with nal_unit_type equal to IDR_NUT or CRA_NUT. }

[0185] { Intra-random access point (IRAP) picture: A coded picture in which each VCL NAL unit has a nal_unit_type of IDR_NUT or CRA_NUT. }

[0186] { NOTE - The value of FirstIrapPictureFlag is equal to 1 for each IDR or CRA access unit that is the first access unit of the CVS in decoding order. If the value of FirstIrapPictureFlag is equal to 1, the value of NoRaslOutputFlag is set equal to 1. An IDR picture is always the first access unit of the CVS. A CRA access unit is the first access unit of the CVS if it is the first picture in the bitstream, if it immediately follows an end-of-sequence NAL unit, or if the associated variable HandleCraAsFirstPicInCvsFlag is equal to 1. The variable HandleCraAsFirstPicInCvsFlag can be set by external means. }

[0187] {non-IRAP tile group: A tile group contained in a VCL NAL unit whose nal_unit_type is not equal to IDR_NUT or CRA_NUT. }

[0188] NAL unit header semantics are provided. Table 71 - NAL unit type codes and NAL unit type classes [Table 1]

[0189] { TemporalID shall be equal to 0 if every tile group of a picture has nal_unit_type equal to IDR_NUT or CRA_NUT, i.e. if the current tile group belongs to an IRAP picture. }

[0190] { TemporalID shall be equal to 0 if the nal_unit_type of a VCL NAL unit is neither equal to IDR_NUT nor CRA_NUT while at least one other VCL NAL unit of the same picture has nal_unit_type equal to IDR_NUT or CRA_NUT. }

[0191] The syntax and semantics of sequence parameter sets are provided. [Table 2]

[0192] {sps_mixed_tile_groups_in_pic_flag equal to 1 specifies that there can be pictures in the CVS that have both IRAP and non-IRAP tile groups. sps_mixed_tile_groups_in_pic_flag equal to 0 specifies that each picture in the CVS has only IRAP or only non-IRAP tile groups. }

[0193] Discusses picture parameter set syntax. [Table 3]

[0194] Discusses tile group syntax. [Table 4] TIFF2025063096000006.tif19159

[0195] poc_msb_reset_flag is used to specify the variable PicRefreshFlag as follows:

[0196] - PicRefreshFlag is set equal to 1 if the current tile group belongs to the first access unit of the bitstream in decoding order.

[0197] - Otherwise, if the current tile group is an IDR tile group, PicRefreshFlag is set equal to sps_mixed_tile_groups_in_pic_flag?poc_msb_reset_flag:1.

[0198] - Otherwise, if the current tile group is a CRA tile group, the following applies:

[0199] - PicRefreshFlag is set equal to 1 if the current access unit immediately follows the end NAL unit of a sequence or if the associated variable HandleCraAsFirstPicInCvsFlag is equal to 1.

[0200] - Otherwise, PicRefreshFlag is set equal to 0.

[0201] Otherwise (the current tile group does not belong to the first access unit of the bitstream in decoding order and the group is not an IRAP tile group), PicRefreshFlag is set equal to 0.

[0202] Note that when sps_mixed_tile_groups_in_pic_flag is equal to 0, the value of poc_msb_reset_flag is ignored.

[0203] It is a bitstream conformance requirement that the following constraints apply when sps_mixed_tile_groups_in_pic_flag is equal to 1:

[0204] - If all tile groups of the current picture are IDR tile groups, the value of poc_msb_reset_flag shall be equal to 1 for all IDR tile groups of the current picture.

[0205] - If the current picture contains an IDR tile group and at least one tile group that is not an IDR tile group, the value of poc_msb_reset_flag shall be equal to 0.

[0206] NOTE – A value of poc_msb_reset_flag equal to 1 indicates that if there are multiple tile groups in the current picture, all of the tile groups are IDR tile groups.

[0207] NOTE - When an MCTS is extracted to be a sub-bitstream from an original bitstream with sps_mixed_tile_groups_in_pic_flagh equal to 1, the value of sps_mixed_tile_groups_in_pic_flag in the active SPS of the extracted sub-bitstream must be set equal to 0, that is, in the extracted sub-bitstream, each picture must contain only IRAP tile groups or only non-IRAP tile groups. In other words, tile groups belonging to one picture and to one MCTS must contain the same NAL unit type.

[0208] The tile group decoding process is now discussed.

[0209] The decoding process operates as follows for the current picture, CurrPic.

[0210] 1. The decoding of NAL units is specified in the NAL unit decoding process clause below.

[0211] 2. The process in the section on tile group decoding process uses syntax elements at the tile group header layer and above to specify the following decoding process.

[0212] As specified in the section on Picture Order Count Decoding Process below, variables and functions related to Picture Order Count are derived, which need to be called only for the first tile group of a picture.

[0213] [ Non-IDR pictures ] At the start of the decoding process for each tile group, a decoding process for reference picture list construction is invoked to derive reference picture list 0 (RefPicList[0]) and reference picture list 1 (RefPicList[1]).

[0214] - The decoding process for reference picture marking is invoked, where reference pictures can be marked as "not used for reference" or "used for long-term reference". This only needs to be invoked for the first tile group of a picture.

[0215] 3. Invoke the decoding process for tree-unit coding, scaling, transformation, in-loop filtering, etc.

[0216] 4. After all slices of the current picture are decoded, the decoded current picture is marked as "used for short-term reference".

[0217] The NAL unit decoding process is discussed.

[0218] The input to this process is the NAL units of the current picture and their associated non-VCL NAL units.

[0219] The output of this process is a parsed RBSP syntax structure encapsulated within a NAL unit.

[0220] The decoding process for each NAL unit extracts the RBSP syntax structure from the NAL unit and then parses the RBSP syntax structure.

[0221] The tile group decoding process is now discussed.

[0222] A decoding process for picture order counting is provided.

[0223] The output of this process is the picture order count of the current picture, PicOrderCntVal.

[0224] The picture order count is used to identify pictures, to derive motion parameters in merge mode and motion vector prediction, and for decoder conformance checks.

[0225] Each coded picture is associated with a picture order count variable, denoted as PicOrderCntVal.

[0226] { PicRefreshFlag equals 0 } In case, [ The current picture is not an IRAP picture. ] , the variables prevPicOrderCntLsb and prevPicOrderCntMsb are derived as follows. - Let prevTid0Pic be the previous picture in decoding order with TemporalID equal to 0. - The variable prevPicOrderCntLsb is set equal to the slice_pic_order_cnt_lsb of prevTid0Pic. - The variable prevPicOrderCntMsb is set equal to the PicOrderCntMsb of prevTid0Pic. The variable PicOrderCntMsb for the current picture is derived as follows: - { PicRefreshFlag equals 1 } In case, [ The current picture is an IRAP picture. ] , PicOrderCntMsb is set equal to 0. Otherwise, PicOrderCntMsb is derived as follows: if((slice_pic_order_cnt_lsb < prevPicOrderCntLsb)&& ((prevPicOrderCntLsb - slice_pic_order_cnt_lsb)> =(MaxPicOrderCntLsb / 2))) PicOrderCntMsb = prevPicOrderCntMsb + MaxPicOrderCntLsb(8 1) else if((slice_pic_order_cnt_lsb > prevPicOrderCntLsb)&& ((slice_pic_order_cnt_lsb - prevPicOrderCntLsb) > (MaxPicOrderCntLsb / 2))) PicOrderCntMsb = prevPicOrderCntMsb - MaxPicOrderCntLsb

[0227] Otherwise, PicOrderCntMsb = prevPicOrderCntMsb PicOrderCntVal is derived as follows: PicOrderCntVal = PicOrderCntMsb + slice_pic_order_cnt_lsb(8-2)

[0228] [ NOTE 1 - Since slice_pic_order_cnt_lsb of IRAP pictures is inferred to be 0 and prevPicOrderCntLsb and prevPicOrderCntMsb are both set to 0, PicOrderCntVal of all IRAP pictures will be set equal to 0. ]

[0229] The value of PicOrderCntVal is -2 31 ~2 31 Must be in the range -1 (inclusive). In one CVS, the PicOrderCntVal values ​​of any two coded pictures must not be the same.

[0230] At any point during the decoding process, the values ​​of PicOrderCntVal & (MaxLtPicOrderCntLsb -1) for any two reference pictures in the DPB must not be the same.

[0231] The function PicOrderCnt(picX) is specified as follows:

[0232] PicOrderCnt(picX) = PicOrderCntVal(8-3) of picture picX

[0233] The function DiffPicOrderCnt(picA, picB) is specified as follows:

[0234] DiffPicOrderCnt(picA, picB) = PicOrderCnt(picA) - PicOrderCnt(picB)(8-4)

[0235] The bit stream is -2 15 ~2 15 It must not contain data that would result in a value of DiffPicOrderCnt(picA, picB) being used in the decoding process that is not in the range -1 (inclusive).

[0236] NOTE 2 - Let X be the current picture, and Y and Z be two other pictures in the same CVS, then Y and Z are considered to be in the same output order direction from X if DiffPicOrderCnt(X,Y) and DiffPicOrderCnt(X,Z) are both positive or both negative.

[0237] A decoding process for reference picture list construction is provided.

[0238] This process is [ For non-IRAP pictures ] Called at the start of the decoding process for each tile group.

[0239] A reference picture is addressed via a reference index, which is an index into a reference picture list. When decoding an I tile group, no reference picture list is used to decode the tile group data. When decoding a P tile group, only reference picture list 0 (i.e., RefPicList[0]) is used to decode the tile group data. When decoding a B tile group, both reference picture list 0 and reference picture list 1 (i.e., RefPicList[1]) are used to decode the tile group data.

[0240] At the beginning of the decoding process for each tile group of a non-IRAP picture (shown in strikethrough), reference picture lists RefPicList[0] and RefPicList[1] are derived. The reference picture lists are used for marking reference pictures or decoding tile group data.

[0241] Note 1- [ Non-IRAP tile group ] For I tile groups, RefPicList[0] and RefPicList[1] may be derived for the purposes of bitstream conformance checking, but their derivation is not required for decoding the current picture or pictures that follow the current picture in decoding order. For P tile groups, RefPicList[1] may be derived for the purposes of bitstream conformance checking, but its derivation is not required for decoding the current picture or pictures that follow the current picture in decoding order.

[0242] The reference picture lists RefPicList[0] and RefPicList[1] are constructed as follows. for( i = 0 ; i < 2 ; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] for( j = 0, pocBase = PicOrderCntVal; j < NumEntriesInList[ i ][ RplsIdx[ i ] ]; j++) { (8-5) if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ] ][ j ] ) { RefPicPocList[ i ][ j ] = pocBase - DeltaPocSt[ i ][ RplsIdx[ i ] ][ j ] if( there is a reference picture picA in the DPB with PicOrderCntVal equal to RefPicPocList[ i ][ j ] ) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = “no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( there is a reference picA in the DPB with PicOrderCntVal & ( MaxLtPicOrderCntLsb - 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ] ][ j ] ) RefPicList[ i ][ j ] = picA else RefPicList[ i ][ j ] = “no reference picture” } } }

[0243] When each i is equal to 0 or 1, the following applies. - The first NumRefIdxActive[i] entries of RefPicList[i] are called the active entries of RefPicList[i], and the other entries of RefPicList[i] are called the inactive entries of RefPicList[i]. - Each entry in RefPicList[i][j] for j in the range 0 to NumEntriesInList[i][RplsIdx[i]] -1 (inclusive) is called a STRP entry if lt_ref_pic_flag[i][RplsIdx[i]][j] is equal to 0, and an LTRP entry otherwise.

[0244] NOTE 2 - A particular picture may be referenced by both an entry in RefPicList[0] and an entry in RefPicList[1]. It is also possible that a particular picture may be referenced by multiple entries in RefPicList[0] or multiple entries in RefPicList[1].

[0245] NOTE 3 - The active entries of RefPicList[0] and the active entries of RefPicList[1] collectively refer to all reference pictures that can be used for inter-prediction of the current picture and one or more pictures that follow the current picture in decoding order. The inactive entries of RefPicList[0] and the inactive entries of RefPicList[1] collectively refer to all reference pictures that are not used for inter-prediction of the current picture, but that may be used for inter-prediction of one or more pictures that follow the current picture in decoding order.

[0246] NOTE 4 - There may be one or more entries in RefPicList[0] or RefPicList[1] equal to "No Reference Picture" because the corresponding picture is not present in the DPB. Each inactive entry in RefPicList[0] or RefPicList[0] equal to "No Reference Picture" MUST be ignored. For each active entry in RefPicList[0] or RefPicList[1] equal to "No Reference Picture", an unintended picture loss MUST be inferred.

[0247] It is a requirement of bitstream conformance that the following constraints apply:

[0248] For each i equal to 0 or 1, NumEntriesInList[i][RplsIdx[i]] must be greater than or equal to NumRefIdxActive[i].

[0249] - The picture referenced by each active entry in RefPicList[0] or RefPicList[1] must be present in the DPB and have a TemporalID that is less than or equal to the TemporalID of the current picture.

[0250] - A STRP entry in RefPicList[0] or RefPicList[1] of a slice of a picture and a LTRP entry in RefPicList[0] or RefPicList[1] of the same slice or of a different slice of the same picture must not reference the same picture.

[0251] - The difference between PicOrderCntVal of the current picture and PicOrderCntVal of the picture referenced by the entry is 2 24 There must be no LTRP entries in RefPicList[0] or RefPicList[1] that are greater than or equal to this number.

[0252] Let setOfRefPics be the set of unique pictures referenced by all entries of RefPicList[0] and all entries of RefPicList[1]. The number of pictures in setOfRefPics must be less than or equal to sps_max_dec_pic_buffering_minus1, and setOfRefPics must be the same for all slices of a picture.

[0253] A decoding process for reference picture marking is provided.

[0254] This process is invoked once per picture after the decoding of the tile group header and the decoding processes for constructing the reference picture list of the tile group, but before the decoding of the tile group data. This process allows one or more reference pictures in the DPB to be marked as "not used for reference" or "used for long term reference".

[0255] A decoded picture in the DPB can be marked as "not used for reference", "used for short-term reference", or "used for long-term reference", but only one of these three can be marked at any given time during the operation of the decoding process. Assigning one of these markings to a picture implicitly removes the other of these markings, if applicable. When a picture is referred to as being marked "used for reference", this refers collectively to pictures that are marked as "used for short-term reference" or "used for long-term reference" (but not both).

[0256] [ If the current picture is an IRAP picture, all reference pictures currently in the DPB (if any) are marked as "unused for reference". ]

[0257] STRPs are identified by their PicOrderCntVal values. LTRPs are identified by the Log2(MaxLtPicOrderCntLsb) LSBs of their PicOrderCntVal values.

[0258] The following applies:

[0259] - { When PicRefreshFlag is equal to 1, all reference pictures in the DPB are marked as "unused for reference". }

[0260] - { Otherwise (sps_mixed_tile_groups_in_pic_flag is equal to 1 or the current tile group is not an IDR tile group), the following applies: }

[0261] { For each reference picture in the DPB marked as "used for short-term reference", if the picture is referenced by an LTRP entry in RefPicList[0] or RefPicList[1], the reference picture is marked as "used for long-term reference." }

[0262] Each reference picture in the DPB that is not referenced by any entry in RefPicList[0] or RefPicList[1] is marked as "unused for reference."

[0263] The concepts of the present disclosure will now be discussed further.

[0264] In order to solve the above problems, the following aspects are disclosed, each of which can be applied individually, and some of which can be applied in combination.

[0265] 1) If a picture has multiple sub-pictures, it is allowed to contain both IRAP and non-IRAP sub-pictures.

[0266] A sub-picture within a picture can be a slice, a tile group, an MCTS, or any other subset of a picture.

[0267] b. A sub-picture is usually, but not always, transmitted exclusively in its own NAL unit.

[0268] 2) Information for MCTS description may be present / signaled in a parameter set, a tile group header, or a supplemental enhancement information (SEI) message.

[0269] 3) Alternatively, item 1) can be expressed such that when a coded picture is transmitted in multiple NAL units, one or more of those NAL units can be an IRAP NAL unit type and one or more of those NAL units can be a non-IRAP NAL unit type (trailing NAL unit type).

[0270] 4) If a picture has multiple sub-pictures and the sub-pictures are a mix of IRAP and non-IRAP sub-pictures, the picture must have TemporalID equal to 0.

[0271] 5) If a picture has multiple sub-pictures and the sub-pictures are a mix of IRAP and non-IRAP sub-pictures, the IRAP sub-pictures may need to be part of the MCTS.

[0272] 6) If a picture has multiple sub-pictures and the sub-pictures are a mix of IRAP and non-IRAP sub-pictures, an access unit delimiter needs to be present in the bitstream and associated with the picture to help the system / application easily identify the access units.

[0273] 7) There is a flag in the parameter set referenced directly or indirectly by the tile group to specify whether there can be pictures that contain mixed IRAP and non-IRAP sub-pictures.

[0274] The flag can be signaled in a parameter set, such as a sequence parameter set, a picture parameter set, or another type of parameter set referenced directly or indirectly by a tile group. In particular, signaling of the flag in a sequence parameter set may be preferred.

[0275] b. The flag may be called sps_mixed_tile_groups_in_pic_flag.

[0276] 8) For a NAL unit that contains an IDR tile group, a flag exists in its tile group header to specify whether the POC MSB is reset in the POC derivation of the picture.

[0277] 9) A variable called PicRefreshFlag is defined and associated with a picture. This flag specifies whether the derivation of the POC and the state of the DPB need to be refreshed when decoding the picture.

[0278] 10) The value of PicRefreshFlag is derived as follows:

[0279] If the current tile group belongs to the first access unit of the bitstream, PicRefreshFlag is set equal to 1.

[0280] b. Otherwise, if the current tile group is an IDR tile group, then PicRefreshFlag is set equal to sps_mixed_tile_groups_in_pic_flag ? poc_msb_reset_flag:1.

[0281] c. Otherwise, if the current tile group is a CRA tile group, the following applies:

[0282] i. If the current access unit is the first access unit of a coded sequence (i.e., if it immediately follows the end NAL unit of the sequence or if the associated variable HandleCraAsFirstPicInCvsFlag is equal to 1), then PicRefreshFlag is set equal to 1.

[0283] ii. Otherwise, PicRefreshFlag is set equal to 0.

[0284] d. Otherwise (the current tile group does not belong to the first access unit of the bitstream and is not an IRAP tile group), PicRefreshFlag is set equal to 0.

[0285] 11) If PicRefreshFlag is equal to 1, during derivation of the POC for a picture, the value of the POC MSB (i.e., PicOrderCntMsb) is reset (i.e., set equal to 0).

[0286] 12) When sps_mixed_tile_group_in_pic_flag is equal to 1, the following constraints apply:

[0287] a. If all tile groups of the current picture are IDR tile groups, then the value of poc_msb_reset_flag shall be equal to 1 for all IDR tile groups of the current picture.

[0288] b. If the current picture contains an IDR tile group and at least one tile group that is not an IDR tile group, the value of poc_msb_reset_flag shall be equal to 0.

[0289] 13) When an MCTS is extracted to be a sub-bitstream from an original bitstream with sps_mixed_tile_groups_in_pic_flag equal to 1, the value of sps_mixed_tile_groups_in_pic_flag of the active SPS of the extracted sub-bitstream shall be set equal to 0, that is, in the extracted sub-bitstream, each picture shall contain only IRAP tile groups or only non-IRAP tile groups. In other words, it is required that the tile groups belonging to one picture and to one MCTS shall contain the same NAL unit type.

[0290] 14) Information required for reference picture management, such as Reference Picture Set (RPS) or Reference Picture List (RPL), is signaled in the tile group header regardless of its NAL unit type. Alternatively, such information may be signaled only in the picture header (if a picture header is present).

[0291] 15) Reference picture lists (e.g., RefPicList[0] and RefPicList[1] in the case of the RPL approach, or RefPicList0[] and RefPicList1[] in the case of the RPS approach, or similar lists containing reference pictures for inter-prediction operations of a picture) are constructed at the start of decoding of each tile group, regardless of its NAL unit type.

[0292] 16) The reference picture marking process is performed by checking whether each reference picture in the DPB is referenced by an entry in the signaled reference picture management information (e.g., RefPicList[0] and RefPicList[1] in the case of the RPL approach, or the RPS subset in the case of the RPS approach).

[0293] 17) When PicRefreshFlag is equal to 1, during the reference picture marking process, all reference pictures in the DPB are marked as "unused for reference".

[0294] 18) Details of the reference picture marking process for the RPL approach are as follows.

[0295] If PicRefreshFlag is equal to 1, all reference pictures in the DPB are marked as "unused for reference".

[0296] b. Otherwise, the following applies:

[0297] i. For each reference picture in the DPB marked as "used for short-term reference", the reference picture is marked as "used for long-term reference" if it is referenced by an LTRP entry in RefPicList[0] or RefPicList[1].

[0298] ii. Each reference picture in the DPB that is not referenced by any entry in RefPicList[0] or RefPicList[1] is marked as "unused for reference".

[0299] FIG. 9 is a schematic diagram of a video coding device 900 (e.g., video encoder 20 or video decoder 30) according to one embodiment of the present disclosure. The video coding device 900 is suitable for implementing the disclosed embodiments described herein. The video coding device 900 includes an input port 910 and a receiver unit (Rx) 920 for receiving data; a processor, logic unit, or central processing unit (CPU) 930 for processing data; a transmitter unit (Tx) 940 and an output port 950 for transmitting data; and a memory 960 for storing data. The video coding device 900 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the input port 910, receiver unit 920, transmitter unit 940, and output port 950 for input or output of optical or electrical signals.

[0300] The processor 930 is implemented by hardware and software. The processor 930 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 930 is in communication with the input port 910, the receiver unit 920, the transmitter unit 940, the output port 950, and the memory 960. The processor 930 includes a coding module 970. The coding module 970 implements the above disclosed embodiments. For example, the coding module 970 implements, processes, prepares, or provides various networking functions. Thus, the inclusion of the coding module 970 provides a substantial improvement to the functionality of the video coding device 900 and results in the transformation of the video coding device 900 into different states. Alternatively, the coding module 970 is implemented as instructions stored in the memory 960 and executed by the processor 930.

[0301] Video coding device 900 may also include input and / or output (I / O) devices 980 for communicating data to and from a user. I / O devices 980 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. I / O devices 980 may include input devices such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0302] Memory 960 may include one or more disks, tape drives, and solid state drives, may be used as overflow data storage, may store programs when such programs are selected for execution, and may store instructions and data retrieved during the execution of programs. Memory 960 may be volatile and / or non-volatile, and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0303] 10 is a schematic diagram of one embodiment of a means for coding 1000. In an embodiment, the means for coding 1000 is implemented in a video coding device 1002 (e.g., video encoder 20 or video decoder 30). The video coding device 1002 includes a receiving means 1001. The receiving means 1001 is configured to receive a picture to be encoded or to receive a bitstream to be decoded. The video coding device 1002 includes a transmitting means 1007 coupled to the receiving means 1001. The transmitting means 1007 is configured to transmit the bitstream to a decoder or transmit a decoded image to a display means (e.g., one of the I / O devices 980).

[0304] The video coding device 1002 includes a storage means 1003. The storage means 1003 is coupled to at least one of the receiving means 1001 or the transmitting means 1007. The storage means 1003 is configured to store instructions. The video coding device 1002 also includes a processing means 1005. The processing means 1005 is coupled to the storage means 1003. The processing means 1005 is configured to execute the instructions stored in the storage means 1003 to perform the methods disclosed herein.

[0305] It should also be understood that the steps of the exemplary methods described herein do not necessarily have to be performed in the order described, and the order of steps in such methods should be understood to be merely exemplary. Similarly, additional steps may be included in such methods, and certain steps may be omitted or combined, in methods consistent with various embodiments of the present disclosure.

[0306] Although the present disclosure provides several embodiments, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples should be considered as illustrative and not limiting, and the intention should not be limited to the details given herein. For example, various elements or components may be combined or integrated into another system, or certain features may be omitted, or not implemented.

[0307] Additionally, the techniques, systems, subsystems, and methods described and illustrated in various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items illustrated or discussed as coupled or directly coupled or in communication with one another may be indirectly coupled or in communication electrically, mechanically, or otherwise, through some interface, device, or intermediate component. Other examples of modifications, substitutions, and alterations are ascertainable by those skilled in the art and may be made without departing from the spirit and scope of the present disclosure.

Claims

1. 1. A method for transmitting a bitstream, the method comprising: Storing at least one video bitstream in at least one storage device, the encoded video bitstream (600) of the at least one video bitstream comprising: encoded data for a mixed picture (500) including a first sub-picture (502) and a second sub-picture (504), the first sub-picture (502) being an Intra Random Access Point (IRAP) sub-picture and the second sub-picture (504) being a non-IRAP sub-picture, the first sub-picture being the IRAP sub-picture contained within a first set of Network Abstraction Layer (NAL) units having an Instantaneous Decode Refresh (IDR) Network Abstraction Layer (NAL) unit type or a Clean Random Access (CRA) NAL unit type, and the second sub-picture being the non-IRAP sub-picture contained within a second set of NAL units having a trailing NAL unit type; encoding data of a reference picture list (RPL) of the second sub-picture; obtaining one or more bitstreams from the at least one storage device; transmitting the one or more bitstreams to a destination device. method.

2. 1. A method for distributing a video bitstream, the method comprising: a video streaming device obtaining the video bitstream from a storage medium, the video bitstream being stored on the storage medium; and encoded data for a mixed picture (500) including a first sub-picture (502) and a second sub-picture (504), the first sub-picture (502) being an Intra Random Access Point (IRAP) sub-picture and the second sub-picture (504) being a non-IRAP sub-picture, the first sub-picture being the IRAP sub-picture contained within a first set of Network Abstraction Layer (NAL) units having an Instantaneous Decode Refresh (IDR) Network Abstraction Layer (NAL) unit type or a Clean Random Access (CRA) NAL unit type, and the second sub-picture being the non-IRAP sub-picture contained within a second set of NAL units having a trailing NAL unit type; encoding data of a reference picture list (RPL) of the second sub-picture; the video streaming device transmitting the bitstream to a terminal device, the video streaming device including a content service server or a content distribution server; method.

3. The method of claim 1 , wherein the encoded data of the mixed picture is carried in a split bitstream comprising a first sub-bitstream (602) and a second sub-bitstream (604).

4. The method of claim 1 , wherein the first sub-picture is placed in a first sub-bitstream and the second sub-picture is placed in a second sub-bitstream.

5. The method according to claim 1 , wherein the IRAP sub-picture is an IDR picture.

6. The method of claim 1 , wherein a flag is present in the bitstream, the flag indicating whether the bitstream contains mixed pictures.

7. The method of claim 6 , wherein the flag is in a Picture Parameter Set (PPS) of the bitstream.

8. The method according to claim 1 , wherein the IRAP sub-picture is a CRA picture.

9. 1. An apparatus for transmitting a bitstream, the apparatus comprising: at least one storage device configured to store at least one bitstream; at least one processor configured to obtain a target bitstream from one of the at least one storage devices and transmit the target bitstream to a destination device; The at least one bitstream bitstream comprises: encoded data for a mixed picture including a first sub-picture and a second sub-picture, the first sub-picture being an Intra Random Access Point (IRAP) sub-picture and the second sub-picture being a non-IRAP sub-picture, the first sub-picture being the IRAP sub-picture contained within a first set of Network Abstraction Layer (NAL) units having an Instantaneous Decode Refresh (IDR) Network Abstraction Layer (NAL) unit type or a Clean Random Access (CRA) NAL unit type, and the second sub-picture being the non-IRAP sub-picture contained within a second set of NAL units having a trailing NAL unit type; and coding data of a reference picture list (RPL) of the second sub-picture. Device.

10. 1. A video streaming device, comprising:

1. A storage medium configured to store at least one bitstream, the at least one bitstream comprising: encoded data for a mixed picture including a first sub-picture and a second sub-picture, the first sub-picture being an Intra Random Access Point (IRAP) sub-picture and the second sub-picture being a non-IRAP sub-picture, the first sub-picture being the IRAP sub-picture contained within a first set of Network Abstraction Layer (NAL) units having an Instantaneous Decode Refresh (IDR) Network Abstraction Layer (NAL) unit type or a Clean Random Access (CRA) NAL unit type, and the second sub-picture being the non-IRAP sub-picture contained within a second set of NAL units having a trailing NAL unit type; the storage medium including encoded data of a reference picture list (RPL) of the second sub-picture; at least one processor configured to retrieve the bitstream from the storage medium and transmit the bitstream to a terminal device; Video streaming device.

11. A non-transitory computer readable medium carrying program code which, when executed by a computing device, causes the computing device to perform the method of any one of claims 1 to 8.

12. A computer program comprising a program code for performing the method according to any one of claims 1 to 8, when the computer program is run on a computer or processor.

13. A non-transitory computer-readable recording medium storing a bitstream, comprising: The bitstream includes: encoded data for a mixed picture including a first sub-picture and a second sub-picture, the first sub-picture being an Intra Random Access Point (IRAP) sub-picture and the second sub-picture being a non-IRAP sub-picture, the first sub-picture being the IRAP sub-picture contained within a first set of Network Abstraction Layer (NAL) units having an Instantaneous Decode Refresh (IDR) Network Abstraction Layer (NAL) unit type or a Clean Random Access (CRA) NAL unit type, and the second sub-picture being the non-IRAP sub-picture contained within a second set of NAL units having a trailing NAL unit type; and coding data of a reference picture list (RPL) of the second sub-picture; When information processing is performed, A decoding process performed by a video decoder based on the bitstream, decoding the second sub-picture using the RPL; and generating the mixed picture based on the decoded second sub-picture. A non-transitory computer-readable recording medium.

Images