In video coding, the output of the previous picture for the picture that starts the newly coded video sequence.

Abstract

The present invention provides a method for decoding a coded video bitstream, a decoding device, and a computer program. [Solution] The method includes the steps of receiving a video bitstream being coded, the coded video bitstream including a first flag having a first value and a clean random access (CRA) picture; setting a second value of a second flag to be equal to a first value of the first flag; and having a video decoder empty a buffer of pictures being coded (DPB) of all previously coded pictures based on a second flag having a second value; and decoding the current picture after the DPB has been emptied.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This patent application claims priority based on U.S. Provisional Patent Application No. 62 / 843,991, titled "Output of Previous Picture for a Picture Starting a Newly Encoded Video Sequence in Video Coding," filed on May 6, 2019, by Ye - Kui Wang, the content of which is hereby incorporated by reference in its entirety.

[0002] [Technical Field] Generally, this disclosure describes a plurality of techniques for supporting the output of previously decoded pictures during video coding. More specifically, this disclosure enables outputting a previously decoded picture corresponding to a random access point picture that starts a coded video sequence (CVS) from a buffer of decoded pictures.

Background Art

[0003] Even the amount of video data required to depict a relatively short video can be substantial, and this amount of video data can be problematic when the data is streamed or, in other cases, transmitted over communication networks with limited bandwidth. Thus, video data is generally compressed before being transmitted over modern communication networks. Because memory resources can be limited, the size of the video can also be a problem when storing it on a storage device. Video compression devices typically encode the video data at the source using software and / or hardware before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. Given limited network resources and the ever-increasing demand for higher video quality, improved compression and decompression techniques that improve the compression ratio with little to no sacrifice in image quality are desirable. [Overview of the project]

[0004] The first aspect relates to a decoding method implemented by a video decoder, the method comprising: receiving a coded video bitstream by the video decoder, the coded video bitstream comprising a clean random access (CRA) picture and a first flag having a first value; setting a second value of a second flag by the video decoder to be equal to the first value of the first flag; clearing a buffer (DPB) of previously decoded pictures by the video decoder based on the second flag having the second value; and decoding the current picture by the video decoder after the DPB has been cleared.

[0005] When encountering random access point pictures other than instantaneous decoder refresh (IDR) pictures (e.g., clean random access (CRA) pictures, sequential random access (GRA) pictures, or sequential decoded refresh (GDR) pictures, CVSS pictures, etc.) in the decoding sequence, the method provides a technique for outputting previous pictures (e.g., previously decoded pictures) from the buffer (DPB) of the picture being decoded. Emptying previously decoded pictures from the DPB when reaching a random access point picture prevents the DPB from overflowing and promotes more continuous playback. Thus, the coder / decoder (also known as the "codec") in video coding is an improvement compared to current codecs. In practical terms, when video is transmitted, received, and / or viewed, the improved video coding process provides users with a better user experience.

[0006] Optionally, in any of several prior embodiments, other implementations of that embodiment provide that the CRA picture is not the first picture in the coded video bitstream.

[0007] Optionally, in any of a plurality of prior embodiments, another implementation of that embodiment provides a step of setting the DPB fullness parameter to 0 when the first flag is set to the first value.

[0008] Optionally, in any of several prior embodiments, other implementations of that embodiment provide that the first flag is specified as no_output_of_prior_pics_flag and the second flag is specified as NoOutputOfPriorPicsFlag.

[0009] Optionally, in any of several prior embodiments, other implementations of that embodiment provide that the DPB is emptied after the CRA picture is decoded.

[0010] Optionally, in any of a plurality of prior embodiments, another implementation of that embodiment provides a step of displaying an image generated based on the current picture.

[0011] A second aspect relates to an encoding method implemented by a video encoder. The method includes the steps of: the video encoder determining a random access point for a video sequence; the video encoder encoding a clean random access (CRA) picture to the video sequence at the random access point; the video encoder setting a flag to a first value to instruct a video decoder to empty a buffer of decoded pictures (DPB) of pictures being decoded from any previously decoded pictures; the video encoder generating a video bitstream containing the video sequence having the CRA picture at the random access point and the flag; and the video encoder storing the video bitstream for transmission to the video decoder.

[0012] When encountering random access point pictures other than instantaneous decoder refresh (IDR) pictures (e.g., clean random access (CRA) pictures, sequential random access (GRA) pictures, or sequential decoded refresh (GDR) pictures, CVSS pictures, etc.) in the decoding sequence, the method provides a technique for outputting previous pictures (e.g., previously decoded pictures) from the buffer (DPB) of the picture being decoded. Emptying previously decoded pictures from the DPB when reaching a random access point picture prevents the DPB from overflowing and promotes more continuous playback. Thus, the coder / decoder (also known as the "codec") in video coding is an improvement compared to current codecs. In practical terms, when video is transmitted, received, and / or viewed, the improved video coding process provides users with a better user experience.

[0013] Optionally, in any of a plurality of prior embodiments, other implementations of that embodiment provide that the CRA picture is not the first picture in the video bitstream, and that the video decoder is instructed to empty the DPB after the CRA picture has been decoded.

[0014] Optionally, in any of a plurality of prior embodiments, another implementation of that embodiment provides a step of instructing the video decoder to set the DPB fullness parameter to 0 when the flag is set to the first value.

[0015] Optionally, in any of several prior embodiments, other implementations of that embodiment provide that the flag is specified as no_output_of_prior_pics_flag.

[0016] Optionally, in any of a plurality of prior embodiments, other implementations of that embodiment provide that the first value of the flag is 1.

[0017] A third aspect relates to a decoding device. The decoding device includes a receiver configured to receive a video bitstream being coded, and a memory coupled to the receiver, the memory comprising a memory for storing instructions, and a processor coupled to the memory, the processor being configured to execute the instructions to cause the decoding device to receive the video bitstream being coded, the video bitstream being coded comprising a clean random access (CRA) picture and a first flag having a first value, set a second value of a second flag to be equal to the first value of the first flag, and based on the second flag having the second value, to empty any previously decoded pictures from a buffer (DPB) of decoded pictures, and then decode the current picture after the DPB is empty.

[0018] When encountering a random access point picture other than an instantaneous decoder refresh (IDR) picture (e.g., a clean random access (CRA) picture, a sequential random access (GRA) picture, or a sequential decoded refresh (GDR) picture, CVSS picture, etc.) in the decoding sequence, the decoding device provides a technique for outputting the previous picture (e.g., a previously decoded picture) from the buffer (DPB) of the picture being decoded. Emptying the DPB with previously decoded pictures when reaching a random access point picture prevents the DPB from overflowing and facilitates more continuous playback. Thus, the coder / decoder (also known as a "codec") in video coding is an improvement compared to current codecs. In practical terms, when video is transmitted, received, and / or viewed, an improved video coding process provides users with a better user experience.

[0019] Optionally, in any of several prior embodiments, other implementations of that embodiment provide that the CRA picture is not the first picture in the coded video bitstream.

[0020] Optionally, in any of several prior embodiments, other implementations of that embodiment provide that the first flag is specified as no_output_of_prior_pics_flag and the second flag is specified as NoOutputOfPriorPicsFlag.

[0021] Optionally, in any of a plurality of prior embodiments, another implementation of that embodiment provides a display configured to display an image generated based on the current picture.

[0022] A fourth aspect relates to an encoding device. The encoding device includes a memory for storing instructions; a processor coupled to the memory, the processor being configured to implement the instructions, causing the encoding device to determine a random access point for a video sequence, encode a clean random access (CRA) picture into the video sequence at the random access point, set a flag to a first value, instruct a video decoder to empty any previously decoded pictures from a buffer of decoded pictures (DPB), and generate the video bitstream containing the video sequence having the CRA picture at the random access point and the flag; and a transmitter coupled to the processor, the transmitter being configured to transmit the video bitstream to the video decoder.

[0023] When encountering random access point pictures other than instantaneous decoder refresh (IDR) pictures (e.g., clean random access (CRA) pictures, sequential random access (GRA) pictures, or sequential decoded refresh (GDR) pictures, CVSS pictures, etc.) in the decoding sequence, the encoding device provides a technique for outputting previous pictures (e.g., previously decoded pictures) from the buffer of the picture being decoded (DPB). Emptying the DPB with previously decoded pictures when reaching a random access point picture prevents the DPB from overflowing and facilitates more continuous playback. Thus, the coder / decoder (also known as the "codec") in video coding is an improvement over current codecs. In practical terms, when video is transmitted, received, and / or viewed, the improved video coding process provides users with a better user experience.

[0024] Optionally, in any of the plurality of preceding aspects, other implementations of that aspect provide that the CRA picture is not the first picture of the video bitstream.

[0025] Optionally, in any of the plurality of preceding aspects, other implementations of that aspect provide that the flag is specified as the no_output_of_prior_pics_flag.

[0026] Optionally, in any of the plurality of preceding aspects, other implementations of that aspect provide that the memory stores the bitstream before the transmitter transmits the bitstream to the video decoder.

[0027] A fifth aspect relates to a coding apparatus. The coding apparatus includes a receiver configured to receive a picture for encoding or a bitstream for decoding, a transmitter coupled to the receiver, the transmitter being configured to transmit the bitstream to a decoder or transmit a decoded image to a display, a memory coupled to at least one of the receiver or the transmitter, the memory being configured to store instructions, and a processor coupled to the memory, the processor being configured to execute the instructions stored in the memory to perform any of the methods disclosed herein.

[0028] When encountering in decoding order random access point pictures other than instantaneous decoder refresh (IDR) pictures (e.g., clean random access (CRA) pictures, gradual random access (GRA) pictures, or gradual decoding refresh (GDR) pictures, CVSS pictures, etc.), the coding device provides techniques for output of previous pictures (e.g., previously decoded pictures, etc.) in the buffer of decoded pictures (DPB). Emptying the previously decoded pictures from the DPB when reaching a random access point picture prevents the DPB from causing an overflow and promotes more continuous playback. Thus, the coder / decoder (also known as the "codec") in video coding is improved compared to the current codec. As a practical matter, when video is transmitted, received, and / or viewed, the improved video coding process provides a better user experience to the user.

[0029] Optionally, in any of the plurality of preceding aspects, other implementations of that aspect provide a display configured to display an image.

[0030] A sixth aspect relates to a system. The system includes an encoder and a decoder that communicates with the encoder, and the encoder or the decoder includes a decoding device, an encoding device, or a coding device disclosed herein.

[0031] When encountering a random access point picture other than an instantaneous decoder refresh (IDR) picture (e.g., a clean random access (CRA) picture, a sequential random access (GRA) picture, or a sequential decoded refresh (GDR) picture, CVSS picture, etc.) in the decoding sequence, the system provides a technique for outputting the previous picture (e.g., a previously decoded picture) from the buffer of the picture being decoded (DPB). Emptying the DPB with previously decoded pictures when reaching a random access point picture prevents the DPB from overflowing and facilitates more continuous playback. Thus, the coder / decoder (also known as a "codec") in video coding is an improvement over current codecs. In practical terms, when video is transmitted, received, and / or viewed, an improved video coding process provides users with a better user experience.

[0032] A seventh aspect relates to means for coding. The means for coding includes a receiving means configured to receive a picture to encode or a bitstream to decode; a transmitting means coupled to the receiving means, the transmitting means configured to transmit the bitstream to a decoding means or an image being decoded to a display means; a storage means coupled to at least one of the receiving means or the transmitting means, the storage means configured to store instructions; and a processing means coupled to the storage means, the processing means configured to execute the instructions stored in the storage means to perform any of the methods disclosed herein.

[0033] When encountering random access point pictures other than instantaneous decoder refresh (IDR) pictures (e.g., clean random access (CRA) pictures, sequential random access (GRA) pictures, or sequential decoded refresh (GDR) pictures, CVSS pictures, etc.) in the decoding sequence, the means for coding provides a technique for outputting previous pictures (e.g., previously decoded pictures) from the buffer (DPB) of the picture being decoded. Emptying the DPB with previously decoded pictures when reaching a random access point picture prevents the DPB from overflowing and facilitates more continuous playback. Thus, the coder / decoder (also known as the "codec") in video coding is an improvement over current codecs. In practical terms, when video is transmitted, received, and / or viewed, the improved video coding process provides users with a better user experience.

[0034] For clarity, any one of the above embodiments may be combined with any one or more of the other embodiments described above to create new embodiments that fall within the scope of this disclosure.

[0035] These and other features will be more clearly understood from the following detailed description made in relation to the attached drawings and claims. [Brief explanation of the drawing]

[0036] To better understand this disclosure, the following brief descriptions are made in relation to the attached drawings and detailed descriptions, where similar reference numbers refer to similar parts.

[0037] [Figure 1] This is a block diagram illustrating one example coding system that can utilize GDR technology. [Figure 2]This is a block diagram illustrating one example video encoder that can implement GDR technology. [Figure 3] This is a block diagram illustrating one example of a video decoder that can implement GDR technology. [Figure 4] This represents the relationship between the IRAP picture corresponding to the reading picture and the last picture, shown in the decoding order, and the IRAP picture corresponding to the reading picture and the last picture, shown in the presentation order. [Figure 5] This diagram illustrates the gradual decoding refresh technique. [Figure 6] This is a schematic diagram illustrating the search for undesirable movements. [Figure 7] This diagram illustrates a video bitstream configured to implement Clean Random Access (CRA) technology. [Figure 8] This is one embodiment of a method for decoding a coded video bitstream. [Figure 9] This is one embodiment of a method for encoding a video bitstream that is being coded. [Figure 10] This is a schematic diagram of a video coding device. [Figure 11] Figure 11 is a schematic diagram of one embodiment of a means for coding. [Modes for carrying out the invention]

[0038] First, it should be understood that while the following provides one or more exemplary implementations of multiple embodiments, it is possible to implement the disclosed systems and / or methods using any number of techniques, whether currently known or existing. This disclosure is not to be limited in any way to the exemplary implementations, drawings, and techniques described below, including the multiple exemplary designs and implementations described and explained herein, but may be modified within the scope of the invention described in the appendix claims, along with the entire scope of the invention equivalent to the invention described in the claims.

[0039] Figure 1 is a block diagram showing an exemplary coding system 10, which is capable of utilizing the video coding techniques described herein. As shown in Figure 1, the coding system 10 includes a source device 12, which provides encoded video data to be decoded by a destination device 14 at a later time. In particular, the source device 12 may provide the video data to the destination device 14 via a computer-readable medium 16. The source device 12 and the destination device 14 may include any of a wide range of devices, which include desktop computers, notebook computers (e.g., laptop computers), tablet computers, set-top boxes, telephone handsets such as so-called "smartphones," so-called "smart" pads, televisions, cameras, display devices, digital media players, video game consoles, or video streaming devices. In some cases, the source device 12 and the destination device 14 may be capable of wireless communication.

[0040] The destination device 14 may receive encoded video data that is decoded via a computer-readable medium 16. The computer-readable medium 16 may include any type of medium or device that can move the encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable medium 16 may include a communication medium that enables the source device 12 to directly transmit the encoded video data to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard such as a wireless communication protocol and then transmitted to the destination device 14. The communication medium may include any wireless communication medium or wired communication medium such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or other equipment that can be useful in facilitating communication from the source device 12 to the destination device 14.

[0041] In some of the multiple examples, the data encoded may be output from the output interface 22 to a storage device. Similarly, the encoded data can be accessed from that storage device via the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, digital video disc (DVD), compact disc read-only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing the encoded video data. In further examples, the storage device may correspond to a file server or other intermediate storage device, which can store the encoded video generated by the source device 12. The destination device 14 can access the stored video data from the storage device by streaming or downloading. The file server may be any type of server, which can store the encoded video data and transmit the encoded video data to the destination device 14. An exemplary file server includes, for example, a web server (for a website), a File Transfer Protocol (FTP) server, a Network Attached Storage (NAS) device, or a local disk drive. The destination device 14 can access its encoded video data via any standard data connection, including an Internet connection. This standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem), or a combination of both a wireless channel and a wired connection suitable for accessing the encoded video data stored on the file server. Transmission of the encoded video data from the storage device may be streaming transmission, download transmission, or a combination thereof.

[0042] The technologies described herein are not necessarily limited to wireless applications or settings. These technologies can be applied to video coding supporting any of a variety of multimedia applications, such as wireless television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission including dynamic adaptive streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some of the examples, the coding system 10 may be configured to support applications such as video streaming, video playback, video broadcasting, and / or video telephony by supporting one-way or two-way video transmission.

[0043] In the example shown in Figure 1, the source device 12 includes a video source 18, a video encoder 20, and an output interface 22. The destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. According to this disclosure, the video encoder 20 of the source device 12 and / or the video decoder 30 of the destination device 14 may be configured to apply multiple techniques for video coding. In other examples, the source device and destination device may include other components or arrangements. For example, the source device 12 may receive video data from an external video source such as an external camera. Similarly, the destination device 14 may not include an integrated display device but may instead provide an interface to an external display device.

[0044] The coding system 10 illustrated in Figure 1 is merely one example. Techniques for video coding may be performed by any digital video coding device and / or decoding device. While some of these techniques in this disclosure are generally performed by a video coding device, some of these techniques may also be performed by a video encoder / decoder, typically referred to as a "CODEC". Furthermore, some of these techniques in this disclosure may also be performed by a video processor. The video encoder and / or video decoder may be a graphics processing unit or a similar device.

[0045] The source device 12 and destination device 14 are merely examples of multiple coding devices in which the source device 12 generates video data that is coded for transmission to the destination device 14. In some of these examples, the source device 12 and destination device 14 may operate in a substantially symmetrical manner, thereby each of the source device 12 and destination device 14 including a video coding component and a video decoding component. Thus, the coding system 10 can support one-way or two-way video transmission between multiple video devices 12 and 14 for, for example, video streaming, video playback, video broadcasting, or video phone calls.

[0046] The video source 18 of the source device 12 may include a video capture device such as a video camera, a video archive containing previously captured video, and / or a video supply interface for receiving video from a video content provider. As a further alternative, the video source 18 may generate computer graphics-based data as source video, or as a combination of live video, archived video, and computer-generated video.

[0047] In some cases, when the video source 18 is a video camera, the source device 12 and the destination device 14 may form a so-called cameraphone or videophone. On the other hand, as mentioned above, the technology described in this disclosure may generally be applicable to video coding and may be applicable to wireless and / or wired applications. In each case, the captured video, pre-captured video, or computer-generated video may be encoded by the video encoder 20. The encoded video information may then be output to a computer-readable medium 16 via the output interface 22.

[0048] The computer-readable medium 16 may include temporary media such as wireless broadcast transmission or wired network transmission, or storage media (i.e., non-temporary media) such as hard disks, flash drives, compact discs, digital video discs, Blu-ray discs, or other computer-readable media. In some of the multiple examples, a network server (not shown) may receive encoded video data from a source device 12 and provide the encoded video data to a destination device 14, for example, by network transmission. Similarly, a computing device in media manufacturing equipment, such as a disc engraving machine, may receive encoded video data from a source device 12 and manufacture a disc containing the encoded video data. Thus, the computer-readable medium 16 may be understood in various examples to include one or more of various forms of computer-readable media.

[0049] The input interface 28 of the destination device 14 receives information from the computer-readable medium 16. The information on the computer-readable medium 16 may include syntax information defined by the video encoder 20, which is also used by the video decoder 30, and which includes syntax elements that describe the features and / or processing of blocks and other coded units, such as multiple picture groups (GOPs). The display device 32 displays the decoded video data to the user and may include any of various display devices such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or other types of display devices.

[0050] The video encoder 20 and video decoder 30 may operate in accordance with a video coding standard, such as the High Efficiency Video Coding (HEVC) standard currently under development, and may conform to the HEVC Test Model (HM). Alternatively, the video encoder 20 and video decoder 30 may operate in accordance with other proprietary or industry standards, such as the Video Expert Group (MPEG) 4, Part 10, Advanced Video Coding (AVC), H.265 / HEVC, or the International Telecommunication Union Communications Standardization Sector (ITU-T) H.264 standard, which may be referred to as an extension of such standards. On the other hand, the various technologies of this disclosure are not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. Although not shown in Figure 1, in some of the multiple embodiments, the video encoder 20 and video decoder 30 may be integrated with an audio encoder and decoder, respectively, and may include a suitable multiplexer-demultiplexer (MUX-DEMUX) unit or other hardware and software to handle the encoding of both audio and video in a common data stream or separate data streams. Where applicable, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol or other protocols such as the User Datagram Protocol (UDP).

[0051] The video encoder 20 and video decoder 30 may each be implemented as one or more suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the multiple technologies are partially implemented by software, the device may implement the multiple technologies of this disclosure by storing multiple instructions for the software in a suitable non-transient computer-readable medium and executing those multiple instructions in hardware using one or more processors. Each of the video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, any of which may be integrated to form part of a combined encoder / decoder (CODEC) in each device. The device including the video encoder 20 and / or video decoder 30 may include integrated circuits, microprocessors, and / or wireless communication devices such as cellular phones.

[0052] Figure 2 is a block diagram illustrating one example of a video encoder 20 capable of implementing video coding techniques. The video encoder 20 may perform in-frame and inter-frame coding of video blocks in a video slice. In-frame coding relies on spatial prediction to reduce or eliminate spatial redundancy in video within a given video frame or picture. Inter-frame coding relies on temporal prediction to reduce or eliminate temporal redundancy in video within adjacent frames or pictures of a video sequence. The in-frame mode (I mode) may refer to any of several spatial-based coding modes. Inter-frame modes, such as one-way prediction (P mode) (also known as single prediction) or bi-prediction (B mode) (also known as bi-prediction), may refer to any of several temporal-based coding modes.

[0053] As shown in Figure 2, the video encoder 20 receives the current video block in the video frame to be encoded. In the example in Figure 2, the video encoder 20 includes a mode selection unit 40, a reference frame memory 64, a summation adder 50, a transformation processing unit 52, a quantization unit 54, and an entropy coding unit 56. The mode selection unit 40 similarly includes a motion compensation unit 44, a motion estimation unit 42, an in-frame prediction unit 46 (also known as intra-prediction), and a segmentation unit 48. For the reconstruction of the video block, the video encoder 20 also includes an inverse quantization unit 58, an inverse transformation unit 60, and a summation adder 62. It may also include a deblocking filter (not shown in Figure 2) for filtering block boundaries to remove block noise artifacts from the video being reconstructed. If necessary, the deblocking filter would typically filter the output of the summation adder 62. In addition to the deblocking filter, additional filters (such as in-loop or post-loop filters) may be used. Such filters are not shown for brevity, but if necessary, the output of the summation adder 50 may be filtered (as an in-loop filter).

[0054] During the encoding process, the video encoder 20 receives a video frame or slice to be coded. This frame or slice can be divided into multiple video blocks. The motion estimation unit 42 and the motion compensation unit 44 perform inter-frame predictive coding of the received video block for one or more blocks within one or more reference frames to provide a temporal prediction. Alternatively, the intra-frame prediction unit 46 may perform intra-frame predictive coding of the received video block for one or more adjacent blocks within the same frame or slice as the block being coded to provide a spatial prediction. The video encoder 20 may perform multiple coding passes to select, for example, an appropriate coding mode for each block of video data.

[0055] Furthermore, the partitioning unit 48 may partition blocks of video data into sub-blocks based on an evaluation of a previous partitioning scheme in a previous coding pass. For example, the partitioning unit 48 may first partition a frame or slice into the largest coding unit (LCU), and then partition each of those LCUs into a sub-coding unit (sub-CU) based on a rate distortion analysis (e.g., rate distortion optimization). The mode selection unit 40 may further generate a quadtree data structure that shows the partitioning of LCUs into sub-CUs. The leaf nodes CU of the quadtree may include one or more prediction units (PUs) and one or more transformation units (TUs).

[0056] This disclosure uses the term “block” to refer to any of CUs, PUs, or TUs in the context of HEVC, or to similar data structures in the context of other standards (e.g., macroblocks and their subblocks in the context of H.264 / AVC). A CU includes a coding node, a PU, and a TU associated with the coding node. The size of a CU corresponds to the size of the coding node and is square in shape. The size of a CU ranges from 8x8 pixels to a maximum of 64x64 pixels or the size of a tree block with larger pixels. Each CU may contain one or more PUs and one or more TUs. Syntactic data associated with a CU may, for example, describe the partitioning of the CU into one or more PUs. The partitioning mode may differ depending on whether the CU is encoded in skip mode or direct mode, in intra-frame prediction mode, or in inter-frame prediction mode (also known as inter-prediction). A PU may be partitioned into a non-square shape. Furthermore, the syntactic data associated with the CU may, for example, describe the partitioning of the CU into one or more TUs according to a quadtree. The TUs may be square or non-square in shape (e.g., rectangles).

[0057] The mode selection unit 40 may, for example, select one of the coding modes, such as an in-frame coding mode or an inter-frame coding mode, based on the result of the error, and provide the summation adder 50 with the resulting in-frame coded or inter-frame coded blocks to generate residual block data, and then provide the summation adder 62 with the residual block data to reconstruct the coded blocks for use as a reference frame. The mode selection unit 40 also provides the entropy coding unit 56 with syntactic elements such as motion vectors, an in-frame mode indicator, segmentation information, and other such syntactic information.

[0058] The motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes. The motion estimation performed by the motion estimation unit 42 is a process that generates motion vectors, which estimate the motion of a video block. The motion vectors can, for example, indicate the displacement of the PU of a video block in the current video frame or picture relative to a predicted block (or other coded unit) in a reference frame related to the current block (or other coded unit) coded in the current frame. The predicted block is a block found to exactly match a block coded with respect to pixel differences, and the predicted block may be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference metrics. In some of several examples, the video encoder 20 can calculate the value of a sub-integer pixel position of a reference picture stored in the reference frame memory 64. For example, the video encoder 20 may interpolate the value of a quarter-pixel position, an eighth-pixel position, or other fractional pixel position of the reference picture. Therefore, the motion estimation unit 42 may perform motion search with respect to full pixel positions and fractional pixel positions, and output motion vectors with fractional pixel precision.

[0059] The motion estimation unit 42 calculates a motion vector for the PU of a video block in a slice being interframe coded by comparing the position of the predicted block of a reference picture with the position of the PU. The reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1), each of which identifies one or more reference pictures stored in the reference frame memory 64. The motion estimation unit 42 transmits the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

[0060] The motion compensation performed by the motion compensation unit 44 may involve retrieving or generating a predicted block based on a motion vector determined by the motion estimation unit 42. Again, in some of the multiple examples, the motion estimation unit 42 and the motion compensation unit 44 may be functionally integrated. Upon receiving a motion vector for the PU of the current video block, the motion compensation unit 44 can determine the location of the predicted block that the motion vector points to in one of its multiple reference picture lists. The summation adder 50 forms a residual video block by subtracting the pixel values ​​of the predicted block from the pixel values ​​of the current video block to be coded, as described below, and consequently forms a pixel difference value. Generally, the motion estimation unit 42 performs motion estimation for the luminance component, and the motion compensation unit 44 uses a motion vector calculated based on the luminance component for both the saturation and luminance components. The mode selection unit 40 can also generate syntactic elements associated with video blocks and video slices for use by the video decoder 30 when decoding video blocks of video slices.

[0061] The in-frame prediction unit 46 may perform an in-frame prediction on the current block instead of the inter-frame prediction performed by the motion estimation unit 42 and the motion compensation unit 44, as described above. In particular, the in-frame prediction unit 46 may determine the in-frame prediction mode to use to encode the current block. In some of the examples, the in-frame prediction unit 46 may encode the current block using different in-frame prediction modes, for example, during individual encoding passes, and the in-frame prediction unit 46 (or, in some of the examples, the mode selection unit 40) can select an appropriate in-frame prediction mode to use from the modes being tested.

[0062] For example, the in-frame prediction unit 46 may calculate rate distortion values ​​using rate distortion analysis for various in-frame prediction modes being tested, and then select the in-frame prediction mode with the best rate distortion characteristics among those modes being tested. Rate distortion analysis generally determines not only the amount of distortion (or error) between the original unencoded block previously encoded to generate the encoded block and the encoded block, but also the bit rate (i.e., the number of bits) used to generate the encoded block. For various encoded blocks, the in-frame prediction unit 46 may calculate a ratio from the distortion and rate to determine which in-frame prediction mode exhibits the best rate distortion value for that block.

[0063] In addition, the in-frame prediction unit 46 may be configured to code depth blocks of the depth map using a depth modeling mode (DMM). The mode selection unit 40 may, for example, use rate distortion optimization (RDO) to determine whether an available DMM mode produces better coding results than the in-frame prediction mode and other DMM modes. Data for texture images corresponding to the depth map may be stored in the reference frame memory 64. The motion estimation unit 42 and the motion compensation unit 44 may also be configured to predict depth blocks of the depth map across frames.

[0064] After selecting an in-frame prediction mode for a block (for example, a conventional in-frame prediction mode or one of several DMM modes), the in-frame prediction unit 46 may provide the entropy coding unit 56 with information indicating the selected in-frame prediction mode for that block. The entropy coding unit 56 may encode the information indicating the selected in-frame prediction mode. The video encoder 20 may include configuration data in the transmitted bitstream, which may include several modified in-frame prediction mode index tables (also referred to as codeword mapping tables), definitions of encoding for several contexts for various blocks, and the most likely in-frame prediction mode to use for each of those contexts, an in-frame prediction mode index table, and a modified in-frame prediction mode index table.

[0065] The video encoder 20 forms a residual video block by subtracting predicted data from the mode selection unit 40 from the original video block to be coded. The summation adder 50 represents one or more components that perform this subtraction operation.

[0066] The transformation processing unit 52 applies a transformation such as a discrete cosine transform (DCT) or a conceptually similar transformation to the residual block, thereby generating a video block containing residual transformation coefficient values. The transformation processing unit 52 may also perform other transformations that are conceptually similar to the DCT. Alternatively, it may use wavelet transforms, integer transforms, subband transforms, or other types of transformations.

[0067] The conversion processing unit 52 applies a transformation to the residual block, thereby generating a block of residual transformation coefficients. The transformation may convert the residual information from the pixel value domain to a transformation domain, such as the frequency domain. The conversion processing unit 52 may transmit the resulting transformation coefficients to the quantization unit 54. The quantization unit 54 quantizes these transformation coefficients and further reduces the bit rate. The quantization process can reduce the bit depth associated with some or all of these coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some of the examples, the quantization unit 54 may then perform a scan of the matrix containing the quantized transformation coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0068] Following quantization, the entropy coding unit 56 entropy codes the quantized transformation coefficients. For example, the entropy coding unit 56 may perform context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), stochastic interval-partitioned entropy (PIPE) coding, or other entropy coding techniques. In the case of context-based entropy coding, the context may be based on multiple adjacent blocks. Following entropy coding by the entropy coding unit 56, the encoded bitstream may be transmitted to another device (e.g., a video decoder 30) or archived for later transmission or retrieval.

[0069] The inverse quantization unit 58 and the inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel region, for example, to be used later as a reference block. The motion compensation unit 44 may compute the reference block by adding its residual block to one of the prediction blocks in a plurality of frames of the reference frame memory 64. The motion compensation unit 44 may also apply one or more interpolation filters to its reconstructed residual block to compute sub-integer pixel values ​​for use in motion estimation. The summation adder 62 adds its reconstructed residual block to the motion-compensated prediction block generated by the motion compensation unit 44 to generate a reconstructed video block for storage in the reference frame memory 64. The reconstructed video block may be used by the motion estimation unit 42 and the motion compensation unit 44 as a reference block for inter-frame coding of the block in subsequent video frames.

[0070] Figure 3 is a block diagram illustrating one example of a video decoder 30 capable of implementing video coding techniques. In the example of Figure 3, the video decoder 30 includes an entropy decoding unit 70, a motion compensation unit 72, an in-frame prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference frame memory 82, and an adder 80 that takes sums. In some of several examples, the video decoder 30 may perform a decoding path that is roughly the reverse of the coding path described with respect to the video encoder 20 (Figure 2). The motion compensation unit 72 may generate prediction data based on motion vectors received from the entropy decoding unit 70, while the in-frame prediction unit 74 may generate prediction data based on an in-frame prediction mode indicator received from the entropy decoding unit 70.

[0071] During the decoding process, the video decoder 30 receives an encoded video bitstream from the video encoder 20, representing the video blocks and associated syntactic elements of the video slice being encoded. The entropy decoding unit 70 of the video decoder 30 entropy-decodes the bitstream to generate quantized coefficients, motion vectors or in-frame predictive mode indicators, and other syntactic elements. The entropy decoding unit 70 transfers the motion vectors and other syntactic elements to the motion compensation unit 72. The video decoder 30 may also receive syntactic elements at the video slice level and / or video block level.

[0072] When a video slice is coded as an in-frame coded (I) slice, the in-frame prediction unit 74 may generate prediction data for the video blocks of the current video slice based on the in-frame prediction mode sent by signaling and data from previously decoded blocks of the current frame or picture. When a video frame is coded as an interframe coded slice (e.g., B, P, or GPB), the motion compensation unit 72 generates prediction blocks for the video blocks of the current video slice based on the motion vector and other syntactic elements received from the entropy decoding unit 70. These prediction blocks may be generated from one of several reference pictures in one of several reference picture lists. The video decoder 30 may construct reference frame lists List0 and List1 using default construction techniques based on the reference pictures stored in the reference frame memory 82.

[0073] The motion compensation unit 72 determines prediction information for the video block of the current video slice by analyzing motion vectors and other syntactic elements, and uses that prediction information to generate a prediction block for the current video block to be decoded. For example, the motion compensation unit 72 uses some of the multiple received syntactic elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) used to code the video block of the video slice, the inter-frame prediction slice type (e.g., B-slice, P-slice, or GPB-slice), construction information for one or more of the multiple reference picture lists for the slice, motion vectors for each inter-frame coded video block of the slice, inter-frame prediction states for each inter-frame coded video block of the slice, and other information for decoding the video block in the current video slice.

[0074] The motion compensation unit 72 may also perform interpolation based on an interpolation filter. The motion compensation unit 72 may calculate interpolated values ​​for sub-integer pixels of the reference block using the interpolation filter that the video encoder 20 uses when encoding video blocks. In this case, the motion compensation unit 72 may determine the interpolation filter that the video encoder 20 uses from the received syntax elements and then use those interpolation filters to generate the predicted block.

[0075] Data for the texture image corresponding to the depth map may be stored in the reference frame memory 82. The motion compensation unit 72 may also be configured to predict the depth blocks of the depth map between frames.

[0076] In one embodiment, the video decoder 30 includes a user interface (UI) 84. The user interface 84 is configured to receive input from a user of the video decoder 30 (e.g., a network administrator). Through the user interface 84, the user can manage or change settings in the video decoder 30. For example, the user can input or otherwise provide values ​​for parameters (e.g., flags) to control the configuration and / or operation of the video decoder 30 according to the user's preferences. The user interface 84 may be, for example, a graphical user interface (GUI), which allows the user to interact with the video decoder 30 by graphical icons, drop-down menus, checkboxes, etc. In some cases, the user interface 84 may receive information from the user by a keyboard, mouse, or other peripheral device. In one embodiment, the user can access the user interface 84 by a smartphone, tablet device, or personal computer located remotely from the video decoder 30. As used herein, the user interface 84 may be referred to as an external input or external means.

[0077] With the above in mind, video compression techniques reduce or eliminate redundancy inherent in video sequences by performing spatial (in-picture) predictions and / or temporal (between-picture) predictions. For block-based video coding, a video slice (i.e., a video picture or a portion of a video picture) may be divided into multiple video blocks, which may also be referred to as tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. In-frame coded (I) video blocks in a slice of a picture are coded using spatial predictions about multiple reference samples in multiple adjacent blocks within the same picture. Inter-frame coded (P or B) video blocks in a slice of a picture may use spatial predictions about multiple reference samples in multiple adjacent blocks within the same picture or temporal predictions about multiple reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame.

[0078] Spatial or temporal predictions result in predicted blocks for the blocks to be coded. Residual data represents the pixel difference between the original blocks to be coded and the predicted blocks. Interframe coded blocks are coded according to a motion vector, which points to a reference sample block that forms the predicted block, and the residual data represents the difference between the coded blocks and the predicted blocks. Intraframe coded blocks are coded according to the intraframe coding mode and residual data. For further compression, the residual data may be transformed from the pixel region to the transformation region, resulting in residual transformation coefficients, which may then be quantized. The quantized transformation coefficients may initially be arranged in a two-dimensional array and scanned to generate a one-dimensional vector of transformation coefficients, and entropy coding may be applied to achieve even greater compression.

[0079] Image and video compression continues to experience rapid growth, giving rise to a variety of coding standards. Such video coding standards include Advanced Video Coding (AVC), also known as ITU-T H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) MPEG-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as MPEG-H Part 2, as of ITU-T H.265. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding Plus Depth (MVC+D), and 3D AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC).

[0080] Furthermore, there is a new video coding standard called Multipurpose Video Coding (VVC) that is being developed by the Joint Video Expert Team (JVET) of the ITU-T and ISO / IEC. The VVC standard has several working drafts, but in this specification, we refer to one working draft (WD) of VVC, namely, in particular, B. Bross, J. Chen and S. Liu, "Multipurpose Video Coding (Draft 5)", JVET-N1001-v3, 13th JVET Meeting, March 27, 2019 (VVC Draft 5).

[0081] The descriptions of several technologies disclosed herein are based on the Universal Video Coding (VVC) standard, currently under development by the Joint Video Expert Team (JVET) of the ITU-T and ISO / IEC. These technologies, however, are also applicable to other video codec standards.

[0082] Figure 4 shows the relationship between the Intraframe Random Access Point (IRAP) picture 402 for the leading picture 404 and trailing picture 406, shown in decoding order 408, and the IRAP picture 402 for the leading picture 404 and trailing picture 406, shown in presentation order 410 400. In one embodiment, the IRAP picture 402 is referred to as a Clean Random Access (CRA) picture or an Instantaneous Decoder Refresh (IDR) picture with a Random Access Decodeable (RADL) picture. In the case of HEVC, the IDR picture, CRA picture, and Broken Link Access (BLA) picture are all considered to be IRAP pictures 402. For VVC, it was agreed at the 12th JVET meeting in October 2018 that both IDR pictures and CRA pictures should be included as IRAP pictures. In one embodiment, a Broken Link Access (BLA) picture and a Gradual Decoder Refresh (GDR) picture may also be considered as an IRAP picture. The decoding process for the coded video sequence always starts with IRAP.

[0083] A CRA picture is an IRAP picture in which each Video Coding Layer (VCL) Network Abstraction Layer (NAL) unit has a nal_unit_type equal to C CRA_NUT. A CRA picture does not reference any other picture for interframe prediction in its decoding process and may be the first picture in the bitstream represented in decoding order, or it may appear later in the bitstream. A CRA picture may have associated RADL or Random Access Skip Reading (RASL) pictures. When a CRA picture has a NoOutputBeforeRecoveryFlag equal to 1, associated RASL pictures may not be decodeable because they may contain references to pictures that do not exist in the bitstream, and therefore those associated RASL pictures are not output by the decoder.

[0084] As shown in Figure 4, the reading pictures 404 (e.g., pictures 2 and 3) follow the IRAP picture 402 in the case of decoding order 408, but are placed before the IRAP picture 402 in the case of presentation order 410. The trailing picture 406 follows the IRAP picture 402 in both the case of decoding order 408 and presentation order 410. Although two reading pictures 404 and one trailing picture 406 are shown in Figure 4, those skilled in the art will understand that in practical applications, there may be more or fewer reading pictures 404 and / or trailing pictures 406 in the case of decoding order 408 and presentation order 410.

[0085] In Figure 4, the reading picture 404 is divided into two types: Random Access Skip Reading (RASL) and RADL. When decoding starts with the IRAP picture 402 (e.g., picture 1), it is possible to properly decode the RADL picture (e.g., picture 3), but it is not possible to properly decode the RASL picture (e.g., picture 2). Therefore, the RASL picture is discarded. Considering the distinction between RADL and RASL pictures, the type of reading picture 404 associated with the IRAP picture 402 needs to be identified as either RADL or RASL for efficient and proper coding. In the case of HEVC, when RASL and RADL pictures are present, the requirement that the RASL picture must be placed before the RADL picture in presentation order 410 is suppressed for RASL and RADL pictures associated with the same IRAP picture 402.

[0086] IRAP picture 402 provides two important functions / benefits. First, the presence of IRAP picture 402 indicates that the decoding process can start from that picture. As long as IRAP picture 402 is present in that position, this function enables a random access function in which the decoding process can start at that position in the bitstream, rather than necessarily at the beginning of the bitstream. Second, the presence of IRAP picture 402 updates the decoding process so that the picture to be coded starting at IRAP picture 402 is coded without referencing the preceding picture, with the exception of RASL picture. Allowing IRAP picture 402 to be present in the bitstream will prevent errors that may occur during the decoding of the coded picture before IRAP picture 402 from propagating to IRAP picture 402 and the pictures that follow it in the decoding order 408.

[0087] While IRAP picture 402 provides important functionality, it also detriments the compression efficiency. The presence of IRAP picture 402 causes a significant increase in bitrate. This disadvantage to compression efficiency is due to two reasons. Firstly, because IRAP picture 402 is an intra-frame predicted picture, it will require relatively more bits to present compared to other pictures that are inter-frame predicted pictures (e.g., leading picture 404, trailing picture 406, etc.). Secondly, the presence of IRAP picture 402 interrupts the temporal prediction (this interruption is due to the fact that in its decoding process, one of the operations of the decoding process for this IRAP picture 402 is to remove the previous reference picture from the buffer (DPB) of the picture being decoded, and the decoder will update its decoding process), and the IRAP picture 402 makes the coding of the picture that follows the IRAP picture 402 less efficient in the case of decoding order 408 because those other pictures have fewer reference pictures for the interframe prediction coding of those other pictures (i.e., they require more bits for presentation).

[0088] Among the picture types considered to be IRAP picture 402, the IDR picture in HEVC has different signaling and derivation when compared to other picture types. Some of the differences are as follows:

[0089] For signaling and derivation of the Picture Order Count (POC) value of an IDR picture, the most significant bit (MSB) portion of the POC is not derived from the previous key picture, but is simply set to equal to 0.

[0090] Regarding the signaling information required for managing reference pictures, the slice header of an IDR picture does not contain any information that needs to be sent via signaling to assist in managing reference pictures. For other picture types (i.e., CRA pictures, tail pictures, time-axis partial access (TSA) pictures, etc.), the reference picture marking process (i.e., the process of determining the state of reference pictures in the buffer of decoded pictures (DPB), whether they are used for reference or not) requires information such as the reference picture set (RPS) or other forms of similar information (e.g., a reference picture list) described below. On the other hand, for IDR pictures, the presence of an IDR indicates that the decoding process simply needs to mark all of the multiple reference pictures in the DPB as not being used for reference, so there is no need to send such information via signaling.

[0091] In the case of HEVC and VVC, the IRAP picture 402 and the reading picture 404 may each be contained within a single Network Abstraction Layer (NAL) unit. A set of these NAL units may be referred to as an access unit. The IRAP picture 402 and the reading picture 404 are given different NAL unit types, thereby allowing system-level applications to easily identify these different NAL unit types. For example, a video splicer needs to understand the type of picture being coded without needing to understand the overly detailed contents of the syntactic elements in the coded bitstream, in particular, without needing to identify the IRAP picture 402 from the non-IRAP picture and without needing to identify the reading picture 404 from the trailing picture 406 by determining the RASL and RADL pictures. The trailing picture 406 is associated with the IRAP picture 402 and is the picture that follows the IRAP picture 402 in the case of presentation order 410. A picture may follow its specific IRAP picture 402 in the case of decoding order 408, and may be placed before any other IRAP picture 402 in the case of decoding order 408. For this reason, giving IRAP pictures 402 and reading pictures 404 their own NAL unit types is useful for such applications.

[0092] In the case of HEVC, the NAL unit types for IRAP pictures include the following NAL unit types:

[0093] BLA with Reading Picture (BLA_W_LP): A NAL unit of a corrupted link access (BLA) picture that may be followed by one or more reading pictures in the decryption order.

[0094] BLA with RADL (BLA_W_RADL): A NAL unit of a BLA picture that may be followed by one or more RADL pictures in the decoding order, but does not have any RASL pictures.

[0095] BLA without a leading picture (BLA_N_LP): A NAL unit of a BLA picture that does not have a leading picture following it in the decoding order.

[0096] IDR with RADL (IDR_W_RADL): NAL units of an IDR picture that may be followed by one or more RADL pictures in the decoding order, but do not have any RASL pictures.

[0097] IDR without a leading picture (IDR_N_LP): NAL units of IDR pictures that are not followed by a leading picture in the decoding order.

[0098] CRA: A NAL unit of a clean random access (CRA) picture that may be followed by a leading picture (i.e., either a RASL picture or a RADL picture, or both).

[0099] RADL: NAL unit of RADL picture.

[0100] RASL: NAL unit for RASL pictures.

[0101] In the case of VVC, the NAL unit types for IRAP picture 402 and reading picture 404 are as follows:

[0102] IDR with RADL (IDR_W_RADL): NAL units of an IDR picture that may be followed by one or more RADL pictures in the decoding order, but do not have any RASL pictures.

[0103] IDR without a leading picture (IDR_N_LP): NAL units of IDR pictures that do not follow the leading picture in the decoding order.

[0104] CRA: A NAL unit of a clean random access (CRA) picture that may be followed by a leading picture (either a RASL picture or a RADL picture, or both).

[0105] RADL: NAL unit of RADL picture.

[0106] RASL: NAL unit for RASL pictures.

[0107] Figure 5 illustrates a video bitstream 550 configured to implement a Gradual Decoded Refresh (GDR) technique 500. As used herein, the video bitstream 550 may also be referred to as the coded video bitstream, bitstream, or a variation thereof. As shown in Figure 5, the bitstream 550 includes a sequence parameter set (SPS) 552, a picture parameter set (PPS) 554, a slice header 556, and image data 558.

[0108] SPS552 contains data common to all pictures in a picture sequence (SOP). In contrast, PPS554 contains data common to all pictures. The slice header 556 contains information about the current slice, such as the slice type and which of several reference pictures is used. SPS552 and PPS554 may generally be referred to as parameter sets. SPS552, PPS554, and slice header 556 are types of network abstraction layer (NAL) units. A NAL unit is a syntactic structure that contains an indicator of the type of data that follows it (e.g., video data being coded). NAL units are classified into video coding layer (VCL) and non-VCL NAL units. A VCL NAL unit contains data representing the values ​​of samples in a video picture, while a non-VCL NAL unit contains any relevant additional information, such as a parameter set (important header data that can be applied to a large number of VCL NAL units) and supplemental enhancement information (timing information and other supplemental data that is not necessary for decoding the values ​​of samples in a video picture but can enhance the usefulness of the video signal being decoded). Those skilled in the art will understand that bitstream 550 may include other parameters and information in its actual application.

[0109] The image data 558 in Figure 5 includes data associated with an image or video to be encoded or decoded. The image data 558 may simply be referred to as the payload or data carried in the bitstream 550. In one embodiment, the image data 558 includes a CVS 508 (or CLVS), which includes a GDR picture 502, one or more trailing pictures 504, and a recovery point picture 506. In one embodiment, the GDR picture 502 is referred to as the CVS start (CVSS) picture. The CVS 508 is a coded video sequence for any coded layer video sequences (CLVSs) in the video bitstream 550. In particular, the CVS and CLVS are the same when the video bitstream 550 contains a single layer. The CVS and CLVS are different only when the video bitstream 550 contains multiple layers. In one embodiment, the trailing picture 504 is placed before the recovery point picture 506 in the GDR period, so the trailing pictures 504 may be considered a form of the GDR picture.

[0110] In one embodiment, the GDR picture 502, the tail picture 504, and the recovery point picture 506 may define the GDR period in the CVS 508. In one embodiment, the decoding sequence starts with the GDR picture 502, continues to the tail picture 504, and then proceeds to the recovery picture 506.

[0111] CVS508 is a series of pictures (or a portion thereof) starting with GDR picture 502, including all pictures (or a portion thereof) up to the next GDR picture or the end of the bitstream, but not including the next GDR picture. The GDR period is a series of pictures starting with GDR picture 502, including all pictures up to recovery point picture 506, including recovery point picture 506. The decoding process for CVS508 always starts at GDR picture 502.

[0112] As shown in Figure 5, the GDR technique 500 or principle operates across a series of pictures starting with GDR picture 502 and ending with recovery point picture 506. GDR picture 502 includes an updated / clean region 510 containing blocks that are all coded using in-frame prediction (i.e., in-frame predicted blocks) and an unupdated / dirty region 512 containing blocks that are all coded using inter-frame prediction (i.e., inter-frame predicted blocks).

[0113] The trailing picture 504, immediately adjacent to the GDR picture 502, contains an updated / clean region 510 having a first portion 510A coded using intra-frame prediction and a second portion 510B coded using inter-frame prediction. The second portion 510B is coded, for example, by referencing the updated / clean region 510 of a preceding picture in the GDR period of CVS 508. As shown, the updated / clean region 510 of the trailing picture 504 expands as the coding process moves or progresses in a consistent direction (e.g., from left to right), and correspondingly shrinks the unupdated / dirty region 512. Finally, from that coding process, a recovery point picture 506 containing only the updated / clean region 510 is obtained. In particular, the second portion 510B of the updated / clean region 510, coded as an inter-frame prediction block, may refer only to the updated / clean region 510 in the reference picture, as will be further described below.

[0114] As shown in Figure 5, the GDR picture 502, the trailing picture 504, and the recovery point picture 506 in the CVS 508 are each contained within their own VCL NAL units 530. The set of VCL NAL units 530 in the CVS 508 may be referred to as an access unit.

[0115] In one embodiment, a VCL NAL unit 530 containing the GDR picture 502 in CVS508 has a GDR NAL unit type (GDR_NUT). That is, in one embodiment, a VCL NAL unit 530 containing the GDR picture 502 in CVS508 has its own unique NAL unit type for the trailing picture 504 and the recovery point picture 506. In one embodiment, GDR_NUT allows the bitstream 550 to start with the GDR picture 502 instead of the bitstream 550 having to start with the IRAP picture. Specifying the VCL NAL unit 530 of the GDR picture 502 as GDR_NUT allows the decoder to indicate, for example, that the initial VCL NAL unit 530 in CVS508 contains the GDR picture 502. In one embodiment, the GDR picture 502 is the initial picture in CVS508. In one embodiment, GDR picture 502 is an early picture within the GDR period.

[0116] Figure 6 is a schematic diagram illustrating an undesirable motion search 600 when using encoder constraints to support GDR. As shown, the motion search 600 shows the current picture 602 and the reference picture 604. The current picture 602 and the reference picture 604 each contain an updated region 606 coded using intra-frame prediction, an updated region 608 coded using inter-frame prediction, and an unupdated region 608, respectively. The updated regions 604, 606, and 608 are similar to the first portion 510A, the second portion 510B, and the unupdated / dirty region 512 of the updated / clean region 510 in Figure 5.

[0117] During the motion search process, the encoder suppresses or prevents selecting any motion vector 610 that results in some of several samples of the reference block 612 located outside the region being updated 606. This occurs even when the reference block 612 provides the best rate-distortion cost criterion when predicting the current block 614 in the current picture 602. Thus, Figure 6 illustrates the reasons for the inoptimality in motion search 600 when using encoder constraints to support GDR.

[0118] Figure 7 illustrates a video bitstream 750 configured to implement clean random access (CRA) technology 700. As used herein, the video bitstream 750 may also be referred to as the coded video bitstream, bitstream, or a variation thereof. As shown in Figure 7, the bitstream 750 includes a sequence parameter set (SPS) 752, a picture parameter set (PPS) 754, a slice header 756, and image data 758. The bitstream 750, SPS 752, PPS 754, and slice header 756 in Figure 7 are similar to the bitstream 550, SPS 552, PPS 554, and slice header 556 in Figure 5. Therefore, for brevity, a description of these elements will not be repeated.

[0119] The image data 758 in Figure 7 includes an image or video to be encoded or decoded and associated data. The image data 758 may simply be referred to as the payload or data carried in the bitstream 750. In one embodiment, the image data 758 includes a CVS 708 (or CLVS) which includes a CRA picture 702, one or more trailing pictures 704, and the end of a sequence picture picture 706. In one embodiment, the CRA picture 702 is referred to as a CVSS picture. The decoding process of the CVS 708 always begins with the CRA picture 702.

[0120] As shown in Figure 7, the terminations of the CRA picture 702, the end picture 704, and the sequence picture 706 in the CVS 708 are each contained within their own VCL NAL units 730. The set of VCL NAL units 730 in the CVS 708 may be referred to as an access unit.

[0121] In the latest draft specification of VVC, the output of a preceding picture for an IRAP picture is defined as follows: A preceding picture for an IRAP picture (e.g., a previously decoded picture) refers to those pictures that (1) are decoded earlier than the IRAP picture, (2) are shown for output, (3) are present in the decoded picture buffer (DPB) at the start of the decoding of the IRAP picture, and (4) are not output at the start of the decoding of the IRAP picture. As used herein, a preceding picture may be referred to as a previously decoded picture.

[0122] The slice header syntax includes the syntax element no_output_of_prior_pics_flag for IDR pictures and CRA pictures. Its semantics are as follows:

[0123] The no_output_of_prior_pics_flag, as specified in Appendix C of VVC Manuscript 5, affects the output of previously decoded pictures in the buffer of decoded pictures after decoding of an IDR picture that is not the first picture in the bitstream.

[0124] Section C.3.2 of VVC Draft 5 (Deletion of pictures from the DPB before decoding of the current pictures) includes the following:

[0125] - When the current picture is an IRAP picture with a NoIncorrectPicOutputFlag equal to 1 and not picture0, apply the following ordered steps:

[0126] 1. The variable NoOutputOfPriorPicsFlag is derived for the decoder under test as follows:

[0127] - If the current picture is a CRA picture, NoOutputOfPriorPicsFlag is set to equal 1 (regardless of the value of no_output_of_prior_pics_flag).

[0128] - Otherwise, the values ​​of pic_width_in_luma_samples, pic_height_in_luma_samples, chroma_format_idc, separate_colour_plane_flag, bit_depth_luma_minus8, bit_depth_chroma_minus8, or sps_max_dec_pic_buffering_minus1[HighestTid] derived from the active SPS are respectively used for the preceding picture. If the value differs from ], NoOutputOfPriorPicsFlag may be set to 1 in the decoder under test, regardless of the value of no_output_of_prior_pics_flag (but it does not need to be set to 1).

[0129] Note - Under these conditions, it is desirable to set NoOutputOfPriorPicsFlag to be equal to no_output_of_prior_pics_flag, but in this case, the decoder under test is allowed to set NoOutputOfPriorPicsFlag to 1.

[0130] - Otherwise, NoOutputOfPriorPicsFlag is set to be equal to no_output_of_prior_pics_flag.

[0131] 2. The value of NoOutputOfPriorPicsFlag derived for the decoder under test is applied to the hypothetical reference decoder (HRD), thereby setting the DPB fullness to equal 0, so that when the value of NoOutputOfPriorPicsFlag is equal to 1, all picture memory buffers in the DPB are emptied without using the output of the pictures they contain.

[0132] Section C.5.2.2 (Output and Deletion of Pictures from DPB) of VVC Draft 5 includes the following description:

[0133] - If the current picture is an IRAP picture with a NoIncorrectPicOutputFlag equal to 1 and not picture0, apply the following ordered steps:

[0134] 1. The variable NoOutputOfPriorPicsFlag is derived for the decoder under test as follows:

[0135] - If the current picture is a CRA picture, NoOutputOfPriorPicsFlag is set to equal 1 (regardless of the value of no_output_of_prior_pics_flag).

[0136] - Otherwise, the values ​​of pic_width_in_luma_samples, pic_height_in_luma_samples, chroma_format_idc, separate_colour_plane_flag, bit_depth_luma_minus8, bit_depth_chroma_minus8, or sps_max_dec_pic_buffering_minus1[HighestTid] derived from the active SPS are respectively used for the preceding picture. If the value is different from ], NoOutputOfPriorPicsFlag may be set to 1 by the decoder under test, regardless of the value of no_output_of_prior_pics_flag (but it is not required to set it to 1).

[0137] Note - Under these conditions, it is preferable to set NoOutputOfPriorPicsFlag to be equal to no_output_of_prior_pics_flag, but in this case, the decoder under test is allowed to set NoOutputOfPriorPicsFlag to 1.

[0138] - Otherwise, NoOutputOfPriorPicsFlag is set to be equal to no_output_of_prior_pics_flag.

[0139] 2. The NoOutputOfPriorPicsFlag value derived for the decoder under test is applied to the HRD as follows:

[0140] - If NoOutputOfPriorPicsFlag is equal to 1, all picture memory buffers in the DPB are emptied without using the output of the pictures they contain, and the DPB fullness is set to equal to 0.

[0141] - Otherwise (when NoOutputOfPriorPicsFlag is equal to 0), all picture storage buffers containing pictures marked as "not needed for output" and "not used for reference" are emptied (without using output), all non-empty picture storage buffers in the DPB are emptied by repeatedly calling the "bumping" process specified in Section C.5.2.4, and the DPB fullness is set to equal to 0.

[0142] I have explained several problems with the existing design.

[0143] In the latest draft of the VVC specification, for CRA pictures where NoIncorrectPicOutputFlag is equal to 1 (i.e., CRA pictures that initiate a new CVS), the value of NoOutputOfPriorPicsFlag is set to equal to 1 regardless of the value of no_output_of_prior_pics_flag, so the value of no_output_of_prior_pics_flag is not used. This means that the picture preceding each CRA picture that initiates a CVS is not output. On the other hand, as with IDR pictures, outputting / displaying the previous picture can provide more continuous playback, and therefore a better user experience can be provided as long as the DPB does not overflow when decoding the picture initiating a new CVS and subsequent pictures in the decoding order.

[0144] To solve the problems described above, this disclosure provides the following inventive aspect: The value of no_output_of_prior_pics_flag is used in the specification of outputting the picture prior to each CRA picture that is not the first picture in the bitstream when a new CVS is started. This enables more continuous playback and therefore a better user experience.

[0145] This disclosure also applies to other types of pictures that initiate new CVS, such as the Gradual Random Access (GRA) picture currently specified in the latest draft of the VVC specification. In one embodiment, the GRA picture may be referred to as, or is synonymous with, a GDR picture.

[0146] For example, when decoding a video bitstream, a flag corresponding to a clean random access (CRA) picture is signaled within the bitstream. This flag specifies whether a decoded picture that is in the buffer of the decoded picture and was decoded earlier than the CRA picture should be output when the CRA picture starts a new video sequence being coded. That is, the previous picture is output when the value of the flag indicates that the previous picture should be output (for example, when its value is equal to 0). In one embodiment, the flag is specified as no_output_of_prior_pics_flag.

[0147] As another example, when decoding a video bitstream, a flag corresponding to a sequential random access (GRA) picture is signaled within the bitstream. This flag specifies whether a decoded picture that is in the buffer of the decoded pictures and has been decoded earlier than the GRA picture should be output when that GRA picture starts a new video sequence being coded. That is, the previous picture is output when the value of the flag indicates that the previous picture should be output (for example, when its value is equal to 0). In one embodiment, the flag is specified as no_output_of_prior_pics_flag.

[0148] A technique for outputting previous pictures (e.g., previously decoded pictures) from the picture buffer (DPB) when encountering random access point pictures other than instantaneous decoder refresh (IDR) pictures (e.g., clean random access (CRA) pictures, sequential random access (GRA) pictures, or sequential decoded refresh (GDR) pictures, CVSS pictures, etc.) in the decoding sequence is disclosed herein. Emptying previously decoded pictures from the DPB when reaching a random access point picture prevents the DPB from overflowing and facilitates more continuous playback. Thus, the coder / decoder (also known as the "codec") in video coding is improved compared to current codecs. In practical terms, the improved video coding process provides a better user experience when video is transmitted, received, and / or viewed.

[0149] Figure 8 shows one embodiment of a method 800 for decoding a coded video bitstream implemented by a video decoder (e.g., video decoder 30). The method 800 may be performed after receiving the bitstream being decoded, either directly or indirectly, from a video encoder (e.g., video encoder 20). The method 800 improves the decoding process by clearing the DPB when encountering a random access point picture and before decoding the current picture. The method 800 prevents the DPB from overflowing and promotes more continuous playback. Thus, in practical terms, the codec performance is improved, leading to a better user experience.

[0150] In block 802, the video decoder receives a video bitstream being coded (e.g., bitstream 750). The coded video bitstream includes a first flag having a first value and a clean random access (CRA) picture. In one embodiment, the CRA picture is not the first picture in the coded video bitstream. In one embodiment, the first flag is specified as no_output_of_prior_pics_flag.

[0151] In block 804, the video decoder sets the second value of the second flag to be equal to the first value of the first flag. In one embodiment, the second flag is specified as NoOutputOfPriorPicsFlag. In one embodiment, the second flag resides internally within the decoder.

[0152] In block 806, the video decoder clears the picture buffer (DPB) being decoded of any previously decoded pictures based on a second flag having a second value. In one embodiment, after decoding a CRA picture, previously decoded pictures are cleared from the DPB. That is, the video decoder removes previously decoded pictures from the picture storage buffer in the DPB. In one embodiment, when previously decoded pictures are removed from the DPB, those previously decoded pictures are not output or displayed. In one embodiment, when the first flag is set to a first value, the DPB fullness parameter is set to 0. The DPB fullness parameter indicates how many pictures are stored in the DPB. Setting the DPB fullness parameter to 0 indicates that the DPB is empty.

[0153] In block 808, the video decoder decodes the current picture after the DPB is empty. In one embodiment, the current picture is a picture from the same CVS as the CRA picture, and in the decoding order, the current picture is retrieved or encountered after the CRA. In one embodiment, the image generated based on the current picture is displayed for the user of an electronic device (e.g., a smartphone, tablet, laptop, personal computer).

[0154] Figure 9 shows one embodiment of a method 900 for encoding a video bitstream implemented by a video encoder (e.g., video encoder 20). The method 900 may be performed when a picture (e.g., from a video) is encoded into a video bitstream and then transmitted to a video decoder (e.g., video decoder 30). The method 900 improves the encoding process by instructing the video decoder to empty the DPB when encountering a random access point picture and before decoding the current picture. The method 900 prevents the DPB from overflowing and facilitates more continuous playback. Thus, in practical terms, the codec performance is improved, leading to a better user experience.

[0155] In block 902, the video encoder determines random access points for the video sequence. In block 904, the video encoder encodes clean random access (CRA) pictures into a video sequence at the random access points. In one embodiment, the CRA picture is not the first picture in the video bitstream.

[0156] In block 906, the video encoder instructs the video decoder to clear any previously decoded pictures from the picture buffer (DPB) being decoded by setting the flag to a first value. In one embodiment, after decoding the CRA picture, the video decoder is instructed to clear any previously decoded pictures from the DPB. In one embodiment, the flag is specified as no_output_of_prior_pics_flag. In one embodiment, when the flag is set to a first value, the video encoder instructs the video decoder to set the DPB fullness parameter to 0. In one embodiment, the first value of the flag is 1.

[0157] In block 908, the video encoder generates a video bitstream containing a video sequence and flags having CRA pictures at random access points. In block 910, the video encoder stores the video bitstream for transmission to the video decoder.

[0158] The following syntax and semantics may be used to implement some of the embodiments disclosed herein. The following description relates to the base text, which is a draft of the latest VVC specification. In other words, only deltas are described, while text in the base text not mentioned below remains applicable. Text added compared to that base text is shown in bold, and text deleted is shown in italics.

[0159] [Table 1]

[0160]

number

[0161] Figure 10 is a schematic diagram of a video coding device 1000 according to one embodiment of this disclosure (e.g., video encoder 20 or video encoder 30). The video coding device 1000 is suitable for implementing multiple disclosed embodiments as described herein. The video coding device 1000 includes an inlet port 1010 and a receiver unit (Rx) 1020 for receiving data, a processor, logic unit, or central processing unit (CPU) 1030 for processing data, a transmitter unit (Tx) 1040 and an exit port 1050 for transmitting data, and a memory 1060 for storing data. The video coding device 1000 may also include optoelectronic (OE) and electrooptic (EO) components coupled to the inlet port 1010, receiver unit 1020, transmitter unit 1040, and exit port 1050 for optical or electrical signal input or output.

[0162] The processor 1030 is implemented by hardware and software. The processor 1030 may be implemented as one or more CPU chips, cores (such as a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 1030 communicates with the inlet port 1010, the receiver unit 1020, the transmitter unit 1040, the exit port 1050, and the memory 1060. The processor 1030 includes a coding module 1070. The coding module 1070 implements one of the multiple disclosed embodiments described above. For example, the coding module 1070 implements, processes, prepares, or provides various codec functions. Thus, including the coding module 1070 results in a substantial improvement to the functionality of the video coding device 1000 and results in the conversion of the video coding device 1000 to different states. Alternatively, the coding module 1070 is implemented as an instruction stored in memory 1060 and executed by processor 1030.

[0163] The video coding device 1000 may also include an input and / or output (I / O) device 1080 for communicating data with the user. The I / O device 1080 may include output devices such as a display for displaying video data and a speaker for outputting audio data. The I / O device 1080 may also include input devices such as a keyboard, mouse, or trackball, and / or corresponding interfaces for interacting with such output devices.

[0164] Memory 1060 may include one or more disks, tape drives, and solid-state drives, and may be used as overflow data storage devices to store programs when a program is selected for execution, and to store instructions and data to be read during program execution. Memory 1060 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), tri-level content addressable memory (TCAM), and / or static random access memory (SRAM).

[0165] Figure 11 is a schematic diagram of one embodiment of the coding means 1100. In one embodiment, the coding means 1100 is implemented by a video coding device 1102 (e.g., a video encoder 20 or a video decoder 30). The video coding device 1102 includes a receiving means 1101. The receiving means 1101 is configured to receive a picture to encode or a bitstream to decode. The video coding device 1102 includes a transmitting means 1107 coupled to the receiving means 1101. The transmitting means 1107 is configured to transmit a bitstream to a decoder or to a display means (e.g., one of a plurality of I / O devices 1080).

[0166] The video coding device 1102 includes a storage means 1103. The storage means 1103 is coupled to at least one of the receiving means 1101 or the transmitting means 1107. The storage means 1103 is configured to store instructions. The video coding device 1102 also includes a processing means 1105. The processing means 1105 is coupled to the storage means 1103. The processing means 1105 is configured to execute instructions stored in the storage means 1103 and perform the method disclosed herein.

[0167] Furthermore, it should be understood that the steps of the exemplary methods described herein do not necessarily have to be performed in the order described, and the order of the steps of such methods should be understood to be exemplary only. Similarly, additional steps may be included in such methods, and some steps may be omitted or combined in a manner consistent with various embodiments of this disclosure.

[0168] While several embodiments are provided by this disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of this disclosure. These examples are illustrative and not limiting, and their intent is not limited to the details given herein. For example, various elements or components may be combined or integrated to form other systems, or certain features may be omitted or not implemented.

[0169] In addition, without departing from the scope of this disclosure, technologies, systems, subsystems, and methods described and illustrated separately or individually in various embodiments may be combined or integrated with other systems, modules, technologies, or methods. Other items that are coupled to or directly coupled to or communicate with one another may be indirectly coupled or communicated by some interface, device, or intermediate component, electrically, mechanically, or otherwise. Other examples of changes, substitutions, and modifications are evident to those skilled in the art and can be made without departing from the spirit and scope disclosed herein.

Claims

1. Decryption method: A step of receiving a video bitstream, wherein the video bitstream includes a slice header of slices, the slice header includes syntactic elements that affect the output of previously decoded pictures in a buffer of decoded pictures (DPB) after decoding of a coded video sequence start (CVSS) picture that is not the first picture in the video bitstream; Steps to derive the flag based on the syntax element by setting the value of the flag to be equal to the value of the syntax element, before decoding the current picture and if the current picture is a CVSS picture that is not the first picture in the video bitstream; Steps include: applying the flag for a hypothetical reference decoder (HRD), and if the value of the flag is equal to 1, all picture memory buffers in the DPB are emptied without using the output of the pictures they contain, and the DPB fullness is set to equal to 0; A method that includes this.

2. The method according to claim 1, wherein the CVSS picture is a clean random access (CRA) picture.

3. The method according to claim 1, wherein the CVSS picture is a progressively decoded refresh (GDR) picture.

4. The method according to any one of claims 1 to 3, wherein the flag is specified as NoOutputOfPriorPicsFlag.

5. The method according to any one of claims 1 to 4, wherein the syntactic element is specified as no_output_of_prior_pics_flag.

6. The method according to any one of claims 1 to 5, wherein if the NAL unit type of the slice is equal to IDR_W_RADL, IDR_N_LP, CRA_NUT, or GDR_NUT, the syntactic element is present in the slice header.

7. A decoding device configured to perform the method described in any one of claims 1 to 6.

8. A computer program that causes a processor to perform the steps of the method described in any one of claims 1 to 6.

Images