Switching indicator on video encoding layer

By introducing an on-layer switching indication (LA indication), the problem of unstable up-switching in the VVC specification is solved, ensuring the correct decoding of the STSA picture, and realizing data integrity during up-switching.

CN114009032BActive Publication Date: 2024-12-06TELEFONAKTIEBOLAGET LM ERICSSON (PUBL)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080045127.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-06-21
Filing Date
2020-05-20
Publication Date
2024-12-06
Estimated Expiration
2040-05-20

AI Technical Summary

Technical Problem

The constraints specified in the VVC specification cannot guarantee the effectiveness of the switching on the switch in all cases, especially when the STSA picture and the APS are at the same time layer, which may cause the APS to be not decoded, affecting the decoder's correct decoding of the STSA picture.

Method used

Introduce an on-layer switching indication (LA indication) to ensure that when switching on, all NAL units belonging to the access unit of the STSA picture are decoded, and the LA picture is decoded in the parameter set before the LA position to ensure that the decoder obtains the necessary data.

Benefits of technology

Through the LA indication mechanism, we ensure that all necessary data is provided to the decoder after switching up, allowing the encoder to use APS data before the STSA picture in the decoding order in the STSA picture, solving the problem that the APS is not decoded. , ensures the correct decoding of STSA images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114009032B_ABST
    Figure CN114009032B_ABST
Patent Text Reader

Abstract

A mechanism for decoding a set of pictures from a bitstream is provided. The method comprises obtaining a layer access, LA, indication, the LA indication identifying a LA picture in an access unit, wherein i) the LA picture has a layer identifier value identifying a layer, and ii) the LA picture is followed by other pictures in decoding order. The method comprises determining a LA position in the bitstream based on the LA indication. The method comprises decoding the LA picture and other pictures following the LA picture in decoding order, wherein the LA picture is not decoded using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) precedes the LA position.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to video encoding and decoding. Background Art

[0002] 1.1HEVC and VVC

[0003] High Efficiency Video Coding (HEVC) is a block-based video codec standardized by ITU-T and MPEG. HEVC video codec uses both temporal prediction and spatial prediction. Spatial prediction is achieved using intra (I) prediction from the current picture. Temporal prediction is achieved using unidirectional (P) or bidirectional (B) inter prediction at the block level based on previously decoded reference pictures. In the encoder, the difference between the original pixel data and the predicted pixel data (called the residual) is transformed to the frequency domain, quantized, and then entropy encoded before being sent together with the necessary prediction parameters (such as prediction mode and motion vector) that are also entropy encoded. The decoder performs entropy decoding, inverse quantization, and inverse transformation to obtain the residual, which is then added to the intra prediction or inter prediction to reconstruct the picture.

[0004] MPEG and ITU-T are developing a successor to HEVC within the Joint Video Exploration Team (JVET). The name of this developing video codec is Versatile Video Coding (VVC). At the time of writing, the current VVC draft can be found in the JVET document JVET-N1001-V7.

[0005] 1.2 NAL Unit

[0006] Both HEVC and VVC define a Network Abstraction Layer (NAL). All data in HEVC and VVC (i.e., both Video Coding Layer (VCL) data and non-VCL data) are encapsulated in NAL units. VCL NAL units contain data representing sample values. Non-VCL NAL units contain additional associated data, such as parameter sets and supplemental enhancement information (SEI) messages.

[0007] The NAL unit in HEVC starts with a NAL unit header (or simply "header"), which specifies: i) the NAL unit type of the NAL unit, identifying the type of data carried in the NAL unit, ii) the layer ID to which the NAL unit belongs, and iii) the time ID to which the NAL unit belongs. The NAL unit type is encoded in the nal_unit_type codeword in the NAL unit header. The NAL unit type indicates and defines how the NAL unit should be parsed and decoded. The remaining bytes of the NAL unit are payloads of the type indicated by the NAL unit type. The bitstream consists of a series of NAL units. The syntax of the NAL unit header of HEVC is shown in Table 1:

[0008] Table 1 (HEVC NAL unit header syntax)

[0009]

[0010] The first byte of each NAL unit in HEVC contains the nal_unit_type syntax element (aka codeword). A decoder or bitstream parser can draw conclusions about how the NAL unit should be processed (e.g., parsed and decoded) after looking at the first byte and determining the NAL unit type value encoded in the nal_unit_type codeword.

[0011] In VVC, the NAL unit type is signaled differently, but the NAL unit header in VVC contains information about the NAL unit type for the decoder to derive the NAL unit. The NAL unit types of the current version of the VVC draft JVET-N1001-v7 are shown in Table 2 below.

[0012] The decoding order is the order in which the NAL units should be decoded, which is the same as the order of the NAL units within the bitstream. The decoding order may be different from the output order, which is the order in which the decoder is to output (e.g., for display) the decoded pictures.

[0013] Table 2 – NAL unit types in the current version of the VVC draft

[0014]

[0015]

[0016]

[0017] 1.3 Time Layer

[0018] In HEVC, all pictures are associated with a TemporalId value that specifies the temporal layer to which the picture belongs. The TemporalId value is decoded from the nuh_temporal_id_plus1 syntax element in the NAL unit header. In HEVC, the encoder needs to set the TemporalId value so that when the higher temporal layer is discarded, the pictures belonging to the lower layer can be perfectly decoded. For example, suppose the encoder outputs a bitstream using temporal layers 0, 1, and 2. Then removing all layer 2 NAL units or removing all layer 1 and layer 2 NAL units will result in a bitstream that can be decoded without any problems. This is ensured by constraints in the HEVC specification that the encoder must comply with. For example, pictures of a temporal layer are not allowed to reference pictures of a higher temporal layer.

[0019] 1.4STSA pictures

[0020] In HEVC (and in the current VVC draft), there is a picture type called step-by-step temporal sub-layer access (STSA) picture. There are two types of STSA pictures in HEVC, namely STSA_R and STSA_N, STSA_R is a STSA picture that is also a reference picture, and STSA_N is a STSA picture that is a non-reference picture. In the current VVC draft, only one type of STSA picture is specified, and there is no distinction between whether the STSA picture is a reference picture or a non-reference picture.

[0021] The STSA picture is intended to indicate a location in the bitstream where the temporal layer can be "up-switched", that is, from a lower temporal layer to a higher temporal layer. For example, a decoder can decode temporal layer N, which means that all NAL units with TemporalId equal to or less than N are decoded, and all NAL units with TemporalId greater than N are ignored. If there is an STSA picture with TemporalId N+1, the decoder is guaranteed to be able to decode the STSA picture and all NAL units that follow the STSA picture in decoding order and have TemporalId equal to or less than N+1.

[0022] The temporal layer switching feature described above is implemented by the following three main constraints (listed and described in Table 3) on STSA pictures in HEVC.

[0023] Table 3 – HEVC STSA pictures

[0024]

[0025] 1.5TSA Pictures

[0026] HEVC also specifies temporal sublayer access (TSA) pictures. Similar to STSA pictures, there are two types of TSA pictures in HEVC, namely TSA_R and TSA_N. TSA_R is a TSA picture that is also a reference picture, and TSA_N is a TSA picture that is a non-reference picture. TSA pictures are not specified in the current VVC draft.

[0027] Similar to STSA pictures, TSA pictures are intended to indicate a location in the bitstream where up-switching from a lower temporal layer to a higher temporal layer is possible, but in contrast to STSA pictures (for which only one immediately adjacent layer can be added for each STSA picture), TSA pictures indicate that up-switching to any higher layer is possible.

[0028] 1.6 Intra-frame random access point (IRAP) pictures and coded video sequences (CVS).

[0029] For single-layer coding in HEVC, an access unit (AU) is a coded representation of a single picture. An AU may consist of several video coding layer (VCL) NAL units as well as non-VCL NAL units.

[0030] An intra random access point (IRAP) picture in HEVC is a picture that is not predicted with reference to any picture other than itself during its decoding process. In HEVC, the first picture in the bitstream in decoding order must be an IRAP picture, but IRAP pictures may also appear later in the bitstream. HEVC specifies three types of IRAP pictures: i) Broken Link Access (BLA) pictures, ii) Instantaneous Decoder Refresh (IDR) pictures, and iii) Clean Random Access (CRA) pictures.

[0031] A coded video sequence (CVS) is a series of access units starting from an IRAP access unit up to (but not including) the next IRAP access unit in decoding order. In VVC, there are also GRA pictures, which can start a CVS without intra pictures.

[0032] An IDR picture always starts a new CVS. A CRA picture may have an associated RADL or RASL picture. A CRA picture may contain syntax elements that specify a non-empty set of reference pictures. A CRA may or may not start a CVS.

[0033] 1.7 Parameter Set

[0034] HEVC specifies three types of parameter sets: i) Picture Parameter Set (PPS), ii) Sequence Parameter Set (SPS), and iii) Video Parameter Set (VPS). PPS contains data common to the entire picture, SPS contains data common to a coded video sequence (CVS), and VPS contains data common to multiple CVSs.

[0035] In HEVC, the TemporalId of VPS and SPS shall be equal to 0. The TemporalId of PPS shall be equal to or greater than the TemporalId of the picture in the same access unit as the PPS.

[0036] There are also adaptation parameter sets (APS) and decoding parameter sets (DPS) in VVC. APS can contain information that can be used for multiple slices, and two slices of the same picture can use different APS. In the current VVC draft, APS can contain ALF parameters or shaper parameters. DPS constitutes the "worst case" information that specifies the profile and level that the decoder will encounter in the entire bitstream.

[0037] VVC inherits the rule that the TemporalId of a PPS should be equal to or greater than the TemporalId of a picture in the same access unit as the PPS. In VVC, the TemporalId of an APS should be equal to the TemporalId of a picture in the same access unit.

[0038] 1.8 Tiles and Bricks

[0039] The VVC video coding standard draft includes a tool called tiles, which divides a picture into rectangular spatially independent regions. The tiles in the VVC coding standard draft are similar to the tiles used in HEVC, but have a two-step partitioning mechanism. Using tiles, a picture can be divided into multiple rows of samples and multiple columns of samples in VVC, where a tile is the intersection of rows and columns. For example, a picture can be divided into 4 tile rows and 5 tile columns, so that the picture has a total of 20 tiles.

[0040] The tile structure is signaled in the PPS by specifying the thickness of the rows and the width of the columns. The individual rows and columns can have different sizes, but the divisions always span the entire picture, from left to right and from top to bottom, respectively.

[0041] There are no decoding dependencies between tiles of the same picture. This includes intra prediction, context selection for entropy coding, and motion vector prediction. An exception is that loop filtering dependencies are usually allowed between tiles.

[0042] The two-step tile partitioning in VVC starts by partitioning the picture into tiles as in HEVC. Each tile is then optionally partitioned into bricks along horizontal boundaries. In the current VVC draft specification, the word brick is also used to refer to tiles that are not further partitioned into bricks.

[0043] 1.9 pieces

[0044] The concept of slices in HEVC divides a picture into independently coded slices, where decoding of one slice in a picture is independent of other slices in the same picture. Different coding types can be used for slices of the same picture, i.e., a slice can be an I slice, a P slice, or a B slice. The main purpose of slices is to enable resynchronization in case of data loss.

[0045] In the current version of VVC, a slice consists of multiple complete tiles or just a contiguous sequence of complete bricks of a tile.

[0046] 1.10 Reference Image Management

[0047] In HEVC, pictures are identified by their picture order count (POC) values ​​(also called full POC values). Each slice contains a codeword pic_order_cnt_lsb, which should be the same for all slices in a picture. Pic_order_cnt_lsb is also called the least significant bit (lsb) of the full POC because it is a fixed-length codeword and only signals the least significant bit of the full POC. Both the encoder and decoder keep track of the POC and assign the POC value to each coded / decoded picture. The pic_order_cnt_lsb can be signaled by 4 to 16 bits. The variable MaxPicOrderCntLsb is used in HEVC, which is set to the maximum pic_order_cnt_lsb value plus 1. This means that if 8 bits are used to signal pic_order_cnt_lsb, the maximum value is 255, and MaxPicOrderCntLsb is set to 2^8=256. The picture order count value of a picture is called PicOrderCntVal in HEVC. Typically, the PicOrderCntVal of the current picture is simply referred to as PicOrderCntVal.

[0048] Reference picture management in HEVC is done using Reference Picture Sets (RPS). RPS is a set of reference pictures signaled in the slice header. When the decoder has decoded a picture, it is placed in the Decoded Picture Buffer (DPB) along with its POC value. When decoding subsequent pictures, the decoder parses the RPS syntax from the slice header and builds a list of reference picture POC values. These lists are compared with the POC values ​​of the stored pictures in the DPB, and the RPS specifies which pictures in the DPB are to be kept in the DPB and which are to be removed. All pictures not included in the RPS are marked as being removed from the DPB. Based on the decoded RPS information, the pictures that remain in the DPB are marked as short-term reference pictures or long-term reference pictures.

[0049] A key feature of the HEVC reference picture management system is to signal, for each slice, the state that the DPB should have before decoding the current picture. This enables the decoder to compare the signaled state with the actual state of the DPB and determine if any reference pictures are missing.

[0050] Reference picture management in the VVC draft specification is slightly different from that in HEVC. In HEVC, the RPS is signaled and the reference picture list for inter prediction is derived from the RPS. In the VVC draft specification, the reference picture list (RPL) is signaled and the RPS is derived. However, in both specifications, the signaling of which pictures to keep in the DPB and which pictures should be short-term and long-term is done. In both specifications, the POC is used for picture identification and determination of missing reference pictures.

[0051] 1.11 HEVC Scalability and Multi-view Extensions

[0052] HEVC specifies a small number of extensions, including the Scalable HEVC (SHVC) extension and the Multi-view HEVC (MV-HEVC) extension.

[0053] SHVC provides support for spatial, SNR and color gamut scalability on top of the temporal scalability already provided by the first version of HEVC. SHVC defines a base layer and one or more scalable enhancement layers. The layer id is decoded from the layer_id codeword in the NAL unit header. The layer id of the base layer is always equal to 0, while the layer id of the scalable enhancement layer is greater than 0. In order to decode an enhancement layer, the referenced layers must first be decoded, i.e. the referenced base layer picture and potentially other referenced enhancement layer pictures. With spatial scalability, a layer with a higher resolution can be predicted from a layer with a lower resolution. SNR scalability allows a layer with a higher quality to be predicted from a layer with a lower quality, while color gamut scalability provides an extended color gamut in the scalable enhancement layer.

[0054] MV-HEVC provides support for encoding multiple views using inter-layer prediction. Each view in MV-HEVC is identified by the layer_id codeword in the NAL unit header. Summary of the invention

[0055] There are certain challenges. For example, the constraints specified in the STSA pictures and the VVC specification do not guarantee that up-switching works well in all cases, because there is nothing preventing the VVC bitstream from allowing STSA pictures (and pictures after the STSA pictures in decoding order) to use the APS that precedes the STSA pictures in decoding order and has the same temporal layer as the STSA pictures. The same is true for the PPS, prefix SEI messages, and any undefined NAL units before the first STSA NAL unit in the access unit where the STSA picture is located.

[0056] As another example, in VVC, the up-switch indication is designed as a VCL NAL unit type, and it can be expected that for VVC access units, at least one APS is usually included in the access unit, which contains data to be used by the picture. If the APS must precede the VCL NAL unit and the up-switch indication is placed with the VCL NAL unit (which is the case for VVC and HEVC STSA pictures), the decoder cannot perform up-switching at the VCL NAL unit because this will cause the APS to not be decoded.

[0057] Therefore, the present disclosure proposes an embodiment in which a constraint is added to the bitstream so that the encoder is not allowed to let the up-switched picture of layer N (and pictures after the up-switched picture) use any PPS, APS, etc. that precedes the access unit containing the up-switched picture in decoding order in layer N. That is, in one embodiment, up-switching is done at the granularity of the access unit so that when up-switching occurs, all NAL units of the access unit belonging to the STSA picture are decoded.

[0058] According to a first aspect of the present disclosure, a method for decoding a set of pictures from a bitstream is provided. The method comprises: obtaining a layer access LA indication, the LA indication identifying a LA picture in an access unit, wherein i) the LA picture has a layer identifier value identifying a layer, and ii) other pictures follow the LA picture in decoding order. The method further comprises: determining the LA position in the bitstream based on the LA indication. The method comprises: decoding the LA picture and other pictures following the LA picture in decoding order, wherein the LA picture is not decoded using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0059] According to a second aspect of the present disclosure, a method for decoding a set of pictures from a bitstream is provided. The method comprises: obtaining a layer access LA indication, the LA indication identifying a LA picture in an access unit, wherein i) the LA picture has a layer identifier value identifying a layer, and ii) at least one picture follows the LA picture in decoding order, wherein the layer identifier value of the picture following the LA picture is equal to or greater than the layer identifier value of the LA picture. The method comprises: determining an LA position in a bitstream based on the LA indication. The method further comprises: determining that the LA picture or the picture following the LA picture uses a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position. The method comprises: determining that an error condition exists in response to determining that the LA picture or the picture following the LA picture in decoding order uses the parameter set.

[0060] According to a third aspect of the present disclosure, a method for encoding a picture is provided. The method includes: encoding a layer access LA picture, wherein the LA picture has a layer identifier value that identifies the layer. The method also includes: encoding an LA indication indicating the LA picture so that a decoder can identify the LA picture according to the LA indication, wherein the LA position in the bitstream can be identified based on the LA indication, wherein the LA picture is not encoded using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0061] According to a fourth aspect of the present disclosure, a video decoder is provided. The video decoder is adapted to obtain a layer access LA indication, wherein the LA indication identifies an LA picture in an access unit, wherein i) the LA picture has a layer identifier value identifying a layer, and ii) other pictures follow the LA picture in decoding order. The video decoder is also adapted to determine the LA position in a bitstream based on the LA indication. The video decoder is adapted to decode the LA picture and other pictures following the LA picture in decoding order, wherein the LA picture is decoded without using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0062] According to a fifth aspect of the present disclosure, a video decoder is provided. The video decoder is adapted to obtain a layer access LA indication, wherein the LA indication identifies an LA picture in an access unit, wherein i) the LA picture has a layer identifier value identifying a layer, and ii) at least one picture follows the LA picture in decoding order, wherein the layer identifier value of the picture following the LA picture is equal to or greater than the layer identifier value of the LA picture. The video decoder is adapted to determine an LA position in a bitstream based on the LA indication. The video decoder is also adapted to determine that the LA picture or the picture following the LA picture uses a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position. The video decoder is also adapted to determine that an error condition exists in response to determining that the LA picture or the picture following the LA picture in decoding order uses the parameter set.

[0063] According to a sixth aspect of the present disclosure, a video encoder is provided. The video encoder is suitable for encoding a layer access LA picture, wherein the LA picture has a layer identifier value that identifies the layer. The video encoder is also suitable for encoding an LA indication indicating the LA picture so that a decoder can identify the LA picture according to the LA indication, wherein the LA position in the bitstream can be identified based on the LA indication, wherein the LA picture is not encoded using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0064] According to a seventh aspect of the present disclosure, there is provided a computer program comprising instructions, which, when executed by a processing circuit, causes the processing circuit to perform the method according to the first aspect or the second aspect.

[0065] According to an eighth aspect of the present disclosure, there is provided a carrier including the computer program according to the fifth aspect, wherein the carrier is one of an electric signal, an optical signal, a radio signal and a computer-readable storage medium (942).

[0066] Advantageously, the proposed up-switching indication mechanism ensures that all necessary data required for decoding a picture after up-switching is provided to the decoder. For example, the proposed mechanism also allows the encoder to use APS data that precedes the STSA VCL NAL unit in the decoding order when decoding a STSA picture (and pictures that follow the STSA picture in the decoding order), even when the APS data belongs to the same temporal layer as the STSA picture, provided that the APS data is placed in the same access unit as the STSA picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a schematic block diagram of a video encoder according to one embodiment.

[0068] Figure 2 is a schematic block diagram of a video decoder according to an embodiment.

[0069] Figure 3 The bit stream is shown.

[0070] Figure 4 is a flow chart illustrating a decoding process according to an embodiment.

[0071] Figure 5 is a flow chart illustrating a decoding process according to an embodiment.

[0072] Figure 6 is a flow chart illustrating a decoding process according to an embodiment.

[0073] Figure 7is a flow chart illustrating a decoding process according to an embodiment.

[0074] Figure 8 is a flow chart illustrating an encoding process according to an embodiment.

[0075] Fig. 9 is a block diagram of an apparatus according to an embodiment.

[0076] Fig. 10A is a block diagram of a video decoding apparatus according to an embodiment.

[0077] Fig. 10B is a block diagram of a video encoding apparatus according to an embodiment.

[0078] Fig. 10C is a block diagram of a video decoding apparatus according to an embodiment. DETAILED DESCRIPTION

[0079] The embodiments described herein may be used in a video encoder or a video decoder.

[0080] In the following embodiments, various methods are described. It will be appreciated by those skilled in the art that two or more embodiments or parts of embodiments may be combined to form new solutions that are still covered by the inventions described in this disclosure. Some of the following embodiments relate to different layers and their identifiers. The layers in the described solutions may be, for example, time layers, spatial layers, quality layers, view layers, any other type of layers, and combinations of different layer types. An encoder or decoder keeps track of the coded data via a layer identifier. Typically, a tag identifier consists of one or more syntax elements for a large block of coded data, where the syntax element expresses an integer representing a layer identifier. The following embodiments use "layer id" and "layer identifier" for the same thing.

[0081] In this specification, "used for decoding" or "used for encoding" means that earlier data in the bitstream is referenced and used in the decoding / encoding process. For example, a parameter set used to encode a picture can be interpreted as the parameter set preceding the picture in the bitstream, and the decoding process of the picture uses data from the parameter set, so that if the parameter set is not available and decoded by the decoder before the decoding of the picture, the decoder may not be able to decode the picture.

[0082] Figure 11 is a schematic block diagram of a video encoder 140 according to one embodiment. The current pixel block is predicted by performing motion estimation using a motion estimator 150 based on pixel blocks already provided in the same frame or in a previous frame. In the case of inter-frame prediction, the result of the motion estimation is a motion vector or displacement vector associated with a reference block. The motion vector can be used by a motion compensator 150 to output an inter-frame prediction of the pixel block. An intra-frame predictor 149 calculates an intra-frame prediction of the current pixel block. The outputs of the motion estimator / compensator 150 and the intra-frame predictor 149 are input to a selector 151, which selects intra-frame prediction or inter-frame prediction for the current pixel block. The output of the selector 151 is input to an error calculator in the form of an adder 141, which also receives the pixel values ​​of the current pixel block. The adder 141 calculates and outputs a residual error as the difference in pixel values ​​between the pixel block and its prediction. The error is transformed in a transformer 142, for example by a discrete cosine transform, and quantized by a quantizer 143, and then encoded in an encoder 144 (for example by an entropy encoder). In inter-frame coding, the estimated motion vector is also taken to the encoder 144 to generate an encoded representation of the current pixel block. The transformed and quantized residual error of the current pixel block is also provided to the inverse quantizer 145 and the inverse transformer 146 to obtain the original residual error. The adder 147 adds the error to the block prediction output of the motion compensator 150 or the intra-frame predictor 149 to create a reference pixel block that can be used for prediction and coding of the next pixel block. The new reference block is first processed by the deblocking filter 100. The processed new reference block is then temporarily stored in the frame buffer 148, where it can be used for the intra-frame predictor 149 and the motion estimator / compensator 150.

[0083] Figure 2260 is a block diagram of a video decoder 260 according to some embodiments. The decoder 260 includes a decoder 261, such as an entropy decoder, to decode the encoded representation of the pixel block to obtain a set of quantized and transformed residual errors. These residual errors are dequantized by an inverse quantizer 262 and inversely transformed by an inverse transformer 263 to provide a set of residual errors. An adder 264 adds these residual errors to the pixel values ​​of the reference pixel block. Depending on whether inter-frame prediction or intra-frame prediction is performed, the reference block is determined by a motion estimator / compensator 267 or an intra-frame predictor 266. A selector 268 is thus interconnected to the adder 264 as well as the motion estimator / compensator 267 and the intra-frame predictor 266. The resulting decoded pixel block output from the adder 264 is input into the deblocking filter 200. The filtered pixel block is output from the decoder 260 and can be further temporarily provided to a frame buffer 265 to be used as a reference pixel block for a subsequent pixel block to be decoded. The frame buffer 265 is thereby connected to the motion estimator / compensator 267 so that the stored pixel blocks are available to the motion estimator / compensator 267. The output of the adder 264 may also be input to the intra predictor 266 for use as an unfiltered reference pixel block.

[0084] As mentioned above, there are certain challenges, such as the fact that the constraints specified in the VVC specification do not guarantee that up-switching works well in all cases, because there is nothing preventing the VVC bitstream from allowing STSA pictures (and pictures after the STSA pictures in decoding order) to use an APS that precedes the STSA pictures in decoding order and has the same temporal layer as the STSA pictures. The same is true for the PPS, prefix SEI messages, and any as-yet-undefined NAL units before the first STSA NAL unit in the access unit where the STSA picture is located.

[0085] Figure 3 and Table 4 below show an example of a VVC bitstream 300 using STSA pictures that illustrate the problem. There are three pictures in the example (i.e., pictures 301, 302, and 303), each in their own access unit. The first picture 301 is an IDR picture of temporal layer 0, and the two subsequent pictures (i.e., pictures 302 and 303) are both STSA pictures of temporal layer 1. Looking at access unit 2, the STSA B picture there uses a PPS of ID=4 and an APS of ID=8. However, the up-switch at the STSA B picture means that the parameter set data for access unit 1 is not decoded, and since neither the PPS of ID=4 nor the APS of ID=8 is decoded, the STSA B picture cannot be decoded correctly.

[0086] Table 4 – STSA Examples

[0087]

[0088]

[0089] Another problem with the above is that in VVC, the up-switch indication is designed as a VCL NAL unit type, and it is expected that for VVC access units, at least one APS is usually included in the access unit, which contains the data to be used by the picture. If the APS must precede the VCL NAL unit, and the up-switch indication is placed with the VCL NAL unit (which is the case for VVC and HEVC STSA pictures), the decoder cannot perform up-switching at the VCL NAL unit, because this will result in the APS not being decoded. View Figure 3 As in the example in Table 4, the STSA A picture uses the PPS data of ID = 4 and the APS data of ID = 8 in the bitstream that precedes the STSA A picture. If the decoder is decoding only temporal layer 0, finds the upper switching point as the first VCL NAL unit of the STSA A picture, and decodes temporal layer 1 NAL units starting from the STSA A picture, the decoder will not decode the PPS and APS of access unit 1. This has the undesirable effect that the STSA picture cannot reference parameter sets in its own access unit.

[0090] 1.0 LA Instructions, LA Images, LA Locations and Rules

[0091] In one embodiment, a layer up-switching indication is introduced. The layer up-switching indication is referred to herein as a layer access (LA) indication. In certain embodiments where up-switching must be performed step by step, the LA indication may be referred to as a "step-by-step LA" (SLA) indication.

[0092] In HEVC, there are step-by-step temporal sub-layer access (STSA) pictures and temporal sub-layer access (TSA) pictures, but in the embodiments described herein, the LA indication does not need to be a picture. In addition, HEVC STSA and TSA pictures indicate temporal layer up-switching, while LA indications can be used for layers other than temporal layers. Another difference between HEVC and the embodiments of the present disclosure is that HEVC does not address parameter set availability for either of its two up-switching pictures, TSA and STSA.

[0093] In one embodiment, an LA picture is associated with an LA indication. If an up-switching operation is performed on the bitstream, the LA picture is the first picture in a higher layer that the LA indication specifies can be decoded. In one embodiment, the LA picture and the corresponding LA indication are placed together, for example, in the same access unit in the bitstream. In this case, the decoder can identify the LA picture by first finding the LA indication, and then derive the LA picture as a picture in the same access unit as the LA indication. That is, if the decoder finds the LA indication in the access unit, it determines that the picture in the access unit is a LA picture. In embodiments where more than one picture can be present in an access unit, each picture can have its own LA indication.

[0094] In one embodiment, a position in the bitstream called a layer access (LA) position is introduced. In one embodiment, the LA position is derived from the position or placement of the LA indication in the bitstream. In another embodiment, the LA position is derived from a combination of the position or placement of the LA indication in the bitstream and one or more syntax elements in the bitstream.

[0095] In one embodiment, the source layer and the target layer are derived. The decoder can, for example, decode layers 0, 1, and 2. The decoder can then find an LA indication in the bitstream, which provides information to the decoder, from which the decoder can derive a source layer id equal to 2 and a target layer id equal to 4. This means that since the decoder is decoding layer 2, the decoder can perform an up-switch to layer 4. Alternatively, the decoder can perform an up-switch to layer 3, since there is no data from a higher layer to decode any data from a lower layer. This means that it is possible to up-switch to a layer equal to or lower than the derived target layer. However, if the decoder in the example were to encounter an LA indication and derive a source layer id equal to 3 and a target layer id equal to 4, up-switching is not possible since the decoder is not decoding layer 3.

[0096] In one embodiment, the target layer id is derived as the layer id value of the LA indication, for example as the layer id value of the NAL unit containing the LA indication or as the layer id value of the access unit containing the LA indication. In another embodiment, the target layer id is derived as the highest possible layer id value, for example as the highest layer id value specified by a profile for the bitstream or as the maximum layer id value indicated by a syntax element in the bitstream (e.g., in an SPS, VPS, or DPS). In yet another embodiment, the target layer id value is signaled by a syntax element associated with the LA indication. The target layer id value should be greater than the source layer id value.

[0097] In one embodiment, the source layer id is derived as the target layer id minus 1, for example as the layer id value of the LA indication minus 1. In another embodiment, the source layer id value is signaled by a syntax element associated with the LA indication.

[0098] In another embodiment, a rule is introduced, wherein the rule specifies that when decoding an LA picture or a picture that follows the LA picture in decoding order and whose layer id value is greater than the source layer id value and less than or equal to the target layer id value, the following data are not allowed to be used: i) before the LA position, and ii) with a layer id value greater than the source layer id value and less than or equal to the target layer id value.

[0099] Therefore, in one embodiment, the decoder may perform the following process 400 (see Figure 4 ) to decode a set of pictures from a bitstream. Process 400 may start from step s402.

[0100] Step s402 comprises: the decoder obtains an LA indication, the LA indication identifies an LA picture in an access unit, wherein the LA picture is followed by other pictures in a decoding order.

[0101] Step s404 includes: the decoder determines the LA position in the bitstream based on the LA indication (e.g., based only on the LA indication or at least partially based on the LA indication). For example, in one embodiment, the LA position is determined based only on the position of the LA indication in the bitstream. In another embodiment, the LA position is determined based on the position indicated by the LA indication.

[0102] Step s406 comprises: the decoder determines the source layer id and the target layer id based on the LA indication.

[0103] Step s408 comprises: the decoder decodes the LA picture and the pictures following the LA picture in the decoding order, wherein the parameter set of a specific type that i) is before the LA position and ii) has a layer identifier value greater than the source layer id and less than or equal to the target layer id is not used to decode any picture of the LA picture and the pictures following the LA picture in the decoding order, having a layer identifier value greater than the source layer id and less than or equal to the target layer id.

[0104] In some embodiments, the specific type is any type included in a set of one or more specific types, wherein the set of one or more specific types includes an adaptation parameter set (APS) type.

[0105] In some embodiments, the specific type is any type included in a set of one or more specific types, wherein the set of one or more specific types includes a picture parameter set (PPS) type.

[0106] In some embodiments, the specific type is any type included in a set of one or more specific types, wherein the set of one or more specific types includes the SEI message type.

[0107] In another embodiment, the decoder may perform the following process 500 (see Figure 5 ) all or a subset of the steps. Process 500 may start from step s502.

[0108] Step s502 comprises: the decoder obtains an LA indication, the LA indication identifies an LA picture in an access unit, wherein the LA picture is followed by other pictures in a decoding order.

[0109] Step s504 is the same as step s404, comprising: the decoder determining the LA position in the bitstream based on the LA indication.

[0110] Step s506 comprises: the decoder determines the source layer id and the target layer id based on the LA indication.

[0111] Step s508 includes: the decoder determines that the LA picture (or a picture that follows the LA picture in the decoding order and whose layer identifier value is greater than the source layer id and less than or equal to the target layer id) uses a specific type of parameter set, which is before the LA position and has a layer identifier value that is greater than the source layer id and less than or equal to the target layer id.

[0112] Step s510 includes: the decoder determines, in response to the determination, that the bitstream does not conform to the codec specification and may interpret this as bit errors, data loss, or a non-conforming bitstream or a non-conforming encoder. The decoder may report an error, perform error concealment, or take other actions based on the knowledge that the bitstream does not conform.

[0113] 1.1 The target layer ID is equal to the layer ID of the LA image

[0114] In one embodiment, the target layer id is derived as equal to the layer id value of the LA picture, and the source layer id is derived as the target layer id value minus 1. In such an embodiment, the rule can be expressed as: when decoding an LA picture or any picture that follows the LA picture in decoding order and belongs to the same layer as the LA picture, data before the LA position in the same layer as the LA picture is not allowed to be used. This rule can alternatively be expressed using the following two rules:

[0115] Rule 1: When the current picture is a picture that follows the LA picture in decoding order and the current picture and the LA picture have the same layer id, the current picture should not use or reference the parameter set that also has the same layer id and is before the LA position corresponding to the LA picture (for example, the first NAL unit in the access unit belonging to the LA picture in decoding order).

[0116] Rule 2: When the current picture is a LA picture, the current picture shall not use or reference a parameter set whose layer id is equal to the layer id of the current picture and before the LA position associated with the LA picture (e.g., the first NAL unit in decoding order in an access unit belonging to the LA picture).

[0117] Therefore, in one embodiment, the decoder may perform the following process 600 (see Figure 6 ) to decode a set of pictures from a bitstream. Process 600 may start at step s602.

[0118] Step s602 comprises: the decoder obtains an LA indication, the LA indication identifying an LA picture in an access unit, wherein i) the LA picture has a layer identifier value identifying a layer, and ii) the LA picture is followed by other pictures in decoding order.

[0119] Step s604 is the same as step s404, comprising: the decoder determining the LA position in the bitstream based on the LA indication.

[0120] Step s606 comprises: the decoder decodes the LA picture and the pictures following the LA picture in the decoding order, wherein the LA picture is decoded without using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0121] In some embodiments, any other picture that i) follows the LA picture in decoding order and ii) has a layer identifier value equal to the layer identifier value of the LA picture is decoded without using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) precedes the LA position.

[0122] In some embodiments, any other picture that i) follows the LA picture in decoding order and ii) has a layer identifier value greater than the layer identifier value of the LA picture is decoded without using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) precedes the LA position.

[0123] In some embodiments, a parameter set that a) has a specific type, b) has a layer identifier value greater than the layer identifier value of the LA picture, and c) is before the LA position is not used to decode any picture of the LA picture or other pictures that follow the LA picture in decoding order and have a layer identifier value equal to or greater than the layer identifier value of the LA picture.

[0124] In another embodiment, the decoder may perform the following process 700 (see Figure 7 ) all or a subset of the steps. Process 700 may start from step s702.

[0125] Step s702 includes: the decoder obtains an LA indication, the LA indication identifies an LA picture in the access unit, wherein i) the LA picture has a layer identifier value identifying the layer, and ii) there is at least one picture following the LA picture in the decoding order, wherein the picture following the LA picture has a layer identifier value equal to or greater than the layer identifier value of the LA picture.

[0126] In some embodiments, obtaining the LA indication comprises deriving the LA indication from a VCL NAL unit type.

[0127] In some embodiments, obtaining the LA indication comprises deriving the LA indication from a NAL unit type of the non-VCL NAL unit.

[0128] In some embodiments, obtaining the LA indication comprises deriving the LA indication from one or more syntax elements in a parameter set.

[0129] In some embodiments, obtaining the LA indication comprises deriving the LA indication from one or more syntax elements in the non-VCL NAL unit.

[0130] In some embodiments, obtaining the LA indication comprises deriving the LA indication from one or more syntax elements in an access unit delimiter NAL unit in the access unit.

[0131] In some embodiments, obtaining the LA indication comprises deriving the LA indication from one or more syntax elements in a prefix SEI message.

[0132] In some embodiments, obtaining the LA indication comprises: determining whether the access unit contains an access unit delimiter NAL unit; deriving the LA indication from one or more syntax elements of the access unit delimiter NAL unit as a result of determining that the access unit contains the access unit delimiter NAL unit, otherwise deriving the LA indication from the VCL NAL unit. In such embodiments, if the access unit contains the access unit delimiter NAL unit, the LA position is derived equal to the position of the access unit delimiter, otherwise the LA position is derived equal to the position of the first VCL NAL unit in the access unit.

[0133] Step s704 is the same as step s404, comprising: the decoder determining the LA position in the bitstream based on the LA indication. In some embodiments, the LA position is derived as the position of the NAL unit containing the LA indication.

[0134] Step s706 comprises the decoder determining that the LA picture or the picture after the LA picture uses the following parameter set: a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0135] Step s708 comprises: in response to determining that the LA picture or the picture following the LA picture in decoding order uses the parameter set, determining that an error condition exists.

[0136] In some embodiments, determining that an error condition exists includes the decoder determining that: i) a bit error has occurred, ii) data has been lost, iii) the bitstream is a non-conforming bitstream, or iv) an encoder that generated the bitstream is a non-conforming encoder.

[0137] In some embodiments, process 700 also includes, as a result of determining that an error condition exists, the decoder reporting the error condition and / or performing an error concealment operation.

[0138] In some embodiments, process 700 also includes: the decoder determines that the LA picture or the picture after the LA picture uses the following parameter set: a) has a specific type, b) has a layer identifier value greater than the layer identifier value of the LA picture, and c) is before the LA position; and in response to determining that the LA picture or the picture after the LA picture uses the parameter set, determines that an error condition exists.

[0139] In another embodiment, the encoder may perform the following process 800 (see Figure 8 ) all or a subset of the steps. Process 800 may start from step s802.

[0140] Step s802 includes: the encoder encodes the LA picture, wherein the LA picture has a layer identifier value identifying a layer.

[0141] Step s804 includes: the encoder encodes an LA indication indicating an LA picture so that the decoder can identify the LA picture according to the LA indication, wherein the LA position in the bitstream can be identified based on the LA indication, wherein the LA picture is decoded without using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0142] In some embodiments, process 800 further comprises step s806, in which the encoder encodes other pictures following the LA picture in decoding order.

[0143] In some embodiments, any of the other pictures that follow the LA picture in decoding order and have a layer identifier value equal to the layer identifier value of the LA picture are not encoded using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0144] In some embodiments, any of the other pictures that follow the LA picture in decoding order and have a layer identifier value greater than the layer identifier value of the LA picture are not encoded using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0145] In some embodiments, process 800 also includes the encoder outputting (eg, sending or storing): the LA picture, the LA indication identifying the LA picture, and other pictures that follow the LA picture in decoding order.

[0146] 1.2 Export of LA position

[0147] As described above, the LA position is derived from the LA indication. For example, in one embodiment, the LA position is determined based on (e.g., based only on or based in part on) the position of the LA indication in the bitstream. In another embodiment, the LA position is determined based on a combination of the position of the LA indication in the bitstream and one or more syntax elements in the bitstream.

[0148] In one embodiment, the LA position is determined as the position of the first NAL unit in decoding order in an access unit belonging to the picture that contains the LA indication.

[0149] In another embodiment, the LA position is determined as the position of the LA picture.

[0150] In another embodiment, the LA position is determined to be the position of the first VCL NAL unit of the LA picture in decoding order.

[0151] In another embodiment, the LA position is determined as the position of the first NAL unit in decoding order in an access unit having a NAL unit type equal to a specific NAL unit type. The specific NAL unit type may be, for example, a NAL unit type having a LA, an access unit delimiter, an SPS, a PPS, an APS, or a SEI message.

[0152] In another embodiment, the LA position is determined as the position of the first NAL unit in the access unit in decoding order whose NAL unit type is equal to a NAL unit type in a defined set of NAL unit types, wherein the set consists of at least two types. For example, the set of NAL unit types may consist of at least two of a PPS NAL unit type, an APS NAL unit type, and a LA VCL NAL unit type. If all three types are included in the set, and the access unit contains the LA VCL NAL unit type but does not contain other NAL unit types in the set, then the LA position is derived as the position of the LA VCL NAL unit. However, if the access unit contains all three types, then the LA position is derived as the position of the first type in the access unit in decoding order.

[0153] In some embodiments, the access unit comprises an ordered sequence of NAL units, and the LA position is determined to be the position of a first NAL unit in the ordered sequence of NAL units.

[0154] 1.3 Parameter Set Details

[0155] As described above, in various embodiments, a LA picture or any other picture after the LA picture having the same layer id as the LA picture is decoded without using a parameter set that a) has a specific type, b) has a layer identifier value equal to the layer identifier value of the LA picture, and c) is before the LA position.

[0156] In one embodiment, a specific type is any type included in a set of one or more specific types.

[0157] In one embodiment, the set of one or more specific types includes an adaptation parameter set (APS) type. This means that in this embodiment, when decoding an LA picture and any picture that belongs to the same layer as the LA picture after the LA picture in decoding order, the APS before the LA position in the same layer as the LA picture is not allowed to be used.

[0158] In one embodiment, one or more sets of specific types include a picture parameter set (PPS). This means that when decoding an LA picture and any picture after the LA picture in decoding order that has the same layer identifier value as the LA picture, the use of a PPS with the same layer id as the LA picture and before the LA position is not allowed.

[0159] In one embodiment, one or more sets of specific types include SEI messages. This means that when decoding an LA picture and any picture that has the same layer identifier value as the LA picture after the LA picture in decoding order, the use of SEI with the same layer identifier as the LA picture and before the LA position is not allowed.

[0160] In one embodiment, the set of one or more specific types includes at least three types: SEI message, PPS, and APS.

[0161] 1.4 LA indication position in the bitstream

[0162] The decoder (or other network node) may monitor the bitstream to search for the LA indication to determine where upswitching can be performed. The location of the LA indication and the LA position may or may not be the same location in the bitstream. However, as described above, the LA position is determined based on the LA indication.

[0163] In one embodiment, the LA position and the position of the LA indication are the same. For example, in one embodiment, the position of the LA indication is the position of the NAL unit that contains the LA indication, and the LA position is determined to be the position of the NAL unit that contains the LA indication. Thus, for example, a decoder may search the bitstream for an LA indication, find the LA indication in a particular NAL unit (e.g., find a NAL unit whose NAL unit type value identifies a particular NAL unit type), and then set the LA position to the position of the particular NAL unit. Thus, for example, the LA position is based solely on the position of the LA indication.

[0164] In one embodiment, the LA indication is derived from a VCL NAL unit type. In one embodiment, the LA indication is derived from a non-VCL NAL unit type. Either an existing non-VCL NAL unit type is used to derive the LA indication, or a new NAL unit type not listed in Table 2 is used.

[0165] If this new NAL unit type is to be introduced and used for VVC, the STSA NAL unit type can be removed and a new separate non-VCL NAL unit type can be added. For example, the RSV_NVCL5 NAL unit type in Table 2 can be used, and the decoder can find the LA indication as a NAL unit with RSV_NVCL5, and its layer id value is equal to the layer id value or time id value of the NAL unit header from the RSV_NVCL5 NAL unit. In one embodiment, the layer id value of the RSV_NVCL5 NAL unit is the same as the layer id of the following picture: the picture in the same access unit as the RSV_NVCL5 NAL unit. In one embodiment, the position of the RSV_NVCL5 NAL unit in the access unit is earlier than any APS, PPS, SEI message, and VCL NAL unit in the access unit. The decoder or network node can derive the LA position as the position of the RSV_NVCL5 NAL unit.

[0166] In one embodiment, the LA indication is derived from a SEI message with a payload type for the LA indication. If such a SEI payload type is to be introduced and used for VVC, the STSA NAL unit type may be removed and a new SEI message added instead. The decoder may find the LA indication as a SEI message with a new payload type as follows and whose layer id value is equal to the layer id value or the temporal id value of the NAL unit header from the SEI message NAL unit. In one embodiment, the layer id value of the SEI message is the same as the layer id of the following picture: the picture in the same access unit as the SEI message. In one embodiment, the position of the SEI message in the access unit is earlier than any APS, PPS, other SEI messages, and VCL NAL units in the access unit. The decoder or network node may derive the LA position as the position of the SEI message with a payload type having the LA indication.

[0167] In one embodiment, the LA indication is derived from one or more syntax elements in the non-VCL NAL unit.There are several options where the LA indication can be placed, three of which are explained below.

[0168] (1) In one option, the LA indication is derived from one or more syntax elements in the access unit delimiter NAL unit. It is assumed here that the access unit delimiter NAL unit is the first NAL unit in the access unit to which the access unit delimiter NAL unit belongs. The picture in the access unit is determined to be a LA picture. One or more syntax elements can be a 1-bit flag, where one value of the flag (e.g., 1) specifies that the access unit delimiter carries the LA indication, and another value (e.g., 0) specifies that it does not carry it. The decoder can find the LA indication as a value of the syntax element of the access unit delimiter, and its layer id value is equal to the layer id value or the time id value from the NAL unit header of the access unit delimiter. In one embodiment, the layer id value of the access unit delimiter NAL unit is the same as the layer id of the following picture: the picture in the same access unit as the access unit delimiter. The LA position can be derived as the position of the access unit delimiter.

[0169] (2) In the second option, the LA indication is derived from a SEI message with a payload type for LA indication. If such a SEI payload type is to be introduced and used for VVC, the STSA NAL unit type may be removed and a new SEI message added instead. The decoder may find the LA indication as a SEI message with a new specific payload type value and whose layer id value is equal to the layer id value or the temporal id value from the NAL unit header of the SEI message NAL unit. In one embodiment, the layer id value of the SEI message is the same as the layer id of the picture in the same access unit as the SEI message. In one embodiment, the position of the SEI message in the access unit is earlier than any APS, PPS, other SEI messages, and VCL NAL units in the access unit. The decoder may derive the LA position as the position of the SEI message with a payload type for LA indication.

[0170] (3) In a third option, the LA indication is derived from one or more syntax elements in the parameter set. The syntax element may be implemented as a flag of the parameter set or as a type of the parameter set. For example, there may be a flag in the APS to specify whether the parameter set carries an LA indication. Similarly, there may be a flag in the PPS to specify whether the parameter set carries an LA indication. If the flag specifies that the parameter set does carry an LA indication, the picture in the same access unit as the parameter set is determined to be an LA picture. The decoder or network node may find the LA indication as a value of a syntax element of the parameter set, and its layer id value is equal to the layer id value or time id value from the NAL unit header of the parameter set. In one embodiment, the layer id value of the parameter set is the same as the layer id of the following picture: the picture in the same access unit as the parameter set. The LA position may be derived as the position of the parameter set containing the LA indication. Alternatively, there may be two APS NAL unit types or two PPS NAL unit types, one of which is an LA indication and the other is not an LA indication.

[0171] In one embodiment, the LA indication is derived as a flag in the NAL unit header. The presence of the flag may depend on the NAL unit type; for example, the flag may be present when the NAL unit type indicates an access unit delimiter, or the flag may be present when the NAL unit type indicates a trailing picture NAL unit.

[0172] In one embodiment, the LA indication is derived from one or more syntax elements in the picture header, for example as a flag. There is no picture header in HEVC and the current VVC draft, but it has been discussed to specify a picture header for VVC, which would contain common information for all slices in the picture.

[0173] An implementation on top of the current version of the VVC draft may be as follows, where added text is shown in italic text:

[0174]

[0175] 1.5 Combination

[0176] In one embodiment, the LA indication may be derived from one or more syntax elements of an access unit delimiter NAL unit or from one or more syntax elements of a VCL NAL unit. For example, if the bitstream contains an access unit with an LA indication, all VCL NAL units in the access unit carry the LA indication. In addition, if an access unit with an LA indication contains an access unit delimiter NAL unit, the access unit delimiter NAL unit also carries the LA indication.

[0177] A network node (eg, a decoder or other network node) searching or scanning a bitstream for a particular up-switching indication may perform all or a subset of the following steps according to one embodiment.

[0178] 1-determine the target layer id value as the layer id value currently being decoded by the network node plus 1 or determine the target layer id value as the layer id value currently being forwarded by the network node plus 1.

[0179] 2- For each NAL unit in decoding order, do the following:

[0180] 2(a) - If up-switching has been done, decode or forward the NAL unit only if it has a layer id value equal to the target layer id value or lower;

[0181] 2(b) - If upswitching has not been done and the NAL unit is an access unit delimiter, then do the following:

[0182] 2(b)(i) Check one or more syntax elements in the access unit delimiter to determine whether the access unit delimiter contains an LA indication.

[0183] 2(b)(ii) if it contains an LA indication, determine the layer id of the access unit delimiter from one or more syntax elements of the access unit delimiter and check whether the layer id value is equal to the target layer id;

[0184] 2(b)(iii) If the layer id value is equal to the target layer id, decode and / or forward the access unit NAL unit and consider the up-switching to be complete.

[0185] 2(c) Otherwise, if upswitching has not been completed and the NAL unit is a VCL NAL unit, then do the following:

[0186] 2(c)(i) Check one or more syntax elements of the VCL NAL unit to determine whether the VCL NAL unit contains a LA indication.

[0187] 2(c)(ii) If it contains a LA indication, determine the layer id of the VCL NAL unit from one or more syntax elements of the VCL NAL unit and check whether the layer id value is equal to the target layer id.

[0188] 2(c)(iii) If the layer id value is equal to the target layer id, decode and / or forward the VCL NAL unit and consider the up-switching to be complete.

[0189] In other words, the LA indication is derived by determining whether the access unit contains an access unit delimiter NAL unit, wherein if the access unit contains an access unit delimiter NAL unit, the LA indication is derived from one or more syntax elements of the access unit delimiter NAL unit, otherwise the LA indication is derived from the VCL NAL unit.

[0190] If the access unit contains an access unit delimiter NAL unit, the LA position may be derived equal to the position of the access unit delimiter, otherwise the LA position may be derived equal to the position of the first VCL NAL unit in the access unit.

[0191] An implementation on top of the current version of the VVC draft may be as follows, where added text is shown in italic text:

[0192]

[0193]

[0194]

[0195] 1.6 Multiple LA Indications

[0196] In one embodiment, there are at least three LA indications derived as follows. The first LA indication is derived from one or more syntax elements of the VCL NAL unit. The second LA indication is derived from one or more syntax elements of the APS NAL unit. The third LA indication is derived from one or more syntax elements of the PPS NAL unit. If an access unit carries an LA indication, all VCL NAL units must carry an LA indication. If an access unit contains an APS, the APS must also carry an LA indication. Similarly, if an access unit contains a PPS, the PPS must also carry an LA indication. If an access unit does not carry an LA indication, the NAL units in the access unit should not carry an LA indication.

[0197] The network node (e.g., decoder) will then look for LA indications in the APS, PPS, and VCL NAL units. When the first LA indication in decoding order is found, where the layer id value of the NAL unit where the LA indication is found is equal to the target layer id, the decoder starts decoding (or the network node starts forwarding) the target layer NAL unit.

[0198] In this embodiment, the LA position may be derived as the position of the NAL unit in which the LA indication is found.

[0199] 1.7 Additional variants

[0200] In one embodiment, the layer identifier is a temporal layer identifier, a scalable or a multi-view layer identifier.

[0201] In one embodiment, there is a flag in a parameter set such as an SPS that specifies whether there is a LA indication in the CVS. One value of the flag specifies that there is no LA indication in the CVS. Another value of the flag specifies that there may be a LA indication in the CVS.

[0202] For example, in one embodiment, a first parameter set in a bitstream (e.g., an SPS or PPS) includes information (e.g., a flag) specifying that an LA indication is not present in a first CVS in the bitstream, and a second parameter set in the bitstream includes information (e.g., a flag) specifying that an LA indication may be included in a second CVS in the bitstream.

[0203] 1.8 An indication of LA position and picture group

[0204] In this embodiment, the LA indication can be derived from one or more syntax elements in the parameter set (e.g., a flag in an SPS or a VPS), and the LA indication is used to derive multiple LA positions in the bitstream, so that the first NAL unit of an access unit in the bitstream (e.g., having a specific NAL unit type or a specific layer id value or a combination of a specific NAL unit type and a specific layer id) is exported as the LA position in the bitstream, and the pictures in those access units in the bitstream are exported as LA pictures.

[0205] In a version of this embodiment, one or more codewords in the parameter set indicate the LA position in the bitstream as equal to the first NAL unit in the access unit in decoding order, and the layer id of the access unit is equal to the specific layer id. In one example of this variation, the encoder sets the flag in the SPS to a specific value and uses the syntax element value to signal the specific layer id to indicate that layer switching is allowed for all access units in the bitstream with the layer id equal to the specific layer id. Similarly, the decoder decodes the flag to a specific value and decodes the syntax element representing the specific layer id, indicating to the decoder that layer switching is allowed for all access units with the specific layer id value. The decoder can perform an up-switching from a specific layer to a higher layer at a picture with a layer id equal to the specific layer id.

[0206] 1.9 Explicitly specify the target layer that the decoder can switch to

[0207] There is no need to perform up-switching step by step, i.e. only increment one layer. In such an embodiment, it may be explicitly specified which temporal layer can be up-switched to, i.e. what the possible target layers are. The set of target layers may be defined by a specific rule or indicated by one or more syntax elements in the bitstream (e.g. in an access unit delimiter, in a new LA non-VCL NAL unit, or in an SEI message).

[0208] One or more syntax elements may indicate, for example, like a TSA picture in HEVC, that up-switching to the largest layer, ie the highest layer in the bitstream, is possible.

[0209] In one embodiment, one or more syntax elements indicate the maximum layer to which the decoder can switch up. For example, if the bitstream includes layers 0 to 5 and the decoder decodes layers 0 and 1, the one or more syntax elements may indicate that the maximum target layer is 4, which means that the decoder can decode layers 0 to 4, but not layer 5. The one or more syntax elements may be a delta value relative to another layer (e.g., the layer indicated by LA) or an absolute maximum target layer.

[0210] In one example of this embodiment, the maximum target layer is signaled in the access unit delimiter, where a flag (layer_access_indication_flag) indicates whether LA is allowed. If the flag indicates that LA is allowed, the access unit delimiter also contains information indicating the target layer that allows LA. For example, the access unit delimiter can be defined as follows:

[0211]

[0212] In another example of this embodiment, only the incremental maximum target layer is signaled in the access unit delimiter, where a zero increment means that layer access up-switching is not allowed:

[0213]

[0214] In another example of this embodiment, the incremental maximum target layer is signaled in a specific non-VCL layer access (LA) NAL unit, where the presence of the specific NAL unit indicates that up-switching is possible and the maximum target layer is specified by an explicit codeword:

[0215]

[0216] In another variation of this embodiment, one or more syntax elements may also define a sparse set of target layers that can be switched up to. For example, if the bitstream includes layers 0 to 5 and the decoder decodes layers 0 and 1, one or more syntax elements may indicate that it is possible to switch up to target layers 3 and 5, which means that the decoder can decode layers 0, 1, 3, and 5, but not layers 2 and 4.

[0217] 1.10 Buffer Parameter Set

[0218] In one embodiment, the access unit includes a NAL unit including an LA indication and also includes a NAL unit including a parameter set, and process 600 further includes: the decoder receives the NAL unit including the parameter set before receiving the NAL unit including the LA indication; and the decoder buffers (i.e., stores) the parameter set. In some embodiments, the LA position is the position of the NAL unit including the parameter set in the bitstream.

[0219] In one embodiment, the decoder may buffer a parameter set that is part of an access unit until it determines whether a picture in the access unit uses the parameter set. If the decoder determines that a picture in the access unit uses the parameter set, the decoder decodes the cached parameter set and decodes the picture using the decoded parameter set. If the decoder determines that no picture in the access unit uses the parameter set, the decoder discards the stored parameter set.

[0220] Fig. 9 is a block diagram of an apparatus 900 for implementing the video encoder 140 or the video decoder 260 according to some embodiments. Fig. 9As shown, the device 900 may include: a processing circuit (PC) 902, which may include one or more processors (P) 955 (e.g., a general-purpose microprocessor and / or one or more other processors, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.), which may be co-located in a single housing or in a single data center, or may be geographically distributed (i.e., the device 900 may be a distributed computing device); a network interface 948, including a transmitter (Tx) 945 and a receiver (Rx) 947, for enabling the device 900 to send data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network), the network interface 948 being connected (directly or indirectly) to the network 110 (e.g., the network interface 948 may be wirelessly connected to the network 110, in which case the network interface 948 is connected to an antenna arrangement); and a local storage unit (also referred to as a "data storage system") 908, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where the PC 902 includes a programmable processor, a computer program product (CPP) 941 may be provided. The CPP 941 includes a computer readable medium (CRM) 942 storing a computer program (CP) 943 including computer readable instructions (CRI) 944. The CRM 942 may be a non-transitory computer readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a storage device (e.g., a random access memory, a flash memory), etc. In some embodiments, the CRI 944 of the computer program 943 is configured such that when executed by the PC 902, the CRI causes the device 900 to perform the steps described herein (e.g., the steps described herein with reference to the flowcharts). In other embodiments, the device 900 may be configured to perform the steps described herein without requiring code. That is, for example, the PC 902 may consist solely of one or more ASICs. Thus, the features of the embodiments described herein may be implemented in hardware and / or software.

[0221] Fig. 10A The functional units of the video decoding apparatus 1001 according to the embodiment are illustrated.

[0222] Fig. 10B Functional units of the video encoding device 1021 according to an embodiment are illustrated.

[0223] Fig. 10C Functional units of the video decoding device 1031 according to an embodiment are illustrated.

[0224] abbreviation

[0225] ALF Adaptive Loop Filter

[0226] APS Adaptation Parameter Set

[0227] AUD Access Unit Delimiter

[0228] BLA Broken Link Access

[0229] CRA Clean Random Access

[0230] CVS encoded video sequence

[0231] CVSS CVS Start

[0232] CU Coding Unit

[0233] DPS Decoder Parameter Set

[0234] GRA Progressive Random Access

[0235] HEVC High Efficiency Video Coding

[0236] IDR Instantaneous Decode Refresh

[0237] IRAP Intra-frame random access point

[0238] JVET Joint Video Exploration Team

[0239] LA Level Access

[0240] LMCS Luma Mapping and Chroma Scaling

[0241] MPEG Moving Picture Experts Group

[0242] NAL Network Abstraction Layer

[0243] PES Packetized Elementary Stream

[0244] PPS Picture Parameter Set

[0245] RADL Random Access Decodable Preamble

[0246] RASL Random Access Skip Preamble

[0247] SPS Sequence Parameter Set

[0248] STSA Stepwise Temporal Sublayer Access

[0249] TSA Time Sublayer Access

[0250] VCL Video Coding Layer

[0251] VPS Video Parameter Set

[0252] VVC Versatile Video Coding

[0253] SEI Supplementary Enhancement Layer

[0254] Although various embodiments (including the attached contributions) are described herein, it should be understood that they are presented as examples only and not as limitations. Therefore, the breadth and scope of the present disclosure should not be limited by any of the above-mentioned exemplary embodiments. In addition, any combination of the above-mentioned elements with all possible variations thereof is included in the present disclosure, unless otherwise indicated or otherwise clearly conflicting with the context.

[0255] Additionally, although the processes described above and shown in the accompanying drawings are shown as a series of steps, this is for illustrative purposes only. Therefore, it is contemplated that some steps may be added, some steps may be omitted, the order of steps may be rearranged, and some steps may be performed in parallel.

Claims

1. A method (600) for decoding a set of pictures from a bitstream, the method include: obtaining (s602) a temporal layer access, LA, indication, the LA indication identifying a LA picture in an access unit, the access unit comprising an ordered sequence of NAL units, wherein i) the LA picture has a temporal layer identifier value identifying a temporal layer, and ii) the LA picture is followed by other pictures in decoding order; determining (s604) a LA position in said bitstream based on said LA indication; and decoding the LA picture and the other pictures following the LA picture in the decoding order (s606), wherein the LA picture or any other picture i) that follows the LA picture in the decoding order and ii) has a temporal layer identifier value equal to the temporal layer identifier value of the LA picture is decoded without using a parameter set that a) has a specific type, b) has a temporal layer identifier value equal to the temporal layer identifier value of the LA picture, and c) precedes the LA position, and The LA position is determined to be the position of a first NAL unit in the ordered sequence of NAL units.

2. The method according to claim 1, in, The LA indication is a step-by-step temporal sub-layer access (STSA) indication, and the LA picture is a STSA picture.

3. The method according to claim 1, in, The specific type is any type included in a set of one or more specific types, wherein the set of one or more specific types includes an adaptation parameter set APS type.

4. The method according to claim 1, in, The specific type is any type included in a set of one or more specific types, wherein the set of one or more specific types includes a picture parameter set PPS type.

5. The method according to claim 1, in, Obtaining the LA indication includes: deriving the LA indication from a VCL NAL unit type.

6. The method according to claim 1, in, The temporal layer identifier value of the LA picture is not equal to the temporal layer value of the temporal layer to which the LA picture belongs.

7. A method (800) for encoding a picture, the method include: encoding a temporal layer access LA picture (s802), wherein the LA picture has a temporal layer identifier value identifying a temporal layer; encoding an LA indication indicating the LA picture, so that a decoder can identify the LA picture according to the LA indication, wherein an LA position in a bitstream can be identified based on the LA indication; and encoding other pictures following the LA picture in decoding order (s806), wherein any picture having a temporal layer identifier value equal to the temporal layer identifier value of the LA picture in the LA picture or in other pictures following the LA picture in the decoding order is not encoded using a) a specific type, b) a temporal layer identifier value equal to the temporal layer identifier value of the LA picture, and c) a parameter set before the LA position, An access unit consists of an ordered sequence of NAL units, and The LA position is determined to be the position of a first NAL unit in the ordered sequence of NAL units. 8 . The method according to claim 7 , further comprising outputting, by an encoder: the LA picture, an LA indication identifying the LA picture, and the other pictures following the LA picture in the decoding order.

9. The method according to claim 7, in, The specific type is any type included in a set of one or more specific types, wherein the set of one or more specific types includes an adaptation parameter set APS type.

10. The method according to claim 7, in, The specific type is any type included in a set of one or more specific types, wherein the set of one or more specific types includes a picture parameter set PPS type.

11. The method according to claim 7, in, The temporal layer identifier value of the LA picture is not equal to the temporal layer value of the temporal layer to which the LA picture belongs.

12. A video decoder comprising a processor and a memory, wherein the memory stores instructions, which when executed, cause the video decoder to perform the method according to any one of claims 1 to 6.

13. A video encoder comprising a processor and a memory, the memory storing instructions which, when executed, cause the video encoder to perform the method according to any one of claims 7 to 11.

14. A computer readable storage medium (942) having stored thereon a computer program (943) comprising instructions (944) which, when executed by a processing circuit (902), cause the processing circuit (902) to perform the method according to any one of claims 1 to 11.