System and method for encoding and decoding video stream

Abstract

Systems and methods for decoding an encoded video stream are provided. A method comprises: receiving an encoded video stream comprising an access unit, the access unit comprising a picture; signaling a first flag in an access unit delimiter of the encoded video stream, the first flag indicating whether the access unit includes any one of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; signaling a second flag in a picture header of the encoded video stream, the second flag indicating whether the picture is an IRAP picture; the picture is decoded into the current picture based on signaling the first flag and the second flag, where the value of the first flag and the value of the second flag are equal.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 037,903, filed on June 11, 2020; U.S. Provisional Application No. 63 / 036,335, filed on June 8, 2020; U.S. Provisional Application No. 63 / 035,274, filed on June 5, 2020; U.S. Provisional Application No. 63 / 027,826, filed on May 20, 2020; and U.S. Application No. 17 / 320,764, filed on May 14, 2021, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0002] The present disclosure relates to video coding and decoding technology, and more particularly, to a method and system for decoding a video stream. Background Art

[0003] There have been previous efforts to codec video using motion compensated inter-picture prediction. An uncompressed digital video may include a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures may have a fixed or variable picture rate (also informally referred to as a frame rate), such as 60 pictures per second or 60 Hz. Uncompressed video has very high bitrate requirements. For example, 1080p60 4:2:0 video (1920×1080 luma sample resolution at 60 Hz frame rate) with 8 bits per sample requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires over 600 GB of storage space.

[0004] One purpose of video codecs can be to reduce redundancy in the input video signal through compression. Compression can help reduce the bandwidth or storage space requirements mentioned above, in some cases by two orders of magnitude or more. Lossless compression and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to techniques that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be the same as the original signal, but the distortion between the original signal and the reconstructed signal can be small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely adopted. The amount of distortion tolerated depends on the application; for example, users of certain consumer streaming applications will tolerate higher distortion than users of TV contribution applications. The achievable compression ratio can be reflected as follows: higher allowable / tolerable distortion can lead to higher compression ratios.

[0005] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transforms, quantization, and entropy coding, some of which are described below.

[0006] Previously, video encoders and decoders tended to operate on a given picture size, which in most cases was defined and remained constant for a coded video sequence (CVS), a Group of Pictures (GOP) or similar multi-picture temporal frame. For example, in MPEG-2, the system was designed to change the horizontal resolution (and thus the picture size) depending on factors such as scene activity, but only for I pictures, and therefore usually for GOPs. Resampling of reference pictures to use different resolutions within a CVS has been used, for example, in Annex P of ITU-T Recommendation H.263. However, here the picture size is not changed, only the reference picture is resampled, resulting in the possibility that only part of the picture canvas is used (in the case of downsampling), or only part of the scene is captured (in the case of upsampling). In addition, Annex Q of H.263 allows resampling of individual macroblocks up or down by a factor of 2 (in each dimension). Again, the picture size remains constant. The size of the macroblock is fixed in H.263 and therefore does not need to be signaled.

[0007] Changes in picture size in predicted pictures have become more mainstream in modern video coding. For example, VP9 allows reference picture resampling and resolution changes for the entire picture. Similarly, certain proposals for VVC (including, for example, Hendry et al., "On adaptive resolution change (ARC) for VVC", Joint Video Team document JVET-M0135-v1, January 9-19, 2019, incorporated herein in its entirety) allow resampling of the entire reference picture at different (higher or lower) resolutions. In such documents, it is proposed that different candidate resolutions be encoded in a sequence parameter set and referenced by each picture syntax element in the picture parameter set.

[0008] Bross et al., “Versatile Video Coding (Draft 9)”, Joint Video Experts Group document JVET-R2001-vA, April 2020, which is incorporated herein in its entirety.

[0009] In a coded video stream, it is widely used to indicate random access point information in a high-level syntax structure, such as a network abstraction layer (NAL) unit header, a parameter set, a picture header, or a slice header. Based on this random access information, the decoded leading pictures associated with the random access pictures are managed. In the present disclosure, in order to make the decoded picture management associated with the random access process clear, some relevant syntax elements and constraints are explained.

[0010] When a video bitstream is randomly accessed through trick mode playback, an intra random access point (IRAP) picture can achieve random access to the middle point of the bitstream and successfully decode the video bitstream of the random access point. One possible method is to gradually refresh the scene using a certain recovery time. In VVC and other video codecs, gradual decoding refresh (GDR) pictures and access units (AU) are defined to specify the syntax and semantics of random access operations with gradual decoding refresh. In this disclosure, in order to correctly specify the signaling and decoding process of GDR, its syntax, semantics, and constraints are described.

[0011] When one or more reference picture lists are constructed for inter prediction in a P or B slice, one or more pictures may be unavailable due to random access or unexpected picture loss. To avoid any decoder crash or unexpected behavior, unavailable pictures need to be generated with default values ​​for pixels and parameters. After generating unavailable pictures, it may be necessary to check the validity of all reference pictures in the reference picture list. Therefore, how to generate unavailable pictures due to random access or accidental picture loss to avoid decoder crash or unexpected behavior is still a problem that needs to be further studied and solved in this field. Summary of the invention

[0012] Embodiments of the present disclosure relate to random access pictures and output processes thereof in a coded video stream having multiple layers. Embodiments of the present disclosure relate to random access pictures and leading picture output indications thereof in a coded video stream having multiple layers. Embodiments of the present disclosure relate to signaling random access pictures with progressive decoding refresh and recovery points in a coded video stream having multiple layers. Embodiments of the present disclosure relate to reference picture list construction and unavailable picture generation in a coded video stream having multiple layers. Embodiments of the present disclosure include techniques for signaling adaptive picture sizes in a video bitstream.

[0013] One or more embodiments of the present disclosure include a method for decoding a video stream. The method includes: receiving an encoded video stream including an access unit, the access unit including a picture; signaling a first flag in an access unit delimiter of the encoded video stream, the first flag indicating whether the access unit includes any one of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; signaling a second flag in a picture header of the encoded video stream, the second flag indicating whether the picture is an IRAP picture; and decoding the picture as a current picture based on the first flag and the second flag, wherein the value of the first flag and the value of the second flag are equal.

[0014] According to an embodiment, the method further comprises: signaling a third flag in a picture header of the coded video stream, the third flag indicating whether the picture is a GDR picture, wherein the value of the first flag and the value of the third flag are equal.

[0015] According to an embodiment, based on the second flag indicating that the picture is not an IRAP picture, the third flag is signaled.

[0016] According to an embodiment, the first flag has a value indicating that the picture is any one of an IRAP picture and a GDR picture, the second flag has a value indicating that the picture is an IRAP picture, and the method also includes signaling a fifth flag in a slice header of a slice of a picture of the encoded video stream, the fifth flag indicating whether any picture before the IRAP picture is output.

[0017] According to an embodiment, the method further comprises: determining a network abstraction layer (NAL) unit type of the slice, wherein a fifth flag is signaled based on the determined NAL unit type.

[0018] According to an embodiment, the fifth flag is signaled based on the NAL unit type being determined to be equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT.

[0019] According to an embodiment, the method further comprises: signaling a fourth flag in a picture header of the coded video stream, the fourth flag indicating whether the picture is a GDR picture, wherein the value of the first flag and the value of the fourth flag are equal.

[0020] According to an embodiment, the fifth flag is signaled based on the second flag indicating that the picture is not an IRAP picture.

[0021] According to an embodiment, decoding includes: constructing a reference picture list; generating unavailable reference pictures in the reference picture list; and checking bitstream consistency, for reference pictures in the reference picture list, the following constraints apply: the number of entries indicated in the reference picture list is not less than the number of active entries indicated in the reference picture list, each picture referred to by the active entries in the reference picture list exists in a decoded picture buffer (DPB) and has a time identifier value less than or equal to the time identifier value of the current picture, and each picture referred to by the entry in the reference picture list is not the current picture and is indicated by a picture header flag as a potential reference picture.

[0022] According to an embodiment, checking bitstream consistency is performed based on determining whether the current picture is an independent decoder refresh (IDR) picture, a clean random access (CRA) picture, or a progressive decoding refresh (GDR) picture.

[0023] According to one or more embodiments, a system for decoding a video stream is provided. The system includes: at least one processor configured to receive an encoded video stream including an access unit, the access unit including a picture; and a memory storing computer code, the computer code including: a first signaling code configured to cause at least one processor to signal a first flag in an access unit delimiter of the encoded video stream, the first flag indicating whether the access unit includes any one of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; a second signaling code configured to cause at least one processor to signal a second flag in a picture header of the encoded video stream, the second flag indicating whether the picture is an IRAP picture; and a decoding code configured to cause at least one processor to decode the picture as a current picture based on the first flag and the second flag, wherein the value of the first flag and the value of the second flag are equal.

[0024] According to an embodiment, the computer code further comprises: A third signaling code is configured to cause at least one processor to signal a third flag in a picture header of the coded video stream, the third flag indicating whether the picture is a GDR picture, wherein a value of the first flag and a value of the third flag are equal.

[0025] According to an embodiment, based on the second flag indicating that the picture is not an IRAP picture, the third flag is signaled.

[0026] According to an embodiment, the first flag has a value indicating that the picture is any one of an IRAP picture and a GDR picture, the second flag has a value indicating that the picture is an IRAP picture, and the computer code also includes a fifth signaling code, which is configured to cause at least one processor to signal a fifth flag in a slice header of a slice of a picture of the encoded video stream, the fifth flag indicating whether any picture before the IRAP picture is output.

[0027] According to an embodiment, the computer code further comprises: a determination code configured to cause at least one processor to determine a network abstraction layer (NAL) unit type of the slice, wherein the fifth flag is signaled based on the determined NAL unit type.

[0028] According to an embodiment, the fifth flag is signaled based on the NAL unit type being determined to be equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT.

[0029] According to an embodiment, the computer code also includes: a fourth signaling code, which is configured to cause at least one processor to signal a fourth flag in a picture header of the encoded video stream, the fourth flag indicating whether the picture is a GDR picture, wherein the value of the first flag is equal to the value of the fourth flag.

[0030] According to an embodiment, the fifth flag is signaled based on the second flag indicating that the picture is not an IRAP picture.

[0031] According to an embodiment, the decoding code includes: a construction code configured to cause at least one processor to construct a reference picture list; a generation code configured to cause at least one processor to generate an unavailable reference picture in the reference picture list; and a checking code configured to cause at least one processor to check bitstream consistency, for reference pictures in the reference picture list, the following constraints apply: the number of entries indicated in the reference picture list is not less than the number of activated entries indicated in the reference picture list, each picture referred to by the activated entry in the reference picture list exists in a decoded picture buffer (DPB) and has a time identifier value less than or equal to the time identifier value of the current picture, and each picture referred to by the entry in the reference picture list is not the current picture and is indicated by a picture header flag as a potential reference picture.

[0032] According to one or more embodiments, a non-transitory computer-readable medium storing computer instructions is provided. The computer instructions, when executed by at least one processor, receive a coded video stream including an access unit, the access unit including a picture, and cause the at least one processor to: signal a first flag in an access unit delimiter of the coded video stream, the first flag indicating whether the access unit includes any one of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; signal a second flag in a picture header of the coded video stream, the second flag indicating whether the picture is an IRAP picture; and decode the picture as a current picture based on the first flag and the second flag, wherein the value of the first flag and the value of the second flag are equal. According to an embodiment, a system for decoding a video stream is provided, the system comprising: a first signaling unit configured to signal a first flag in an access unit delimiter of an encoded video stream, the first flag indicating whether the access unit includes any one of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; a second signaling unit configured to signal a second flag in a picture header of the encoded video stream, the second flag indicating whether the picture is an IRAP picture; and a decoding unit configured to decode the picture as a current picture based on the first flag and the second flag, wherein the value of the first flag and the value of the second flag are equal. According to an embodiment, a computer device is provided, the device comprising a processor and a memory. The memory is used to store program codes and transmit the program codes to the processor; the processor is used to execute the above method according to instructions in the program codes. According to the system and method for decoding a video stream provided by an embodiment of the present disclosure, a coded video stream including an access unit is received, the access unit including a picture; a first flag is signaled in an access unit delimiter of the coded video stream, the first flag indicating whether the access unit includes any one of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; a second flag is signaled in a picture header of the coded video stream, the second flag indicating whether the picture is an IRAP picture; based on the first flag and the second flag, the picture is decoded as a current picture, wherein the value of the first flag and the value of the second flag are equal. Through the present disclosure, if a reference picture is determined to be unavailable, the unavailable reference picture can be marked as "no reference picture". In order to avoid any decoder crash or unexpected behavior, an unavailable reference picture can be immediately generated with default values ​​of pixels and parameters. After generating an unavailable reference picture (and / or the reference picture is determined to be available), the decoder can check the validity of all reference pictures (including the generated picture) in the reference picture list. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0034] Figure 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0035] Figure 2 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0036] Figure 3 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0037] Figure 4 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0038] Figure 5A is a schematic diagram of a first configuration for signaling ARC parameters according to a comparative embodiment.

[0039] Figure 5B is a schematic diagram of a second configuration for signaling ARC parameters according to a comparative embodiment.

[0040] Fig. 6A is a schematic diagram of a first configuration for signaling ARC parameters according to an embodiment.

[0041] Figure 6B is a schematic diagram of a second configuration for signaling ARC parameters according to an embodiment.

[0042] Figure 6C is a diagram of a third configuration for signaling ARC parameters according to an embodiment.

[0043] Fig. 7A is a schematic diagram of an excerpt of a tile group header according to an embodiment.

[0044] Figure 7B is a schematic diagram of an excerpt of a sequence parameter set according to an embodiment.

[0045] Figure 8 is an example of a prediction structure with scalability for adaptive resolution changes.

[0046] Fig.9A An example of a syntax table according to an embodiment is shown.

[0047] Fig. 9B An example of a syntax table according to an embodiment is shown.

[0048] Fig.10 is a schematic diagram of a simplified block diagram for parsing and decoding a POC period and an access unit count value for each access unit according to an embodiment.

[0049] Fig.11 is a schematic diagram of a video bitstream structure including multiple layers of sub-pictures according to an embodiment.

[0050] Fig.12 is a schematic diagram of display of a selected sub-picture with enhanced resolution according to an embodiment.

[0051] Fig.13 is a block diagram of a decoding and display process for a video bitstream including multiple layers of sub-pictures according to an embodiment.

[0052] Fig.14 is a schematic diagram of a 360 video display with an enhancement layer of sub-pictures according to an embodiment.

[0053] Fig.15A An example of a layout of divided sub-pictures according to an embodiment is shown.

[0054] Fig. 15B An example of corresponding sub-picture size and position information of one sub-picture according to an embodiment is shown.

[0055] Fig.16 Shows FIG. 15A to FIG. 15B The corresponding picture prediction structure of the sub-picture shown in .

[0056] Fig.17 An example of an input picture divided into multiple sub-regions that can be encoded with one or more layers according to an embodiment is shown.

[0057] Fig.18 Shows Fig.17 Corresponding layer and picture prediction structure with local region spatial scalability modality for the sub-regions shown in .

[0058] Fig.19A is a schematic diagram of an excerpt of a video parameter set according to an embodiment.

[0059] Fig.19B is a schematic diagram of an excerpt of a sequence parameter set according to an embodiment.

[0060] Fig. 20 is an example of a syntax table for sub-picture layout information according to an embodiment.

[0061] Fig.21 is an example of a syntax table for indicating output layers and profile / tier / level information for each output layer set according to an embodiment.

[0062] Fig. 22 is an example of a syntax table for indicating output layer mode on for each output layer set according to an embodiment.

[0063] Fig.23 is an example of a syntax table for indicating the current sub-picture of each layer of each output layer set.

[0064] Fig.24 This is an example of a syntax table of a video parameter set raw byte sequence payload (RBSP).

[0065] Fig.25 is an example of a syntax table for indicating an output layer set having an output layer set mode.

[0066] Fig.26 is an example of a syntax table for indicating a random access point associated with an IRAP or GDR picture in a picture header.

[0067] Fig. 27 is an example of a syntax table for indicating a random access AU in an AU delimiter.

[0068] Fig.28 is an example of a syntax table used to indicate the presence of an IRAP picture in a picture header.

[0069] Fig.29 is an example of a syntax table for indicating output of a leading picture in a slice header.

[0070] Fig.30 This is an example of a syntax table for indicating the presence of an IRAP picture and a GDR picture in an AU delimiter.

[0071] Fig.31 is an example of a syntax table used to indicate a GDR picture and its recovery point in a picture header.

[0072] Fig.32 is a schematic diagram of a simplified block diagram of reference picture list construction and generation of unavailable pictures.

[0073] Fig.33 is a block diagram of computer code according to an embodiment.

[0074] Fig.34 is a diagram of a computer system suitable for implementing the embodiments. DETAILED DESCRIPTION

[0075] Figure 1A simplified block diagram of a communication system (100) according to an embodiment of the present disclosure is shown. The system (100) may include at least two terminals (110, 120) interconnected via a network (150). For unidirectional transmission of data, a first terminal (110) may encode video data at a local location for transmission to another terminal (120) via the network (150). A second terminal (120) may receive encoded video data of another terminal from the network (150), decode the encoded data and display the recovered video data. Unidirectional data transmission may be common in media service applications and the like.

[0076] Figure 1 A second pair of terminals (130, 140) is shown, which are provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal (130, 140) can encode video data captured at a local location for transmission to the other terminal via a network (150). Each terminal (130, 140) can also receive encoded video data sent by the other terminal, can decode the encoded data, and can display the recovered video data on a local display device.

[0077] exist Figure 1 In the embodiment of the present invention, the terminals (110-140) may be shown as servers, personal computers and smart phones, and / or any other type of terminals. For example, the terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (150) represents any number of networks that transmit the encoded video data between the terminals (110-140), including, for example, wired and / or wireless communication networks. The communication network (150) may exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (150) may be irrelevant to the operation of the present disclosure unless explained below.

[0078] As examples of applications of the disclosed subject matter, Figure 2 The placement of the video encoder and decoder in a streaming environment is shown. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc., etc.

[0079] like Figure 2As shown, a streaming system (200) may include a capture subsystem (213) that may include a video source (201) and an encoder (203). The video source (201) may be, for example, a digital camera and may be configured to create an uncompressed video sample stream (202). Compared to an encoded video bitstream, the uncompressed video sample stream (202) may provide a high data volume and may be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement various aspects of the disclosed subject matter as described in more detail below. Compared to a sample stream, the encoded video bitstream (204) may include a lower data volume and may be stored on a streaming server (205) for future use. One or more streaming clients (206) may access the streaming server (205) to retrieve a video bitstream (209), which may be a copy of the encoded video bitstream (204).

[0080] In an embodiment, the streaming server (205) may also function as a Media-Aware Network Element (MANE). For example, the streaming server (205) may be configured to prune the encoded video bitstream (204) to customize a potentially different bitstream for one or more of the streaming clients (206). In an embodiment, the MANE may be provided separately from the streaming server (205) in the streaming system (200).

[0081] The streaming client (206) may include a video decoder (210) and a display (212). For example, the video decoder (210) may decode a video bitstream (209) that is an incoming copy of an encoded video bitstream (204) and create an outgoing video sample stream (211) that may be presented on a display (212) or other rendering device (not depicted). In some streaming systems, the video bitstreams (204, 209) may be encoded according to certain video encoding / compression standards. Examples of such standards include, but are not limited to, ITU-T Recommendation H.265. Under development is a video coding standard informally referred to as Versatile Video Coding (VVC). Implementations of the present disclosure may be used in the context of VVC.

[0082] Figure 3 An example functional block diagram of a video decoder (210) attached to a display (212) is shown in accordance with an embodiment of the present disclosure.

[0083] The video decoder (210) may include a channel (312), a receiver (310), a buffer memory (315), an entropy decoder / parser (320), a scaler / inverse transform unit (351), an intra-frame prediction unit (352), a motion compensation prediction unit (353), an aggregator (355), a loop filter unit (356), a reference picture memory (357), and a current picture memory (358). In at least one embodiment, the video decoder (210) may include an integrated circuit, a series of integrated circuits, and / or other electronic circuit systems. The video decoder (210) may also be partially or completely implemented in software running on one or more CPUs with associated memory.

[0084] In this and other embodiments, the receiver (310) can receive one or more encoded video sequences to be decoded by the decoder (210), one at a time, wherein each encoded video sequence is decoded independently of the other encoded video sequences. The encoded video sequence can be received from a channel (312), which can be a hardware / software link to a storage device storing the encoded video data. The receiver (310) can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective consuming entities (not depicted). The receiver (310) can separate the encoded video sequence from the other data. To combat network jitter, a buffer memory (315) can be coupled between the receiver (310) and the entropy decoder / parser (320) (hereinafter referred to as "parser"). When the receiver (310) receives data from a storage / forward device with sufficient bandwidth and controllability or from a synchronous network, the buffer (315) can be not used, or the buffer (315) can be very small. In order to make the best possible use of a packet network such as the Internet, a buffer (315) may be required, which may be relatively large and may have an adaptive size.

[0085] The video decoder (210) may include a parser (320) to reconstruct symbols (321) from an entropy-coded video sequence. The types of symbols include, for example, information for managing the operation of the decoder (210) and potentially for controlling functions such as those that may be coupled to, for example, Figure 2The control information of the rendering device of the display (212) of the decoder shown. The control information of the (multiple) rendering devices can be in the form of, for example, Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not depicted). The parser (320) can parse / entropy decode the received encoded video sequence. The encoded video sequence can be encoded according to a video coding technique or standard, and can follow principles known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Based on at least one parameter corresponding to the group, the parser (320) can extract a subgroup parameter set for at least one of the subgroups of pixels from the encoded video sequence in the video decoder. The subgroup can include a group of pictures (Group of Pictures, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (320) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0086] The parser (320) may perform entropy decoding / parsing operations on the video sequence received from the buffer (315), thereby creating symbols (321).

[0087] The reconstruction of the symbol (321) may involve a number of different units, depending on the type of coded video picture or portion thereof (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors. Which units are involved and how they are involved may be controlled by sub-group control information parsed from the coded video sequence by the parser (320). For clarity, the flow of such sub-group control information between the parser (320) and the following multiple units is not depicted.

[0088] In addition to the functional blocks already mentioned, decoder 210 can be conceptually subdivided into a number of functional units as described below. In a practical implementation running under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the following functional units.

[0089] One unit may be a sealer / inverse transform unit (351). The sealer / inverse transform unit (351) may receive quantized transform coefficients as symbol(s) (321) from the parser (320) along with control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (351) may output a block including sample values, which may be input into an aggregator (355).

[0090] In some cases, the output samples of the scaler / inverse transform (351) may belong to an intra-coded block; that is, a block that does not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) uses surrounding already-reconstructed information obtained from the current (partially reconstructed) picture from the current picture memory (358) to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator (355) adds the prediction information already generated by the intra-prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351) on a per-sample basis.

[0091] In other cases, the output samples of the scaler / inverse transform unit (351) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (353) may access the reference picture memory (357) to obtain samples for prediction. After the obtained samples are motion compensated according to the symbols (321) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (351) (in this case called residual samples or residual signal) by the aggregator (355) to generate output sample information. The address within the reference picture memory (357) may be controlled by a motion vector, and the motion compensated prediction unit (353) obtains the predicted samples from the reference picture memory. The motion vector may be provided to the motion compensated prediction unit (353) in the form of a symbol (321), which may have, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​obtained from the reference picture memory (357) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0092] The output samples of the aggregator (355) may be subjected to various loop filtering techniques in a loop filter unit (356). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video bitstream as symbols (321) from the parser (320) and available to the loop filter unit (356), but the video compression techniques may also be responsive to meta-information obtained during decoding of a coded picture or a previous (in decoding order) portion of a coded video sequence, and to previously reconstructed and loop filtered sample values.

[0093] The output of the loop filter unit (356) may be a sample stream that may be output to a rendering device such as a display (212) and stored in a reference picture memory (357) for future use in inter-frame prediction.

[0094] Once fully reconstructed, certain coded pictures can be used as reference pictures for future prediction. Once a coded picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by a parser (320)), the current reference picture can become part of the reference picture memory (357) and a new current picture memory can be reallocated before starting to reconstruct a subsequent coded picture.

[0095] The video decoder (210) may perform decoding operations according to a predetermined video compression technique, which may be documented in a standard such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard being used, insofar as it complies with the syntax of the video compression technology or standard as specified in a video compression technology document or standard, particularly a profile therein. In addition, in order to comply with certain video compression technologies or standards, the complexity of the encoded video sequence may be within a range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0096] In an embodiment, the receiver (310) may receive additional (redundant) data with the encoded video. The additional data may be included as part of the encoded video sequence (s). The video decoder (210) may use the additional data to correctly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0097] Figure 4 An example functional block diagram of a video encoder (203) associated with a video source (201) is shown according to an embodiment of the present disclosure.

[0098] The video encoder (203) may include, for example, an encoder as a source encoder (430), a coding engine (432), a (local) decoder (433), a reference picture memory (434), a predictor (435), a transmitter (440), an entropy encoder (445), a controller (450), and a channel (460).

[0099] The encoder (203) may receive video samples from a video source (201) (not part of the encoder), which may capture video image(s) to be encoded by the encoder (203).

[0100] The video source (201) may provide a source video sequence to be encoded by the encoder (203) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (201) may be a storage device storing pre-prepared videos. In a video conferencing system, the video source (201) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, where each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0101] According to an embodiment, the encoder (203) can encode and compress the pictures of the source video sequence into an encoded video sequence (443) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (450). The controller (450) can also control other functional units as described below and can be functionally coupled to these units. For clarity, the coupling is not depicted. The parameters set by the controller (450) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, picture group (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller (450) because they may be related to the video encoder (203) optimized for a certain system design.

[0102] Some video encoders operate in a manner that is readily recognizable to those skilled in the art as a "coding loop". As an oversimplified description, the coding loop may consist of the encoding portion of a source encoder (430) (responsible for creating symbols based on the input picture to be encoded and (multiple) reference pictures) and a (local) decoder (433) embedded in the encoder (203), which reconstructs the symbols to create the following sample data, which the (remote) decoder also creates when the compression between the symbols and the encoded video bit stream is lossless in certain video compression techniques. This reconstructed sample stream may be input to a reference picture memory (434). Since the decoding of the symbol stream results in a bit-accurate result that is independent of the decoder location (local or remote), the reference picture memory contents also correspond bit-by-bit between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. The basic principles of reference picture synchronization (and the drift that results when synchronization cannot be maintained, for example due to channel errors) are known to those skilled in the art.

[0103] The operation of the "local" decoder (433) can be combined with the above Figure 3 The operation of the "remote" decoder (210) described in detail is the same. However, when symbols are available and can be losslessly encoded / decoded into a coded video sequence by the entropy encoder (445) and parser (320), the entropy decoding portion of the decoder (210) - including the channel (312), receiver (310), buffer (315) and parser (320) - may not be fully implemented in the local decoder (433).

[0104] It can be observed at this point that any decoder technology other than the parsing / entropy decoding present in the decoder may need to be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter focuses on the decoder operation. Because the encoder technology may be contrary to the decoder technology described comprehensively, the description of the encoder technology can be reduced. Only in certain areas a more detailed description is needed and is provided below.

[0105] As part of its operation, the source encoder (430) may perform motion compensated predictive coding, which predictively encodes an input frame with reference to one or more previously encoded frames from a video sequence designated as "reference frames." In this manner, the encoding engine (432) encodes the differences between pixel blocks of the input frame and pixel blocks of (multiple) reference frames, which may be selected as prediction reference (s) for the input frame.

[0106] The local video decoder (433) can decode the encoded video data of the frame that can be designated as the reference frame based on the symbol created by the source encoder (430). The operation of the encoding engine (432) can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 4 When the video sequence is decoded at a remote video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (433) replicates the decoding process that may be performed by the video decoder on the reference frame and may cause the reconstructed reference frame to be stored in the reference picture memory (434). In this way, the encoder (203) may store a copy of the reconstructed reference frame locally that has the same content (without transmission errors) as the reconstructed reference frame that will be obtained by the remote video decoder.

[0107] The predictor (435) may perform a prediction search for the encoding engine (432). That is, for a new frame to be encoded, the predictor (435) may search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (435) may operate on a pixel block by pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (435), the input picture may have a prediction reference extracted from a plurality of reference pictures stored in the reference picture memory (434).

[0108] The controller (450) may manage encoding operations of the source encoder (430), including, for example, the setting of parameters and sub-group parameters for encoding video data.

[0109] The outputs of all the above functional units may be entropy encoded in an entropy encoder (445). The entropy encoder converts the symbols generated by each functional unit into an encoded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0110] The transmitter (440) can buffer the encoded video sequence(s) created by the entropy encoder (445) in preparation for transmission via a communication channel (460), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (440) can combine the encoded video data from the source encoder (430) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0111] The controller (450) can manage the operation of the encoder (203). During encoding, the controller (450) can assign a specific coded picture type to each coded picture, which can affect the coding techniques that can be applied to the corresponding picture. For example, a picture can generally be assigned as an intra picture (I picture), a predicted picture (P picture), or a bidirectional predicted picture (B picture).

[0112] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh (IDR) pictures. Those skilled in the art are aware of those variants of I pictures and their respective applications and features.

[0113] A predicted picture (P picture) may be a picture that may be encoded and decoded using intra prediction or inter prediction, which predicts sample values ​​of each block using at most one motion vector and a reference index.

[0114] Bidirectional prediction pictures (B pictures) can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-prediction pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0115] The source picture can typically be spatially subdivided into multiple blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and encoded block by block. Blocks can be predictively encoded with reference to other (already encoded) blocks determined by the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture can be non-predictively encoded or they can be predictively encoded with reference to already encoded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture can be non-predictively encoded with reference to one previously encoded reference picture via spatial prediction or via temporal prediction. B picture blocks can be non-predictively encoded with reference to one or two previously encoded reference pictures via spatial prediction or via temporal prediction.

[0116] The video encoder (203) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In its operation, the video encoder (203) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard being used.

[0117] In an embodiment, the transmitter (440) may transmit additional data when transmitting the encoded video. The source encoder (430) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.

[0118] Before describing certain aspects of embodiments of the present disclosure in greater detail, some terms referred to in the remainder of the description are introduced below.

[0119] In some cases, in the following, "sub-picture" refers to a rectangular arrangement of samples, blocks, macroblocks, coding units, or similar entities that are semantically grouped and can be independently encoded at varying resolutions. One or more sub-pictures can form a picture. One or more encoded sub-pictures can form an encoded picture. One or more sub-pictures can be combined into a picture, and one or more sub-pictures can be extracted from a picture. In some circumstances, one or more encoded sub-pictures can be combined in the compressed domain without transcoding to the sample level to become an encoded picture, and in the same or certain other circumstances, one or more encoded sub-pictures can be extracted from the encoded picture in the compressed domain.

[0120] In the following, "Adaptive Resolution Change" (ARC) refers to a mechanism that allows the resolution of pictures or sub-pictures within a coded video sequence to be changed, for example, by resampling reference pictures. "ARC parameters" in the following refers to the control information required to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, resolution of output and / or reference pictures, various control flags, etc.

[0121] The above description focuses on encoding and decoding a single semantically independent coded video picture.Before describing the encoding / decoding of multiple sub-pictures with independent ARC parameters and the implications of the additional complexity it implies, an implementation for signaling ARC parameters will be described.

[0122] Reference FIG. 6A to FIG. 6C , several novel example implementations for signaling ARC parameters are shown. As noted in each of the implementations, they have certain advantages from a coding efficiency, complexity, and architecture perspective. A video coding standard or technology may implement one or more of these implementations, and may also include implementations known in the comparative art for signaling ARC parameters. Comparative art implementations include FIG. 5A to FIG. 5B The novel embodiments may not be mutually exclusive and it is contemplated that they may be included in a standard or technology that also includes comparative technology embodiments, such that one of the novel embodiments may be used based on application needs, the standard technology involved, or the choice of encoder.

[0123] Categories of ARC parameters may include: (1) up / down sampling factors separated or combined in the X and Y dimensions, or (2) up / down sampling factors with an added time dimension, which indicates a constant speed up / down scaling for a given number of pictures. One of the above two may involve encoding or decoding one or more syntax elements, which may point to a table containing the factors. In an embodiment, the length of such syntax elements may be shorter.

[0124] "Resolution" may refer to the resolution of a combined or individual input picture, output picture, reference picture, coded picture in terms of samples, blocks, macroblocks, CUs, or any other suitable granularity in the X or Y dimension. If there is more than one resolution (e.g., one for input pictures and one for reference pictures), in some cases one set of values ​​may be inferred from another set of values. For example, the resolution may be gated using a flag. More detailed examples of resolution are provided further below.

[0125] "Warped" coordinates, similar to those used in H.263 Annex P, can have a suitable granularity as described above. H.263 Annex P defines an efficient way to encode such warped coordinates, but other, potentially more efficient ways may also be considered. For example, the variable-length reversible "Huffman"-style encoding of the warped coordinates of Annex P can be replaced by a binary encoding of a suitable length, where the length of the binary codeword can be derived, for example, from the maximum picture size, possibly multiplied by a specific factor and offset by a specific value, thereby allowing "warping" outside the boundaries of the maximum picture size.

[0126] With respect to upsampling or downsampling filter parameters, in the simplest case, there may be only a single filter for upsampling and / or downsampling. However, in some cases, it may be advantageous to allow more flexibility in filter design, which may be achieved by signaling filter parameters. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g., by a list of filter coefficients using an appropriate entropy coding technique), and / or the filter may be implicitly selected by an up / down sampling ratio signaled according to any of the mechanisms mentioned above, and so on.

[0127] Hereinafter, this specification assumes the following example case: the encoding of a finite set of up / down sampling factors (the same factors used for both the X dimension and the Y dimension) is indicated by a codeword. The codeword can advantageously be variable length encoded, for example, using Ext-Golomb codes common to certain syntax elements in video coding specifications such as H.264 and H.265. For example, a suitable mapping of values ​​to up / down sampling factors can be according to Table 1 below. Table 1 Codeword Ext-Golomb Code Original / target resolution 0 1 1 / 1 1 010 1 / 1.5 (increase by 50%) 2 011 1.5 / 1 (50% reduction) 3 00100 1 / 2 (100% increase) 4 00101 2 / 1 (reduced by 100%)

[0128] Many similar mappings can be designed depending on the needs of the application and the capabilities of the upscaling and downscaling mechanisms available in the video compression technology or standard. The table can be extended to more values. The values ​​can also be represented by entropy coding mechanisms other than Ext-Golomb codes (e.g., using binary encoding), which can have certain advantages when, for example, the MANE is interested in the resampling factors outside the video processing engine (encoder and decoder most importantly) itself. It should be noted that for the (probably) most common case where no resolution change is required, a short Ext-Golomb code can be chosen (e.g., only a single bit as shown in the second row of Table 1), which has coding efficiency advantages over using binary codes for the most common cases.

[0129] The number of entries in the table and their semantics may be fully or partially configured. For example, the basic outline of the table may be communicated in a "high" parameter set such as a sequence or decoder parameter set. Alternatively or additionally, one or more such tables may be defined in a video coding technique or standard, and one or more such tables may be selected by, for example, a decoder or sequence parameter set.

[0130] Provided below is a description of how to include the upsampling / downsampling factors (ARC information) encoded as described above in a video coding technique or standard syntax. Similar considerations can be applied to one or more codewords that control up / downsampling filters. Also provided below is a description of when a filter or other data structure may require a relatively large amount of data.

[0131] Reference Figure 5A , H.263 Annex P includes ARC information 502 in the form of four warp coordinates into the picture header 501, specifically in the H.263PLUSPTYPE (503) header extension. Such a design may be reasonable when (a) picture headers are available, and (b) frequent changes in ARC information are expected. However, because picture headers may have transient characteristics, the overhead when using H.263-type signaling may be quite high, and the scaling factor may not be related to picture boundaries.

[0132] Reference Figure 5B , JVCET-M135-v1 includes ARC reference information (505), (indexed) located in a picture parameter set (504), which indexes a table (506) that includes the target resolution located in a sequence parameter set (507). The placement of feasible resolutions in the table (506) in the sequence parameter set (507) can be justified by using the SPS (507) as an interoperability negotiation point during capability exchange. By referencing the appropriate picture parameter set (504), the resolution can be changed on a picture-by-picture basis within the limits set by the values ​​in the table (506).

[0133] Reference FIG. 6A to FIG. 6C , the following embodiments of the present disclosure can transmit the ARC information in the video bitstream to, for example, a decoder of the present disclosure. Each of these embodiments has certain advantages over the comparative techniques described above. These embodiments can exist simultaneously in the same video coding technology or standard.

[0134] In reference Fig. 6A In an embodiment, for example, ARC information (509) such as a resampling (scaling) factor may be present in a header (508) such as a slice header, a GOB header, a tile header, or a tile group header. As an example, Fig. 6AThe header (508) is shown as a tile group header. For example, as shown in Table 1, such a configuration may be sufficient if the ARC information (e.g., a single variable length ue(v) or a fixed length codeword of several bits) is small. Placing the ARC information directly in the tile group header has the following additional advantage: the ARC information may be applicable to, for example, a sub-picture represented by a tile group corresponding to the tile group header rather than the entire picture. In addition, even if the video compression technique or standard only uses adaptive resolution changes for the entire picture (e.g., compared to adaptive resolution changes based on tile groups), placing the ARC information in the tile group header (e.g., in an H.263-type picture header) also has certain advantages from an error recovery perspective. Although the above description describes ARC information (509) present in a tile group header, it should be understood that the above description may also be similarly applied to situations where the ARC information (509) is present in, for example, a slice header, a GOB header, or a tile header.

[0135] In reference Figure 6B In the same embodiment or another embodiment of the ARC information (512) itself may be present in an appropriate parameter set (511) such as a picture parameter set, a header parameter set, a tile parameter set, an adaptation parameter set, etc. As an example, Figure 6B A parameter set (511) is shown as an adaptive parameter set (APS). Advantageously, the scope of the parameter set may be no larger than a picture. For example, the scope of the parameter set may be a group of tiles. The use of ARC information (512) may be implied by activating a related parameter set. For example, when a video coding technique or standard considers ARC based only on pictures, then a picture parameter set or equivalent may be suitable as a related parameter set.

[0136] In reference Figure 6C In the same or another embodiment of the present invention, the ARC reference information (513) may be present in a tile group header (514) or a similar data structure. The ARC reference information (513) may refer to a subset of the ARC information (515) available in a parameter set (516) that extends beyond a single picture. For example, the parameter set (516) may be a sequence parameter set (SPS) or a decoder parameter set (DPS).

[0137] An additional level of indirect implicit activation of a PPS, SPS or a PPS from a tile group header as used in JVET-M0135-v1 may be unnecessary, since picture parameter sets like sequence parameter sets may be used for capability negotiation or announcement. However, if the ARC information should apply, for example, to sub-pictures that are also represented by tile groups, parameter sets with an activation scope limited to tile groups (e.g., adaptation parameter sets or header parameter sets) may be a better choice. Furthermore, if the size of the ARC information is larger than a negligible size, e.g. containing filter control information such as many filter coefficients, then parameters may be a better choice than using headers directly from a coding efficiency point of view, since these settings can be reused by future pictures or sub-pictures by referencing the same parameter set.

[0138] When using a sequence parameter set or another higher parameter set with a range spanning multiple pictures, certain considerations may apply:

[0139] The parameter set (516) storing the ARC information (515) in a table may in some cases be a sequence parameter set, but may advantageously be a decoder parameter set in other cases. A decoder parameter set may have an activation range of multiple CVSs, i.e., coded video streams, i.e., all coded video bits from the start of a session until the session is torn down. Such a range may be more appropriate because a feasible ARC factor may be a decoder feature that may be implemented in hardware, and hardware features tend not to change with any CVS (which in at least some entertainment systems is a group of pictures of one second or less in length). However, some embodiments may include the ARC information table in the sequence parameter set described herein, particularly in conjunction with point (2) below.

[0140] (2) Advantageously, the ARC reference information (513) may be placed directly in a header (514) (e.g., a picture / slice tile / GOB / tile group header; hereinafter a tile group header) rather than in a picture parameter set as in JVCET-M0135-v1 for the following reasons: When an encoder wants to change a single value in a picture parameter set, such as the ARC reference information, then the encoder may have to create a new PPS and reference the new PPS. In the case where only the ARC reference information changes while other information, such as the quantization matrix information in the PPS remains unchanged, such information may be of considerable size and will need to be retransmitted to make the new PPS complete. Since the ARC reference information may be a single codeword, such as an index into an ARC information table, which will be the only value that changes, retransmitting, for example, all of the quantization matrix information would be cumbersome and wasteful. Therefore, from a coding efficiency point of view, placing the ARC reference information directly in a header (e.g., header (514)) may be much better because the indirection through the PPS as proposed in JVET-M0135-v1 may be avoided. In addition, placing the ARC reference information in the PPS has an additional disadvantage: because the scope of activation of the picture parameter set is the picture, the ARC information referred to by the ARC reference information must be applied to the entire picture rather than a sub-picture.

[0141] In the same or another embodiment, the signaling of ARC parameters may follow the following: FIG. 7A to FIG. 7B A detailed example is outlined in . FIG. 7A to FIG. 7B A syntax diagram is depicted. The notation of such a syntax diagram roughly follows C-style programming. Lines in bold indicate syntax elements that are present in the bitstream, while lines that are not in bold typically indicate control flow or the setting of variables.

[0142] As an exemplary syntax structure for a header applicable to a (possibly rectangular) portion of a picture, the tile group header (600) may conditionally contain a variable-length Exp-Golomb coded syntax element dec_pic_size_idx (602) (depicted in bold). The presence of this syntax element in the tile group header (600) may be gated by the use of adaptive resolution (603). Here, the value of the adaptive resolution flag is not depicted in bold, meaning that the flag is present in the bitstream at the point where it appears in the syntax diagram. Whether adaptive resolution is used for the picture or portion thereof may be signaled in any higher-level syntax structure, either internal or external to the bitstream. In FIG. 7A to FIG. 7B In the example shown, adaptive resolution is signaled in a sequence parameter set (610), as described below.

[0143] Figure 7BAn excerpt of a sequence parameter set (610) is shown. The first syntax element shown is adaptive_pic_resolution_Change_flag (611). When true, such a flag may indicate the use of adaptive resolution, which in turn may require certain control information. In an example, such control information is conditionally presented based on the value of the flag, which is based on an if() statement (612) in the sequence parameter set (610) and the tile group header (600).

[0144] When adaptive resolution is used, in this example, what is encoded is the output resolution (613) in units of samples. In this example implementation, the output resolution (613) refers to both the syntax elements output_pic_width_in_luma_samples and output_pic_high_in_luma_samples, which together can define the resolution of the output picture. Elsewhere in the video coding technology or standard, certain restrictions on any value may be defined. For example, a level definition may limit the number of total output samples, which may be the product of the values ​​of the two syntax elements mentioned above. In addition, certain video coding technologies or standards, or external technologies or standards such as system standards may limit the number range (e.g., one or both dimensions must be divisible by a power of 2) or aspect ratio (e.g., width and height must be in a relationship such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementation or for other reasons.

[0145] In some applications, it is desirable for the encoder to instruct the decoder to use a specific reference picture size rather than implicitly assuming that size as the output picture size. In this example, the syntax element reference_pic_size_present_flag (614) gates the presence of the reference picture size (615) (again, in the example implementation, the numbers refer to both width and height).

[0146] Figure 7B A table of possible decoded picture widths and heights is also shown. For example, such a table may be represented by a table indication (616) (e.g., a syntax element num_dec_pic_size_in_luma_samples_minus1). The "minus1" of a syntax element may refer to the interpretation of the value of that syntax element. For example, if the encoded value of the syntax element is 0, then there is one table entry. If the encoded value is 5, then there are 6 table entries. For each "row" in the table, the decoded picture width and height is then included in the syntax as a table entry (617).

[0147] The syntax element dec_pic_size_idx (602) in the tile group header (600) may be used to index the presented table entry (617), allowing different decode sizes - in effect, scaling factors - per tile group.

[0148] Certain video coding techniques or standards (e.g., VP9) support spatial scalability by implementing some form of reference picture resampling (which may be signaled entirely differently than embodiments of the present disclosure) in conjunction with temporal scalability to achieve spatial scalability. In particular, certain reference pictures may be upsampled to a higher resolution using ARC-type techniques to form the basis of a spatial enhancement layer. Such upsampled pictures may be refined using a normal prediction mechanism at a high resolution to add detail.

[0149] Embodiments of the present disclosure may be used in such an environment. In some cases, in the same embodiment or another embodiment, a value in a network abstraction layer (NAL) unit header, such as a time ID field, may be used to indicate not only a time layer, but also a spatial layer. Doing so has certain advantages for certain system designs; for example, an existing Selected Forwarding Unit (SFU) created and optimized for time layer selective forwarding based on a NAL unit header time ID value may be used without modification for a scalable environment. To enable this, embodiments of the present disclosure may include a mapping between a coded picture size and a time layer, which will be indicated by the time ID field in the NAL unit header.

[0150] In some video coding techniques, an access unit (AU) may refer to a coded picture, slice, tile, NAL unit, etc., captured at a given time instance and synthesized into a corresponding picture / slice / tile / NAL unit bitstream. Such a time instance may be a synthesis time.

[0151] In High Efficiency Video Coding (HEVC) and certain other video coding techniques, a picture order count (POC) value can be used to indicate a selected reference picture among multiple reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) includes one or more pictures, slices, or tiles, each picture, slice, or tile belonging to the same AU can carry the same POC value, from which it can be derived that they are created based on content at the same synthesis time. In other words, when two pictures / slices / units carry the same given POC value, it can be determined that the two pictures / slices / units belong to the same AU and have the same synthesis time. Conversely, two pictures / tiles / slices with different POC values ​​can indicate those pictures / slices / tiles that belong to different AUs and have different synthesis times.

[0152] In embodiments of the present disclosure, the aforementioned strict relationship may be relaxed because an access unit may include pictures, slices, or tiles with different POC values. By allowing different POC values ​​within an AU, the POC value may be used to identify potentially independently decodable pictures / slices / tiles with the same presentation time. Thus, embodiments of the present disclosure are able to support multiple scalable layers without changing reference picture selection signaling (e.g., reference picture set signaling or reference picture list signaling), as described in more detail below.

[0153] In an embodiment, it is still desirable to be able to identify the AU to which a picture / slice / tile belongs from the POC value alone relative to other pictures / slices / tiles with different POC values. This can be achieved in the embodiments described below.

[0154] In the same or other embodiments, an access unit count (AUC) may be signaled in a high-level syntax structure such as a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter. The value of the AUC may be used to identify which NAL units, pictures, slices, or tiles belong to a given AU. The value of the AUC may correspond to different synthesis time situations. The AUC value may be equal to a multiple of the POC value. The AUC value may be calculated by dividing the POC value by an integer value. In some cases, the division operation may impose a certain burden on the decoder implementation. In such a case, a small restriction in the numbering space of the AUC value may allow the shift operation performed by the embodiments of the present disclosure to replace the division operation. For example, the AUC value may be equal to the most significant bit (MSB) value of the POC value range.

[0155] In the same embodiment, the value of the POC cycle per AU (e.g., the syntax element poc_cycle_au) may be signaled in a high-level syntax structure (e.g., a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter). The poc_cycle_au syntax element may indicate how many different and consecutive POC values ​​may be associated with the same AU. For example, if the value of poc_cycle_au is equal to 4, pictures, slices, or tiles with POC values ​​equal to 0 to 3 (including 0 and 3) are associated with AUs with AUC values ​​equal to 0, and pictures, slices, or tiles with POC values ​​equal to 4 to 7 (including 4 and 7) are associated with AUs with AUC values ​​equal to 1. Therefore, embodiments of the present disclosure may infer the value of AUC by dividing the POC value by the value of poc_cycle_au.

[0156] In the same or another embodiment, the value of poc_cycle_au can be derived from information, such as that located in a video parameter set (VPS), identifying the number of spatial or SNR layers in the coded video sequence. Such a possible relationship is briefly described below. Although the derivation described above can save several bits in the VPS and thus improve coding efficiency, it is advantageous to explicitly encode poc_cycle_au in an appropriate high-level syntax structure that is hierarchically lower than the video parameter set so as to be able to minimize poc_cycle_au for a given small portion of the bitstream, such as a picture. Because the POC value (and / or the value of a syntax element that indirectly references the POC) can be encoded in a low-level syntax structure, this optimization can save more bits than the bits saved by the derivation process above.

[0157] In the same embodiment or another embodiment, Fig.9A An example of a syntax table for signaling a syntax element of vps_poc_cycle_au (632) in a VPS (630) or SPS indicating poc_cycle_au for all pictures / slices in a coded video sequence is shown, and Fig. 9BAn example of a syntax table for signaling a syntax element of slice_poc_cycle_au (642) indicating the poc_cycle_au of the current slice in the slice header (640) is shown. If the POC value per AU increases uniformly, vps_contant_poc_cycle_per_au (634) in the VPS (630) is set equal to 1, and vps_poc_cycle_au (632) is signaled in the VPS (630). In this case, slice_poc_cycle_au (642) is not explicitly signaled, and the AUC value of each AU is calculated by dividing the value of the POC by vps_poc_cycle_au (632). If the POC value per AU increases unevenly, vps_contant_poc_cycle_per_au (634) in the VPS (630) is set equal to 0. In this case, vps_Access_unit_cnt is not signaled, but slice_Access_unit_cnt is signaled in the slice header of each slice or picture. Each slice or picture may have a different slice_Access_unit_cnt value. The value of AUC for each AU is calculated by dividing the value of POC by slice_poc_cycle_au (642).

[0158] Fig.10 A block diagram for describing the relevant workflow of an implementation is shown. For example, a decoder (or encoder) parses the VPS / SPS and identifies whether the POC period of each AU is constant (652). Subsequently, the decoder (or encoder) makes a decision based on whether the POC period of each AU is constant within the encoded video sequence (654). That is, if the POC period of each AU is constant, the decoder (or encoder) calculates the value of the access unit count according to the sequence level poc_cycle_au value and the POC value (656). Alternatively, if the POC period of each AU is not constant, the decoder (or encoder) calculates the value of the access unit count according to the picture level poc_cycle_au value and the POC value (658). Then, in either case, the decoder (or encoder) can repeat the process by, for example, parsing the VPS / SPS and identifying whether the POC period of each AU is constant (662).

[0159] In the same or other embodiments, pictures, slices or tiles corresponding to AUs with the same AUC value may be associated with the same decoding or output time even though the POC values ​​of the pictures, slices or tiles may be different. Thus, without any internal parsing / decoding dependencies between pictures, slices or tiles in the same AU, all or a subset of pictures, slices or tiles associated with the same AU may be decoded in parallel and may be output at the same time.

[0160] In the same or other embodiments, pictures, slices or tiles corresponding to AUs with the same AUC value may be associated with the same composition / display time even though the POC values ​​of the pictures, slices or tiles may be different. When the composition time is included in the container format, pictures may be displayed at the same time even if they correspond to different AUs if they have the same composition time.

[0161] In the same or other embodiments, each picture, slice or tile may have the same temporal identifier (e.g., syntax element temporal_id) in the same AU. All or a subset of pictures, slices or tiles corresponding to a time instance may be associated with the same temporal sublayer. In the same or other embodiments, each picture, slice or tile may have the same or different spatial layer id (e.g., syntax element layer_id) in the same AU. All or a subset of pictures, slices or tiles corresponding to a time instance may be associated with the same or different spatial layers.

[0162] Figure 8 An example of a video sequence structure (680) with a combination of temporal_id, layer_id, POC, and AUC values ​​with adaptive resolution change is shown. In this example, a picture, slice, or tile in the first AU with AUC=0 may have temporal_id=0 and layer_id=0 or 1, respectively, while a picture, slice, or tile in the second AU with AUC=1 may have temporal_id=1 and layer_id=0 or 1, respectively. Regardless of the values ​​of temporal_id and layer_id, the value of POC increases by 1 for each picture. In this example, the value of poc_cycle_au may be equal to 2. In an embodiment, the value of poc_cycle_au may be set equal to the number of (spatial scalability) layers. In this example, the POC value increases by 2, while the AUC value increases by 1. As an example, Figure 8 In the first AU (AUC=0), an I-slice (681) with POC 0, TID 0, and LID 0 and a B-slice (682) with POC 1, TID 0, and LID 1 are shown. In the second AU (AUC=1), Figure 8 A B-slice (683) with POC 2, TID 1, and LID 0 and a B-slice (684) with POC 3, TID 1, and LID 1 are shown. In the third AU (AUC=2), Figure 8 A B-slice (685) with POC 4, TID 0, and LID 0 and a B-slice (686) with POC 5, TID 0, and LID 1 are shown.

[0163] In the above-mentioned embodiments, all or a subset of inter-picture or inter-layer prediction structures and reference picture indications may be supported by using an existing reference picture set (RPS) signaling or a reference picture list (RPL) signaling in HEVC. In the RPS or RPL, the selected reference picture is indicated by signaling the value of the POC between the current picture and the selected reference picture or the incremental value of the POC. In the embodiments of the present disclosure, the RPS and RPL may be used to indicate the inter-picture or inter-layer prediction structure without changing the signaling, but with the following limitations. If the value of the temporal_id of the reference picture is greater than the value of the temporal_id of the current picture, the current picture may not use the reference picture for motion compensation or other predictions. If the value of the layer_id of the reference picture is greater than the value of the layer_id of the current picture, the current picture may not use the reference picture for motion compensation or other predictions.

[0164] In the same embodiment and other embodiments, motion vector scaling based on POC differences for temporal motion vector prediction may be disabled across multiple pictures within an access unit. Thus, although each picture may have a different POC value within an access unit, the motion vector may not be scaled and used for temporal motion vector prediction within an access unit, because reference pictures with different POCs in the same AU may be considered to be reference pictures with the same time instant. Thus, in this embodiment, the motion vector scaling function may return 1 when the reference picture belongs to the AU associated with the current picture.

[0165] In the same embodiment and other embodiments, when the spatial resolution of the reference picture is different from the spatial resolution of the current picture, motion vector scaling based on POC difference for temporal motion vector prediction can be optionally disabled across multiple pictures. When motion vector scaling is allowed, the motion vector can be scaled based on the POC difference and spatial resolution between the current picture and the reference picture.

[0166] In the same or another embodiment, the motion vector may be scaled based on the AUC difference instead of the POC difference for temporal motion vector prediction, especially when poc_cycle_au has a non-uniform value (when vps_contant_poc_cycle_per_au==0). Otherwise (when vps_contant_poc_cycle_per_au==1), the motion vector scaling based on the AUC difference may be the same as the motion vector scaling based on the POC difference.

[0167] In the same embodiment or another embodiment, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU (having the same AUC value) as the current picture is not scaled based on the AUC difference and is used for motion vector prediction without scaling or with scaling based on the spatial resolution ratio between the current picture and the reference picture.

[0168] In the same embodiment and other embodiments, the AUC value is used to identify the boundary of the AU and is used for the hypothetical reference decoder (HRD) operation, which requires input and output timing with AU granularity. In most cases, the decoded picture with the highest layer in the AU can be output for display. The AUC value and the layer_id value can be used to identify the output picture.

[0169] In an embodiment, a picture may include one or more sub-pictures. Each sub-picture may cover a partial area or the entire area of ​​the picture. The area supported by a sub-picture may overlap with the area supported by another sub-picture, or may not overlap with the area supported by another sub-picture. The area composed of one or more sub-pictures may cover the entire area of ​​the picture, or may not cover the entire area of ​​the picture. If a picture includes a sub-picture, the area supported by the sub-picture may be the same as the area supported by the picture.

[0170] In the same embodiment, a sub-picture may be encoded by an encoding method similar to that used for an encoded picture. A sub-picture may be encoded independently, or may be encoded based on another sub-picture or an encoded picture. A sub-picture may or may not have any analytical dependencies with another sub-picture or an encoded picture.

[0171] In the same embodiment, the coded sub-picture may be included in one or more layers. The coded sub-pictures in the layers may have different spatial resolutions. The original sub-picture may be spatially resampled (upsampled or downsampled), encoded with different spatial resolution parameters, and included in the bitstream corresponding to the layer.

[0172] In the same or another embodiment, a sub-picture with (W, H) may be encoded and included in a coded bitstream corresponding to layer 0, where W indicates the width of the sub-picture and H indicates the height of the sub-picture, respectively, while being obtained from a sub-picture having an original spatial resolution (W*S w,k ,H*S h,k ) can be encoded and included in the coded bitstream corresponding to layer k, where S w,k , S h,k Indicates the resampling ratio horizontally and vertically. If S w,k , S h,k If the value of is greater than 1, resampling is equal to upsampling. w,k ,、S h,k If the value of is less than 1, resampling is equal to downsampling.

[0173] In the same or another embodiment, a coded sub-picture in a layer may have a different visual quality than a coded sub-picture in another layer in the same sub-picture or in a different sub-picture. i,n Encode sub-picture i in layer n, and use quantization parameter Q j,m Encode sub-picture j in layer m.

[0174] In the same or another embodiment, a coded sub-picture in a layer may be independently decodable without any parsing or decoding dependency on a coded sub-picture in another layer of the same local region. A sub-picture layer that is independently decodable without reference to another sub-picture layer of the same local region is an independent sub-picture layer. A coded sub-picture in an independent sub-picture layer may or may not have a decoding or parsing dependency on a previously coded sub-picture in the same sub-picture layer, but the coded sub-picture may not have any dependency on a coded picture in another sub-picture layer.

[0175] In the same or another embodiment, a coded sub-picture in a layer may be dependently decodable with any parsing or decoding dependency on a coded sub-picture in another layer of the same local region. A sub-picture layer that is dependently decodable with reference to another sub-picture layer of the same local region is a dependent sub-picture layer. A coded sub-picture in a dependent sub-picture may reference a coded sub-picture belonging to the same sub-picture, a previously coded sub-picture in the same sub-picture layer, or both reference sub-pictures.

[0176] In the same or another embodiment, a coded sub-picture includes one or more independent sub-picture layers and one or more dependent sub-picture layers. However, for a coded sub-picture, at least one independent sub-picture layer may exist. An independent sub-picture layer may have a value of a layer identifier (e.g., a syntax element layer_id) equal to 0, which may be present in a NAL unit header or another high-level syntax structure. A sub-picture layer with layer_id equal to 0 may be a base sub-picture layer.

[0177] In the same or another embodiment, a picture may include one or more foreground sub-pictures and one background sub-picture. The area supported by the background sub-picture may be equal to the area of ​​the picture. The area supported by the foreground sub-picture may overlap with the area supported by the background sub-picture. The background sub-picture may be a base sub-picture layer and the foreground sub-picture may be a non-base (enhanced) sub-picture layer. One or more non-base sub-picture layers may reference the same base layer for decoding. Each non-base sub-picture layer with layer_id equal to a may reference a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0178] In the same or another embodiment, a picture may include one or more foreground sub-pictures with or without background sub-pictures. Each sub-picture may have its own base sub-picture layer and one or more non-base (enhancement) layers. Each base sub-picture layer may be referenced by one or more non-base sub-picture layers. Each non-base sub-picture layer with layer_id equal to a may reference a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0179] In the same or another embodiment, a picture may include one or more foreground sub-pictures with or without a background sub-picture. Each coded sub-picture in a (base or non-base) sub-picture layer may be referenced by one or more non-base layer sub-pictures belonging to the same sub-picture and one or more non-base layer sub-pictures not belonging to the same sub-picture.

[0180] In the same or another embodiment, a picture may include one or more foreground sub-pictures with or without background sub-pictures. A sub-picture in layer a may be further partitioned into multiple sub-pictures in the same layer. One or more coded sub-pictures in layer b may reference the partitioned sub-pictures in layer a.

[0181] In the same or another embodiment, a coded video sequence (CVS) may be a group of coded pictures. A CVS may include one or more coded sub-picture sequences (CSPS), where a CSPS may be a group of coded sub-pictures covering the same local area of ​​a picture. A CSPS may have a temporal resolution that is the same as or different from the temporal resolution of the coded video sequence.

[0182] In the same embodiment or another embodiment, the CSPS may be encoded and included in one or more layers. The CSPS may include one or more CSPS layers or be composed of one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS may reconstruct a sequence of sub-pictures corresponding to the same local area.

[0183] In the same embodiment or another embodiment, the number of CSPS layers corresponding to a CSPS may be the same as or different from the number of CSPS layers corresponding to another CSPS.

[0184] In the same or another embodiment, a CSPS layer may have a different temporal resolution (e.g., frame rate) than another CSPS layer. The original (uncompressed) sub-picture sequence may be temporally resampled (upsampled or downsampled), encoded using different temporal resolution parameters, and included in the bitstream corresponding to the layer.

[0185] In the same or another embodiment, a sub-picture sequence having a frame rate F may be encoded and included in a coded bitstream corresponding to layer 0, while a sub-picture sequence having a frame rate F*S t,k The sub-picture sequence upsampled (or downsampled) in time from the original sub-picture sequence of S can be encoded and included in the coded bitstream corresponding to layer k, where S t,k indicates the temporal sampling rate of layer k. If S t,k If the value of S is greater than 1, the temporal resampling process is equivalent to frame rate upconversion. t,k If the value of is less than 1, the temporal resampling process is equal to frame rate down-conversion.

[0186] In the same embodiment or another embodiment, when a sub-picture with CSPS layer a is referenced by a sub-picture with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b, the decoded pixels in CSPS layer a are resampled and used for reference. The resampling process may require upsampling filtering or downsampling filtering.

[0187] Fig.11An example video stream is shown that includes a background video CSPS with layer_id equal to 0 and multiple foreground CSPS layers. Although the encoded sub-picture may include one or more enhanced CSPS layers (704), the background region that does not belong to any foreground CSPS layer may include a base layer (702). The base layer (702) may contain both the background region and the foreground region, while the enhanced CSPS layer (704) contains the foreground region. In the same region, the enhanced CSPS layer (704) may have better visual quality than the base layer (702). The enhanced CSPS layer (704) may refer to the motion vectors and reconstructed pixels of the base layer (702) corresponding to the same region.

[0188] In the same or another embodiment, in a video file, the video bitstream corresponding to the base layer (702) is contained in a track, while the CSPS layer (704) corresponding to each sub-picture is contained in a separate track.

[0189] In the same or another embodiment, the video bitstream corresponding to the base layer (702) is contained in a track, while the CSPS layer (704) with the same layer_id is contained in a separate track. In this example, the track corresponding to layer k includes only the CSPS layer (704) corresponding to layer k.

[0190] In the same or another embodiment, each CSPS layer (704) of each sub-picture is stored in a separate track. Each track may or may not have any parsing or decoding dependencies on one or more other tracks.

[0191] In the same or another embodiment, each track may contain the bitstream corresponding to layers i to j of the CSPS layer (704) of all or a subset of the sub-pictures, where 0 < i <= j <= k, and k is the highest layer of the CSPS.

[0192] In the same or another embodiment, a picture includes one or more associated media data or consists of one or more associated media data. The one or more associated media data include depth maps, alpha maps, 3D geometric data, occupancy maps, etc. Such associated timing media data may be divided into one or more data sub-streams, and each of the data sub-streams corresponds to a sub-picture.

[0193] In the same or another embodiment, Fig.12An example of a video conference based on a multi-layer sub-picture method is shown. In the video stream, a base layer video bitstream corresponding to a background picture and one or more enhancement layer video bitstreams corresponding to a foreground sub-picture are included. Each enhancement layer video bitstream may correspond to a CSPS layer. In the display, the picture corresponding to the base layer (712) is displayed by default. The base layer (712) may include a picture in a picture (PIP) of one or more users. When a client control selects a specific user, an enhanced CSPS layer (714) corresponding to the selected user is decoded and displayed with enhanced quality or spatial resolution.

[0194] Fig.13 A diagram illustrating the operation of an embodiment. In an embodiment, a decoder may decode a video bitstream comprising multiple layers (e.g., a base layer and one or more enhanced CSPS layers) (722). Subsequently, the decoder may identify a background area and one or more foreground sub-pictures (724) and make a decision as to whether a particular sub-picture area is selected (726). If a particular sub-picture area corresponding to, for example, a user's PIP is selected (yes), the decoder may decode and display the enhanced sub-picture corresponding to the selected user (728). For example, the decoder may decode and display a picture corresponding to the enhanced CSPS layer (714). If a particular sub-picture area is not selected (no), the decoder may decode and display a background area (730). For example, the decoder may decode and display a picture corresponding to the base layer (712).

[0195] In the same or another embodiment, a network middlebox (e.g., a router) may select a subset of layers to send to a user based on its bandwidth. Picture / sub-picture organization may be used for bandwidth adaptation. For example, if a user does not have bandwidth, the router may strip layers or select some sub-pictures due to importance or based on the settings used. In an embodiment, such processing may be done dynamically to adapt bandwidth.

[0196] Fig.14An example use case for 360 video is shown. When a spherical 360 picture (742) is projected onto a flat picture, the projected spherical 360 picture (742) can be segmented into multiple sub-pictures (745) as a base layer (744). An enhancement layer (746) of a specific one of the sub-pictures (745) can be encoded and sent to the client. The decoder can decode both the base layer (744) including all sub-pictures (745) and the enhancement layer (746) of the selected one of the sub-pictures (745). When the current viewport is the same as the selected one of the sub-pictures (745), the displayed picture can have a higher quality, where the decoded sub-picture (745) has the enhancement layer (746). Otherwise, the decoded picture with the base layer (744) is displayed at a lower quality.

[0197] In the same or another embodiment, any layout information for display may be present in the file as supplemental information (e.g., SEI messages or metadata). Depending on the signaled layout information, one or more decoded sub-pictures may be repositioned and displayed. The layout information may be signaled by the streaming server or broadcaster, or may be regenerated by a network entity or cloud server, or may be determined by a user's customized settings.

[0198] In an embodiment, when the input picture is divided into one or more (rectangular) sub-regions, each sub-region can be encoded as an independent layer. Each independent layer corresponding to a local region can have a unique layer_id value. For each independent layer, the sub-picture size and position information can be signaled. For example, the offset information (x_offset, y_offset) of the upper left corner and the picture size (width, height) can be signaled. Fig.15A shows an example of the layout of the divided sub-pictures (752), Fig. 15B An example of corresponding sub-picture size and position information for one of the sub-pictures (752) is shown, and Fig.16 The corresponding picture prediction structure is shown.Layout information including sub-picture size and sub-picture position may be signaled in a high-level syntax structure such as a parameter set, slice or tile group header or SEI message.

[0199] In the same embodiment, each sub-picture corresponding to an independent layer may have its unique POC value within the AU. When indicating a reference picture among pictures stored in the DPB by using a syntax element in an RPS or RPL structure, the POC value of each sub-picture corresponding to a layer may be used.

[0200] In the same or another embodiment, to indicate the (inter-layer) prediction structure, instead of using layer_id, the POC (delta) value may be used.

[0201] In the same embodiment, a sub-picture in which the POC value corresponding to a layer (or local area) is equal to N may or may not be used as a reference picture for motion compensated prediction of a sub-picture in which the POC value corresponding to the same layer (or the same local area) is equal to K + N. In most cases, the value of the number K may be equal to the maximum number of (independent) layers, which may be equal to the number of sub-regions.

[0202] In the same embodiment or another embodiment, Figure 17 to Figure 18 Show FIG. 15A to FIG. 15B and Fig.16 When the input picture is divided into multiple (e.g., four) sub-regions, each local region may be encoded using one or more layers. In this case, the number of independent layers may be equal to the number of sub-regions, and one or more layers may correspond to the sub-regions. Therefore, each sub-region may be encoded using one or more independent layers and zero or more dependent layers.

[0203] In the same embodiment, reference Fig.17 , the input picture may be divided into four sub-regions, including an upper left sub-region (762), an upper right sub-region (763), a lower left sub-region (764), and a lower right sub-region (765). The upper right sub-region (763) may be encoded into two layers, layer 1 and layer 4, and the lower right sub-region (765) may be encoded into two layers, layer 3 and layer 5. In this case, layer 4 may refer to layer 1 for motion compensation prediction, and layer 5 may refer to layer 3 for motion compensation.

[0204] In the same or another embodiment, in-loop filtering (e.g., deblocking filtering, adaptive in-loop filtering, shaper, bilateral filtering, or any deep learning based filtering) across layer boundaries may (optionally) be disabled.

[0205] In the same or another embodiment, motion compensated prediction or intra-block copy across layer boundaries may (optionally) be disabled.

[0206] In the same or another embodiment, boundary padding for motion compensated prediction or in-loop filtering at the boundaries of sub-pictures may be optionally processed. A flag may be signaled in a high-level syntax structure (e.g., a parameter set (VPS, SPS, PPS, or APS), a slice or slice group header, or an SEI message) indicating whether boundary padding is processed.

[0207] In the same or another embodiment, the layout information of the sub-region (or sub-picture) may be signaled in the VPS or SPS. Fig.19Ashows an example of syntax elements in a VPS (770), and Fig.19B An example of syntax elements of an SPS (780) is shown. In this example, vps_sub_picture_dividing_flag (772) is signaled in the VPS (770). This flag may indicate whether the input picture is divided into multiple sub-regions. When the value of vps_sub_picture_diving_flag (772) is equal to 0, the input picture in the coded video sequence corresponding to the current VPS may not be divided into multiple sub-regions. In this case, the input picture size may be equal to the coded picture size (pic_width_in_luma_samples (786), pic_height_in_luma_samples (788)), which is signaled in the SPS (780). When the value of vps_sub_picture_diving_flag (772) is equal to 1, the input picture may be divided into multiple sub-regions. In this case, the syntax elements vps_full_pic_width_in_luma_samples (774) and vps_full_pic_height_in_luma_samples (776) are signaled in the VPS (770). The values ​​of vps_full_pic_width_in_luma_samples (774) and vps_full_pic_height_in_luma_samples (776) may be equal to the width and height of the input picture, respectively.

[0208] In the same embodiment, the values ​​of vps_full_pic_width_in_luma_samples (774) and vps_full_pic_height_in_luma_samples (776) may not be used for decoding, but may be used for synthesis and display.

[0209] In the same embodiment, when the value of vps_sub_picture_dividing_flag (772) is equal to 1, the syntax element pic_offset_x may be signaled in the SPS (780) corresponding to the specific layer. (782) and pic_offset_y(784). In this case, the coded picture size (pic_width_in_luma_samples(786), pic_height_in_luma_samples(788)) signaled in SPS(780) may be equal to the width and height of the sub-region corresponding to the specific layer. In addition, the position of the upper left corner of the sub-region (pic_offset_x(782), pic_offset_y(784)) may be signaled in SPS(780).

[0210] In the same embodiment, the position information of the upper left corner of the sub-region (pic_offset_x (782), pic_offset_y (784)) may not be used for decoding, but may be used for synthesis and display.

[0211] In the same or another embodiment, layout information (size and position) of all or a subset of sub-regions of an input picture and dependency information between layers may be signaled in a parameter set or SEI message. Fig. 20 An example of a syntax element indicating information about the layout of sub-regions, dependencies between layers, and relationships between sub-regions and one or more layers is shown. In this example, the syntax element num_sub_region(791) indicates the number of (rectangular) sub-regions in the current coded video sequence. The syntax element num_layers(792) indicates the number of layers in the current coded video sequence. The value of num_layers(792) may be equal to or greater than the value of num_sub_region(791). When any sub-region is coded as a single layer, the value of num_layers(792) may be equal to the value of num_sub_region(791). When one or more sub-regions are coded as multiple layers, the value of num_layers(792) may be greater than the value of num_sub_region(791). The syntax element direct_dependency_flag[i][j](793) indicates a dependency from the jth layer to the ith layer. The syntax element num_layers_for_region[i](794) indicates the number of layers associated with the i-th sub-region. The syntax element sub_region_layer_id[i][j](795) indicates the layer_id of the j-th layer associated with the i-th sub-region. The syntax elements sub_region_offset_x[i](796) and sub_region_offset_y[i](797) indicate the layer_id of the j-th layer associated with the i-th sub-region. (797) indicate the horizontal and vertical positions of the upper left corner of the ith sub-region, respectively. The syntax elements sub_region_width[i] (798) and sub_region_height[i] (799) indicate the width and height of the ith sub-region, respectively.

[0212] In one embodiment, one or more syntax elements specifying an output layer set may be signaled in a high-level syntax structure (e.g., a VPS, DPS, SPS, PPS, APS, or SEI message) to indicate one or more layers to be output with or without profile hierarchy level information. Fig.21 , a syntax element num_output_layer_sets(804) may be signaled in the VPS, indicating the number of output layer sets (OLS) in the coded video sequence that references the VPS. For each output layer set, a syntax element output_layer_flag(810) may be signaled as many times as the output layer is signaled.

[0213] In the same embodiment, the syntax element output_layer_flag(810) equal to 1 specifies that the i-th layer is output. The syntax element output_layer_flag(810) equal to 0 specifies that the i-th layer is not output.

[0214] In the same or another embodiment, one or more syntax elements may be signaled in a high-level syntax structure (e.g., VPS, DPS, SPS, PPS, APS, or SEI message) that specify profile level information for each output layer set. Fig.21 , a syntax element num_profile_tier_level (806) may be signaled in the VPS, indicating the number of profile tier level information per OLS in the coded video sequence referring to the VPS. For each output layer set, the set of syntax elements for profile tier level information or the index of a specific profile tier level information among the entries indicating the profile tier level information may be signaled as many times as the output layers are signaled.

[0215] In the same embodiment, the syntax element profile_tier_level_idx[i][j] (812) specifies the index of a profile_tier_level() (808) syntax structure, which applies to the jth layer of the i-th OLS, into the list of profile_tier_level() (808) syntax structures in the VPS.

[0216] Profiles, tiers and levels (and their corresponding information) can specify restrictions on bitstreams, and therefore specify restrictions on capabilities required for decoding bitstreams. Profiles, tiers and levels (and their corresponding information) can also be used to indicate interoperability points between various decoder implementations. Profiles can be, for example, a subset of the entire bitstream syntax of a standard. Each profile (and its corresponding information) can specify a subset of algorithmic features and restrictions supported by all decoders that conform to the profile. Tiers and levels can be specified within each profile, and the levels of the tiers can be a specified set of constraints imposed on the values ​​of syntax elements in the bitstream. Each level of the tier (and its corresponding information) can specify a set of restrictions on values ​​and / or arithmetic combinations of values ​​that can be adopted by syntax elements of the present disclosure. The same set of tier and level definitions can be used with all profiles, but separate implementations can support different tiers, and use different levels within the tiers for each supported profile. For any given profile, the levels of the tiers can correspond to specific decoder processing loads and memory capabilities. The levels specified for lower tiers can be more constrained than the levels specified for higher tiers.

[0217] In the same embodiment or another embodiment, reference is made to Fig. 22 , when the maximum number of layers is greater than 1 (vps_max_layers_minus1>0), the syntax elements num_profile_tier_level (806) and / or num_output_layer_sets (804) may be signaled.

[0218] In the same embodiment or another embodiment, reference is made to Fig. 22 , the syntax element vps_output_layers_mode[i] (822) - which indicates the mode of output layer signaling for the i-th output layer set - may be present in the VPS.

[0219] In the same embodiment, the syntax element vps_output_layers_mode[i] (822) equal to 0 specifies that only the highest layer is output with the i-th output layer set. (822) equal to 1 specifies that all layers are output with the i-th output layer set. The syntax element vps_output_layers_mode[i] (822) equal to 2 specifies that the layers output with the i-th output layer set are the layers for which vps_output_layer_flag[i][j] is equal to 1. More values ​​may be reserved.

[0220] In the same embodiment, the syntax element output_layer_flag[i][j] (810) may or may not be signaled depending on the value of the syntax element vps_output_layers_mode[i] (822) for the i-th output layer set.

[0221] In the same embodiment or another embodiment, reference is made to Fig. 22 , for the i-th output layer set, the flag vps_ptl_signal_flag[i](824) may be present. Depends on vps_ptl_signal_flag[i] The value of (824) may or may not signal the profile layer level information for the i-th output layer set.

[0222] In the same embodiment or another embodiment, reference is made to Fig.23 , the number of sub-pictures max_subpics_minus1 in the current CVS can be signaled in a high-level syntax structure (e.g., VPS, DPS, SPS, PPS, APS or SEI message).

[0223] In the same embodiment, reference Fig.23 , when the number of sub-pictures is greater than 1 (max_subpics_minus1>0), the sub-picture identifier sub_pic_id[i] for the i-th sub-picture may be signaled (821).

[0224] In the same or another embodiment, one or more syntax elements may be signaled in the VPS indicating the sub-picture identifiers of each layer belonging to each output layer set. Fig.23 , the identifier sub_pic_id_layer[i][j][k](826) indicates the k-th sub-picture present in the j-th layer of the i-th output layer set. By using the information of the identifier sub_pic_id_layer[i][j][k](826), the decoder can identify which sub-picture can be decoded and output for each layer of a specific output layer set.

[0225] In an embodiment, a picture header (PH) is a syntax structure containing syntax elements that apply to all slices of a coded picture. A picture unit (PU) is a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain exactly one coded picture. A PU may contain a picture header (PH) and one or more video coding layer (VCL) NAL units that make up a coded picture.

[0226] In an embodiment, by including the SPS (RBSP) in at least one AU whose TemporalId is equal to 0 or providing it through an external device, the SPS (RBSP) may be available to the decoding process before being referenced.

[0227] In an embodiment, the SPS (RBSP) may be available to the decoding process before being referenced by including the SPS (RBSP) in at least one AU with TemporalId equal to 0 in the CVS or provided by an external device, and the CVS contains one or more PPSs that reference the SPS.

[0228] In an embodiment, the SPS (RBSP) may be available to the decoding process before being referenced by one or more PPSs, the CVS containing one or more PPSs referencing the SPS, by including the SPS (RBSP) in at least one PU having a nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL unit in the CVS or provided by an external device.

[0229] In an embodiment, the SPS (RBSP) may be available to the decoding process before being referenced by one or more PPSs by including the SPS (RBSP) in at least one PU with TemporalId equal to 0 and nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL unit referencing the SPS NAL unit, or provided by an external device.

[0230] In an embodiment, the SPS (RBSP) may be available to the decoding process before being referenced by one or more PPSs, the CVS containing one or more PPSs referencing the SPS, by including the SPS (RBSP) in at least one PU with TemporalId equal to 0 and nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL unit that references the SPS NAL unit in the CVS, or by providing it through an external device.

[0231] In the same or another embodiment, the identifier pps_seq_parameter_set_id specifies the value of the identifier sps_seq_parameter_set_id for the referenced SPS. The value of the identifier pps_seq_parameter_set_id may be the same in all PPSs referenced by a coded picture in a coded layer video sequence (CLVS).

[0232] In the same or another embodiment, all SPS NAL units in a CVS having a particular value of the identifier sps_seq_parameter_set_id may have the same content.

[0233] In the same or another embodiment, SPS NAL units may share the same value space of the identifier sps_seq_parameter_set_id regardless of the nuh_layer_id value.

[0234] In the same or another embodiment, the nuh_layer_id value of the SPS NAL unit may be equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL unit.

[0235] In an embodiment, when an SPS with nuh_layer_id equal to m is referenced by one or more PPSs with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0236] In an embodiment, the PPS (RBSP) may be available to a decoding process before being referenced by including the PPS (RBSP) in at least one AU having a TemporalId equal to the TemporalId of the PPS NAL unit or provided by an external device.

[0237] In an embodiment, the PPS (RBSP) is provided by including the PPS in at least one AU in the CVS whose TemporalId is equal to the TemporalId of the PPS NAL unit or by an external device. A (RBSP) may be available to the decoding process before being referenced, and a CVS contains one or more PHs (or coded slice NAL units) that reference the PPS.

[0238] In an embodiment, the PPS (RBSP) may be available to the decoding process before being referenced by including the PPS (RBSP) in at least one AU having a nuh_layer_id equal to the lowest nuh_layer_id value of the coded slice NAL unit of the referenced PPS NAL unit in the CVS or provided by an external device, and the CVS contains one or more PHs (or coded slice NAL units) that reference the PPS.

[0239] In an embodiment, the PPS (RBSP) may be available to the decoding process before being referenced by including the PPS (RBSP) in at least one PU whose TemporalId is equal to the TemporalId of the PPS NAL unit in the CVS and whose nuh_layer_id is equal to the lowest nuh_layer_id value of the coded slice NAL unit that references the PPS NAL unit, or by providing it through an external device, and the CVS contains one or more PHs (or coded slice NAL units) that reference the PPS.

[0240] In the same or another embodiment, the identifier ph_pic_parameter_set_id in the PH specifies the value of the identifier pps_pic_parameter_set_id for the reference PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0241] In the same or another embodiment, all PPS NAL units within a PU with a specific value of the identifier pps_pic_parameter_set_id may have the same content.

[0242] In the same or another embodiment, PPS NAL units may share the same value space of the identifier pps_pic_parameter_set_id regardless of the nuh_layer_id value.

[0243] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the lowest nuh_layer_id value of the coded slice NAL units of the NAL units that reference the PPS NAL unit.

[0244] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0245] In an embodiment, the PPS (RBSP) may be available to a decoding process before being referenced by including the PPS (RBSP) in at least one AU having a TemporalId equal to the TemporalId of the PPS NAL unit or provided by an external device.

[0246] In an embodiment, the PPS (RBSP) may be available to the decoding process before being referenced by including the PPS (RBSP) in at least one AU whose TemporalId is equal to the TemporalId of the PPS NAL unit in the CVS or by providing it through an external device, and the CVS contains one or more PHs (or coded slice NAL units) that reference the PPS.

[0247] In an embodiment, the PPS (RBSP) may be available to the decoding process before being referenced by including the PPS (RBSP) in at least one PU having a nuh_layer_id equal to the lowest nuh_layer_id value of the coded slice NAL unit of the referenced PPS NAL unit in the CVS or provided by an external device, and the CVS contains one or more PHs (or coded slice NAL units) of the referenced PPS.

[0248] In an embodiment, the PPS (RBSP) may be available to the decoding process before being referenced by including the PPS (RBSP) in at least one PU whose TemporalId is equal to the TemporalId of the PPS NAL unit in the CVS and whose nuh_layer_id is equal to the lowest nuh_layer_id value of the coded slice NAL unit that references the PPS NAL unit, or by providing it through an external device, and the CVS contains one or more PHs (or coded slice NAL units) that reference the PPS.

[0249] In the same or another embodiment, the identifier ph_pic_parameter_set_id in the PH specifies the value of the identifier pps_pic_parameter_set_id for the reference PPS in use. The value of the identifier pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0250] In the same or another embodiment, all PPS NAL units within a PU with a specific value of the identifier pps_pic_parameter_set_id may have the same content.

[0251] In the same or another embodiment, PPS NAL units may share the same value space of the identifier pps_pic_parameter_set_id regardless of the nuh_layer_id value.

[0252] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the lowest nuh_layer_id value of the coded slice NAL units of the reference NAL units that reference the PPS NAL unit.

[0253] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0254] An output layer may be a layer of an output layer set of outputs. An output layer set (OLS) may be a set of designated layers, where one or more layers in the set of layers are designated as output layers. An output layer set (OLS) layer index is an index of a layer in the OLS into a list of layers in the OLS.

[0255] A sublayer may be a temporal scalable layer of a temporal scalable bitstream of a sublayer comprising VCL NAL units and associated non-VCL NAL units with a specific value of the TemporalId variable. A sublayer representation may be a subset of a bitstream comprising NAL units of a specific sublayer and lower sublayers.

[0256] The VPS RBSP may be available to the decoding process before being referenced by including it in at least one AU with TemporalId equal to 0 or by providing it through an external device. All VPS NAL units with a specific value of vps_video_parameter_set_id in CVS may have the same content.

[0257] Reference Figure 24 to Figure 25 , the syntax elements of an example VPS RBSP are described below.

[0258] The syntax element vps_video_parameter_set_id(842) provides an identifier of the VPS for reference by other syntax elements. The value of the syntax element vps_video_parameter_set_id(842) can be greater than zero.

[0259] The syntax element vps_max_layers_minus1 (802) plus 1 specifies the maximum allowed number of layers in each CVS of the reference VPS.

[0260] The syntax element vps_max_sublayers_minus1 (846) plus 1 specifies the maximum number of temporal sublayers that may be present in a layer in each CVS that references a VPS. The value of the syntax element vps_max_sublayers_minus1 (846) may be in the range of 0 to 6, inclusive.

[0261] The syntax element vps_all_layers_same_num_sublayers_flag(848) equal to 1 specifies that the number of temporal sublayers is the same for all layers in each CVS of the referenced VPS. The syntax element vps_all_layers_same_num_sublayers_flag(848) equal to 0 specifies that the layers in each CVS of the referenced VPS may or may not have the same number of temporal sublayers. When not present, the value of vps_all_layers_same_num_sublayers_flag(848) may be inferred to be equal to 1.

[0262] The syntax element vps_all_independent_layers_flag(850) equal to 1 specifies that all layers in the CVS are independently coded without using inter-layer prediction. The syntax element vps_all_independent_layers_flag(850) equal to 0 specifies that one or more layers in the CVS may use inter-layer prediction. When not present, the value of vps_all_independent_layers_flag(850) equal to 1 may be inferred.

[0263] The syntax element vps_layer_id[i](852) specifies the nuh_layer_id value of the i-th layer. For any two non-negative integer values ​​of m and n, when m is less than n, the value of vps_layer_id[m] may be less than vps_layer_id[n].

[0264] The syntax element vps_independent_layer_flag[i](854) equal to 1 specifies that the layer with index i does not use inter-layer prediction. The syntax element vps_independent_layer_flag[i](854) equal to 0 specifies that the layer with index i can use inter-layer prediction, and for j in the range of 0 to i-1 (including 0 and i-1), the syntax element vps_direct_ref_layer_flag[i][j] is present in the VPS. When not present, it can be inferred that the value of the syntax element vps_independent_layer_flag[i](854) is equal to 1.

[0265] The syntax element vps_direct_ref_layer_flag[i][j] (856) equal to 0 specifies that the layer with index j is not a direct reference layer of the layer with index i. (856) is equal to 1 to specify that the layer with index j is the direct reference layer of the layer with index i. When the syntax element vps_direct_ref_layer_flag[i][j] (856) is not present for i and j in the range of 0 to vps_max_layers_minus1 (including 0 and vps_max_layers_minus1), it can be inferred that the syntax element is equal to 0. When the syntax element vps_independent_layer_flag[i] (854) is equal to 0, there may be at least one value of j in the range of 0 to i-1 (including 0 and i-1) such that the value of the syntax element vps_direct_ref_layer_flag[i][j] (856) is equal to 1.

[0266] The variables NumDirectRefLayers[i], DirectRefLayerIdx[i][d], NumRefLayers[i], RefLayerIdx[i][r], and LayerUsedAsRefLayerFlag[j] can be derived as follows:

[0267] The variable GeneralLayerIdx[i], which specifies the layer index of the layer where nuh_layer_id is equal to vps_layer_id[i] (852), can be derived as follows: for(i=0;i<=vps_max_layers_minus1;i++) (38) GeneralLayerIdx[vps_layer_id[i]]=i

[0268] For any two different values ​​of i and j, both in the range of 0 to vps_max_sublayers_minus1(846) (including 0 and vps_max_layers_minus1), when dependencyFlag[i][j] is equal to 1, the following bitstream consistency may be required: the values ​​of chroma_format_idc and bit_depth_minus8 applied to the i-th layer are equal to the values ​​of chroma_format_idc and bit_depth_minus8 applied to the j-th layer, respectively.

[0269] The syntax element max_tid_ref_present_flag[i] (858) equal to 1 specifies that the syntax element max_tid_il_ref_pics_plus1[i] (860) is present. The syntax element max_tid_ref_present_flag[i] (858) equal to 0 specifies that the syntax element max_tid_il_ref_pics_plus1[i] (860) is not present.

[0270] The syntax element max_tid_il_ref_pics_plus1[i](860) equal to 0 specifies that non-IRAP pictures of the i-th layer do not use inter-layer prediction. The syntax element max_tid_il_ref_pics_plus1[i](860) greater than 0 specifies that pictures with TemporalId greater than max_tid_il_ref_pics_plus1[i]-1 are not used as inter-layer reference pictures (ILRP) for decoding pictures of the i-th layer. When not present, the value of the syntax element max_tid_il_ref_pics_plus1[i](860) may be inferred to be equal to 7.

[0271] The syntax element each_layer_is_an_ols_flag(862) equal to 1 specifies that each OLS contains only one layer, and that each layer in the CVS referencing the VPS is itself an OLS with a single included layer as the only output layer. The syntax element each_layer_is_an_ols_flag(862) equal to 0 specifies that an OLS may contain more than one layer. If the syntax element vps_max_layers_minus1 is equal to 0, then the value of the syntax element each_layer_is_an_ols_flag(862) equal to 1 may be inferred. Otherwise, when the syntax element vps_all_independent_layers_flag(854) is equal to 0, then the value of the syntax element each_layer_is_an_ols_flag(862) equal to 0 may be inferred.

[0272] The syntax element ols_mode_idc (864) equal to 0 specifies that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (including 0 and i), and for each OLS, only the highest layer in the OLS is output.

[0273] The syntax element ols_mode_idc (864) equal to 1 specifies that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (inclusive), and for each OLS, all layers in the OLS are output.

[0274] The syntax element ols_mode_idc(864) equal to 2 specifies that the total number of OLSs specified by the VPS is explicitly signaled, and for each OLS, the output layer is explicitly signaled and the other layers are layers that are direct reference layers or indirect reference layers of the output layer of the OLS.

[0275] The value of the syntax element ols_mode_idc(864) may be in the range of 0 to 2, inclusive. A value of 3 for the syntax element ols_mode_idc(864) may be reserved for future use by ITU-T|ISO / IEC.

[0276] When the syntax element vps_all_independent_layers_flag (850) is equal to 1 and each_layer_is_an_ols_flag (862) is equal to 0, the syntax element ols_mode_idc may be inferred. The value of (864) is equal to 2.

[0277] When the syntax element ols_mode_idc (864) is equal to 2, the syntax element num_output_layer_sets_minus1 (866) plus 1 specifies the total number of OLSs specified by the VPS.

[0278] The variable TotalNumOlss that specifies the total number of OLSs specified by VPS can be derived as follows:

[0279] When ols_mode_idc(864) is equal to 2, the syntax element ols_output_layer_flag[i][j] (868) equal to 1 specifies that the layer with nuh_layer_id equal to vps_layer_id[j] is the output layer of the i-th OLS. When the syntax element ols_mode_idc (864) is equal to 2, the syntax element ols_output_layer_flag[i][j] (868) equal to 0 specifies that the layer with nuh_layer_id equal to vps_layer_id[j] is not the output layer of the i-th OLS.

[0280] The variable NumOutputLayersInOls[i] specifying the number of output layers in the i-th OLS, the variable NumSubLayersInLayerInOLS[i][j] specifying the number of sublayers in the j-th layer in the i-th OLS, the variable OutputLayerIdInOls[i][j] specifying the nuh_layer_id value of the j-th output layer in the i-th OLS, and the variable LayerUsedAsOutputLayerFlag[k] specifying whether the k-th layer is used as an output layer in at least one OLS can be derived as follows:

[0281] For each value of i in the range of 0 to vps_max_layers_minus1, inclusive, the values ​​of LayerUsedAsRefLayerFlag[i] and LayerUsedAsOutputLayerFlag[i] may not both be equal to 0. In other words, there may not be a layer that is neither an output layer of at least one OLS nor a direct reference layer of any other layer.

[0282] For each OLS, there can be at least one layer that is an output layer. In other words, for any value of i in the range of 0 to TotalNumOlss-1 (including 0 and TotalNumOlss-1), the value of NumOutputLayersInOls[i] can be greater than or equal to 1.

[0283] The variable NumLayersInOls[i], which specifies the number of layers in the i-th OLS, and the variable LayerIdInOls[i][j], which specifies the nuh_layer_id value of the j-th layer in the i-th OLS, can be derived as follows:

[0284] The variable OlsLayerIdx[i][j] specifying the OLS layer index of the layer where nuh_layer_id is equal to LayerIdInOls[i][j] is given by: for(i=0;i <TotalNumOlss;i++) for j=0;j <NumLayersInOls[i];j++) OlsLayerIdx[i][LayerIdInOls[i][j]]=j

[0285] The lowest layer in each OLS may be an independent layer. In other words, for each i in the range of 0 to TotalNumOlss-1 (including 0 and TotalNumOlss-1), the value of vps_independent_layer_flag[GeneralLayerIdx[LayerIdInOls[i][0]]] may be equal to 1. Each layer may be included in at least one OLS specified by the VPS. In other words, for each layer with a specific value of nuh_layer_id, where nuhLayerId is equal to one of vps_layer_id[k], k is in the range of 0 to vps_max_layers_minus1 (including 0 and vps_max_layers_minus1), there can be at least one pair of values ​​of i and j, where i is in the range of 0 to TotalNumOlss-1 (including 0 and TotalNumOlss-1), and j is in the range of NumLayersInOls[i]-1 (including NumLayersInOls[i]-1), such that the value of LayerIdInOls[i][j] is equal to nuhLayerId.

[0286] In an embodiment, for the current picture (e.g., syntax element CurrPic), the decoding process may be operated as follows to set the syntax element PictureOutputFlag:

[0287] PictureOutputFlag is set equal to 0 if one of the following conditions is true: (1) The current picture is a RASL picture, and the NoOutputBeforeRecoveryFlag of the associated IRAP picture is equal to 1; (2) gdr_enabled_flag is equal to 1, and the current picture is a GDR picture with NoOutputBeforeRecoveryFlag equal to 1; (3) gdr_enabled_flag is equal to 1, the current picture is associated with a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, and the PicOrderCntVal of the current picture is less than the RpPicOrderCntVal of the associated GDR picture; (4) sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0 and the current AU contains a picture (e.g., syntax element picA) that satisfies all of the following conditions: (a) picA has PictureOutputFlag equal to 1, (b) picA has a nuh_layer_id nuhLid greater than the nuh_layer_id nuhLid of the current picture, (c) PicA belongs to the output layer of the OLS (i.e., OutputLayerIdInOls[TargetOlsIdx][0] is equal to nuhLid); (5)sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 2, andols_output_layer_flag[TargetOlsIdx][GeneralLayerIdx[nuh_layer_id]] is equal to 0.

[0288] If none of the above conditions are true, the syntax element PictureOutputFlag may be set equal to the syntax element pic_output_flag.

[0289] After all slices of the current picture have been decoded, the current decoded picture may be marked as “used for short-term reference”, and each ILRP entry in RefPicList[0] or RefPicList[1] may be marked as “used for short-term reference”.

[0290] In the same or another embodiment, when each layer is an output layer set, the syntax element PictureOutputFlag is set equal to pic_output_flag, regardless of the value of the syntax element ols_mode_idc (864).

[0291] In the same or another embodiment, the syntax element PictureOutputFlag is set to 0 when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag (862) is equal to 0, ols_mode_idc (864) is equal to 0, and the current AU contains a picture picA that satisfies all of the following conditions: picA has PictureOutputFlag equal to 1, picA has a nuh_layer_id nuhLid greater than the nuh_layer_id nuhLid of the current picture, and picA belongs to the output layer of the OLS (i.e., OutputLayerIdInOls[TargetOlsIdx][0] is equal to nuhLid).

[0292] In the same or another embodiment, when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is equal to 0, ols_mode_idc is equal to 2, and ols_output_layer_flag[TargetOlsIdx][GeneralLayerIdx[nuh_layer_id]] is equal to 0, the syntax element PictureOutputFlag is set equal to 0.

[0293] An intra random access point (IRAP) picture may be a coded picture used for random access, which may be an instantaneous decoding refresh (IDR) picture of a closed group supporting a picture structure or a clean random access (CRA) picture of an open group supporting a picture structure. A progressive decoding refresh (GDR) picture may be a picture used for progressive random access with partial refresh of a picture.

[0294] Embodiments of the present disclosure may include syntax elements indicating IRAP or GDR pictures. For example, referring to Fig.26, a picture header (1) may be provided. In the picture header (1), a flag ph_gdr_or_irap_pic_flag (2) may be signaled. The flag indicates that an IRAP or GDR picture is present in the current PU associated with the picture header (1).

[0295] In the same embodiment or another embodiment, if Fig.26 As shown, the flag ph_no_output_of_prior_pics_flag(3) can be conditionally signaled only when ph_gdr_or_irap_pic_flag(2) is equal to 1. The value of ph_no_output_of_prior_pics_flag(3) can be used for the process of outputting and removing pictures from the DPB. The value of the flag can affect the output of previously decoded pictures in the DPB after decoding pictures of non-first AUs in the bitstream in the CVSS AU.

[0296] There is a potential problem that an IRAP picture may have ph_gdr_or_irap_pic_flag equal to 0, because the constraints specified by the semantics of ph_gdr_or_irap_pic_flag may be "one-way", as follows: The flag ph_gdr_or_irap_pic_flag equal to 1 specifies that the current picture is a GDR picture or an IRAP picture. The flag ph_gdr_or_irap_pic_flag equal to 0 specifies that the current picture is not a GDR picture and may or may not be an IRAP picture. When the value of ph_gdr_or_irap_pic_flag is equal to 0 for an IRAP picture, the value of ph_no_output_of_prior_pics_flag may be used for DPB operation without signaling or any inference rules.

[0297] To address this potential problem, in an embodiment, ph_gdr_or_irap_pic_flag can be set The constraint of the semantics of (2) is specified as "bidirectional", so that when the current picture is an IRAP picture, it may be necessary to signal ph_no_output_of_prior_pics_flag(3), as follows: The flag ph_gdr_or_irap_pic_flag(2) equal to 1 specifies that the current picture is a GDR picture or an IRAP picture. The flag ph_gdr_or_irap_pic_flag(2) equal to 0 specifies that the current picture is neither a GDR picture nor an IRAP picture.

[0298] In the same or another embodiment, ph_no_output_of_prior_pics_flag (3) The inference rule for the value when not present may be specified as follows: The flag ph_no_output_of_prior_pics_flag(3) may affect the output of previously decoded pictures in the DPB after decoding pictures of non-first AUs in the bitstream in the CVSS AU. When there is a bitstream conformance requirement, the value of ph_no_output_of_prior_pics_flag(3) shall be the same for all pictures in the AU.

[0299] When ph_no_output_of_prior_pics_flag is present in the picture header (1) of the picture in the AU When ph_no_output_of_prior_pics_flag(3) is not present, the value of ph_no_output_of_prior_pics_flag(3) of the AU is the value of ph_no_output_of_prior_pics_flag(3) of the pictures in the AU. When ph_no_output_of_prior_pics_flag(3) is not present, the value of ph_no_output_of_prior_pics_flag(3) is inferred to be 0.

[0300] Reference Fig. 27 , the AU delimiter (10) may be used to indicate the start of an AU, whether the AU is an IRAP or a GDR AU, and the type of slices present in the coded picture in the AU containing the AU delimiter NAL unit. When the bitstream contains only one layer, there may be no canonical decoding process associated with the AU delimiter (10).

[0301] In the AU delimiter (10), aud_irap_or_gdr_au_flag (12) may indicate the presence of IRAP or GDR AU and may be Fig. 27 The flag aud_irap_or_gdr_au_flag(12) equal to 1 may specify that the AU containing the AU delimiter is an IRAP or a GDR AU. The flag aud_irap_or_gdr_au_flag(12) equal to 0 may specify that the AU containing the AU delimiter(10) is neither an IRAP nor a GDR AU.

[0302] In the same or another embodiment, when the bitstream has multiple layers, the flag aud_irap_or_gdr_au_flag(12) for IRAP or GDR AU may be present, where sps_video_parameter_set_id is greater than 0. A video coding technique or standard may require the presence of AU delimiters for a multi-layer bitstream.

[0303] In the same embodiment or another embodiment, reference is made to Figure 26 to Figure 27 , when aud_irap_or_gdr_au_flag(12) is present and the value of aud_irap_or_gdr_au_flag(12) is equal to 1, the value of ph_gdr_or_irap_pic_flag(2) may be required to be equal to 1. This is because when aud_irap_or_gdr_au_flag(12) in the AU delimiter(10) is equal to 1, each PU may have a GDR or IRAP picture.

[0304] In the same or another embodiment, when pps_mixed_nalu_types_in_pic_flag is equal to 1, the value of ph_no_output_of_prior_pics_flag may not be present, and when it is determined (eg, by a decoder) that pps_mixed_nalu_types_in_pic_flag is present, the value of ph_no_output_of_prior_pics_flag (3) may be ignored.

[0305] In the same or another embodiment, ph_no_output_of_prior_pics_flag (3) It can affect the output of previously decoded pictures in the DPB after decoding pictures of non-first AUs in the bitstream in the CVSS AU. When not present, it can be inferred that the value of ph_no_output_of_prior_pics_flag (3) is equal to 1.

[0306] In the same or another embodiment, to address the problem of using the value of ph_no_output_of_prior_pics_flag(3) without an inference rule, such as Fig.28 As shown, when ph_no_output_of_prior_pics_flag(3) is not present and ph_gdr_or_irap_pic_flag(2) is equal to 1, ph_irap_pic_flag(6) may be used in the picture header(1) to replace ph_gdr_or_irap_pic_flag(2). The flag ph_irap_pic_flag(6) equal to 1 may specify that the current picture is an IRAP picture. The flag ph_irap_pic_flag(6) equal to 0 may specify that the current picture is not an IRAP picture.

[0307] In the same or another embodiment, to address the problem of using the value of ph_no_output_of_prior_pics_flag(3) without an inference rule, such as Fig.29 As shown, when ph_no_output_of_prior_pics_flag(3) does not exist and ph_gdr_or_irap_pic_flag(2) is equal to 1, ph_no_output_of_prior_pics_flag(3) in the picture header (1) may be replaced by sh_no_output_of_prior_pics_flag(23) in the slice header (20).

[0308] In the same embodiment, sh_no_output_of_prior_pics_flag is set only when the NAL unit type of the current VCL NAL is equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT. (23) may be conditionally present in the slice header (20). IDR_W_RADL is the NAL unit type for coded slice segments containing an IDR picture that does not have an associated RASL picture present in the bitstream, but may have an associated RADL picture in the bitstream. IDR_N_LP may be the NAL unit type for coded slice segments containing an IDR picture that does not have an associated leading picture present in the bitstream. CRA_NUT is the NAL unit type for coded slice segments containing a CRA picture.

[0309] In the same or another embodiment, sh_no_output_of_prior_pics_flag (23) After decoding a picture other than the first AU in the bitstream in the CVSS AU, the output of the previously decoded picture in the DPB may be affected.

[0310] In the same or another embodiment, when there is a requirement for bitstream consistency, the value of sh_no_output_of_prior_pics_flag(23) shall be the same for all pictures in an AU. When sh_no_output_of_prior_pics_flag(23) is present in the slice header(20) of a picture in the AU, the value of sh_no_output_of_prior_pics_flag(23) of the AU may be the value of sh_no_output_of_prior_pics_flag(23) of the picture in the AU.

[0311] In the same or another embodiment, when pps_mixed_nalu_types_in_pic_flag in the picture parameter set is equal to 1, the value of sh_no_output_of_prior_pics_flag(23) may not be present. When present, the value of sh_no_output_of_prior_pics_flag(23) may be ignored.

[0312] In the same embodiment or another embodiment, if Fig.30 As shown, aud_irap_au_flag(16) may be present in the AU delimiter(10). The flag aud_irap_au_flag(16) equal to 1 may specify that the AU containing the AU delimiter(10) is an IRAP AU. The flag aud_irap_au_flag(16) equal to 0 may specify that the AU containing the AU delimiter(10) is not an IRAP AU.

[0313] In the same embodiment, when aud_irap_au_flag (16) is present, the value of ph_irap_pic_flag (6) in the picture header (1) may be equal to the value of aud_irap_au_flag in the AU delimiter (10). (16). The flag ph_irap_pic_flag(6) equal to 1 may specify that the picture associated with PH(1) is an IRAP picture. The flag ph_irap_pic_flag(6) equal to 0 may specify that the picture associated with PH(1) is not an IRAP picture.

[0314] In the same embodiment or another embodiment, if Fig.30 As shown, aud_gdr_au_flag(17) may be present in the AU delimiter(10). The flag aud_gdr_au_flag(17) equal to 1 may specify that the AU containing the AU delimiter(10) is a GDR AU. The flag aud_gdr_au_flag(17) equal to 0 may specify that the AU containing the AU delimiter(10) is not a GDR AU.

[0315] In the same embodiment, when aud_gdr_au_flag(17) is present in the AU delimiter(10), the value of ph_gdr_pic_flag(7) in the picture header(1) may be equal to aud_gdr_au_flag(17) in the AU delimiter(10). The flag ph_gdr_pic_flag(7) equal to 1 may specify the same as PH (1) The associated picture is a GDR picture. The flag ph_gdr_pic_flag(7) equal to 0 may specify that the picture associated with PH(1) is not a GDR picture.

[0316] Progressive Decoding Refresh (GDR) can be specified by the following definition: GDRAU: AU where there is a PU for each layer specified by the VPS and the coded picture in each current PU is a GDR picture. GDR PU: PU where the coded picture is a GDR picture. GDR picture: A picture in which the nal_unit_type of each VCL NAL unit is equal to GDR_NUT. GDR sub-picture: A sub-picture in which the nal_unit_type of each VCL NAL unit is equal to GDR_NUT. GDR_NUT: NAL unit type that includes the coded tile groups of a GDR picture.

[0317] According to an embodiment, the first picture in the bitstream in decoding order may be an IRAP or GDR picture. A trailing picture associated with an IRAP or GDR picture may also follow the IRAP or GDR picture in decoding order. Pictures that follow the associated IRAP picture in output order and precede the associated IRAP picture in decoding order may not be allowed.

[0318] In an embodiment, reference is made to Fig.31 , a syntax element indicating GDR is signaled in PH(1), such as ph_gdr_pic_flag(7). When not present, the value of ph_gdr_pic_flag(7) may be inferred to be equal to 0. When sps_gdr_enabled_flag is equal to 0, the value of ph_gdr_pic_flag(7) may be inferred to be equal to 0. The syntax element ph_recovery_poc_cnt(34) may specify a recovery point of a decoded picture in output order.

[0319] When the current picture is a GDR picture, the variable recoveryPointPocVal can be obtained as follows: recoveryPointPocVal=PicOrderCntVal+ph_recovery_poc_cnt

[0320] In the same embodiment or another embodiment, if Fig.31As shown, ph_recovery_poc_cnt (34) is signaled after the picture order count (POC) syntax elements (e.g., ph_pic_order_cnt_lsb (32) and ph_poc_msb_cycle_val (36)) are signaled because PicOrderCntVal used to derive recoveryPointPocVal is derived from the values ​​of ph_pic_order_cnt_lsb (32) and ph_poc_msb_cycle_val (36).

[0321] If the current picture is a GDR picture and there is a picture picA that follows the current GDR picture in decoding order in the CLVS with PicOrderCntVal equal to recoveryPointPocVal, then the picture picA can be called the recovery point picture. Otherwise, the first picture in the CLVS in output order that has a PicOrderCntVal greater than recoveryPointPocVal can be called the recovery point picture. The recovery point picture may not be before the current GDR picture in decoding order. A picture associated with the current GDR picture and having a PicOrderCntVal less than recoveryPointPocVal can be called a recovery picture of the GDR picture. The value of ph_recovery_poc_cnt(34) can be in the range of 0 to MaxPicOrderCntLsb-1 (including 0 and MaxPicOrderCntLsb-1).

[0322] In the same or another embodiment, the recovery point picture may not precede the current GDR picture in either decoding order or output order.

[0323] In the same or another embodiment, the recovery picture may not precede the current GDR picture in either decoding order or output order.

[0324] In the same or another embodiment, a recovery picture may precede the associated recovery point picture in both decoding order and output order.

[0325] In the same or another embodiment, when the current picture is a GDR picture or a recovery picture of a GDR picture, and the current picture contains a non-CTU equal boundary between a "refresh area" (when present, that is, an area that has an exact match with the decoded sample value when the decoding process is started from the GDR picture compared to starting the decoding process from the previous IRAP picture in decoding order) and a "dirty area" (when present, that is, an area that may not have an exact match with the decoded sample value when the decoding process is started from the GDR picture compared to starting the decoding process from the previous IRAP picture in decoding order), chroma residual scaling with luma mapping with chroma scaling (LMCS) may have to be disabled in the current picture to avoid the "dirty area" affecting the decoded sample values ​​of the "refresh area".

[0326] In the same or another embodiment, the value of recoveryPointPocVal of layer A may be equal to or greater than the recoveryPointPocVal of layer A's reference layer.

[0327] In the same or another embodiment, the value of recoveryPointPocVal of layer A with layerId equal to m may be equal to or greater than recoveryPointPocVal of another layer B with layerId equal to n, where m is greater than n.

[0328] In the same or another embodiment, the value of recoveryPointPocVal of layer A with layerId equal to m may be equal to or greater than recoveryPointPocVal of another layer B with layerId equal to n, where m is greater than n and layer A and layer B belong to the same output layer set.

[0329] In the same or another embodiment, the value of recoveryPointPocVal of layer A may be equal to the recoveryPointPocVal of layer A's reference layer.

[0330] In the same or another embodiment, the value of recoveryPointPocVal of layer A with layerId equal to m may be equal to recoveryPointPocVal of another layer B with layerId equal to n, where m is greater than n.

[0331] In the same or another embodiment, the value of recoveryPointPocVal of layer A with layerId equal to m may be equal to recoveryPointPocVal of another layer B with layerId equal to n, where m is greater than n and layer A and layer B belong to the same output layer set.

[0332] In the same or another embodiment, when pps_mixed_nalu_types_in_pic_flag is equal to 1, the following may apply: (1) A picture should have at least two sub-pictures. (2) The VCL NAL unit of a picture shall have two or more different nal_unit_type values. (3) There shall be no VCL NAL units for pictures with nal_unit_type equal to GDR_NUT. (4) The picture should not be a restored or recovered picture associated with a GDR picture. (5) When the VCL NAL unit of at least one sub-picture of a picture has a specific value of nal_unit_type equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT, the VCL NAL units of other sub-pictures in the picture shall all have nal_unit_type equal to TRAIL_NUT.

[0333] TRAIL_NUT is a NAL unit type that contains the coded tile groups of a VCL non-STSA trailing picture.

[0334] At the beginning of the decoding process for each slice of a picture, a decoding process for reference picture list construction may be called to obtain reference picture list 0 (RefPicList[0]) and reference picture list 1 (RefPicList[1]). After constructing one or more reference picture lists, a decoding process for reference picture marking may be called, where a reference picture may be marked as "not used for reference", "used for short-term reference", or "used for long-term reference".

[0335] Reference Fig.32, according to an embodiment, a decoding process (40) may be performed by a decoder. In the decoding process (40), one or more reference picture lists (RPLs) (42) may be constructed. When constructing one or more reference picture lists, one or more pictures may be unavailable due to random access or unexpected picture loss. The decoder may determine whether the reference picture in the RPL is available in the DPB (44). If the reference picture is determined to be unavailable, the unavailable reference picture may be marked as "no reference picture". In order to avoid any decoder crash or unexpected behavior, an unavailable reference picture may be immediately generated with default values ​​for pixels and parameters (46). After the unavailable reference picture is generated (and / or the reference picture is determined to be available), the decoder may check the validity of all reference pictures in the reference picture list (including the generated picture) (48).

[0336] In the same or another embodiment, one or more reference picture lists are constructed by parsing RPL syntax elements in the SPS, PH, and / or SH (42). After construction (42), one or more reference pictures in the RPL list may be unavailable because when random access occurs at a CRA picture, a Random Access Skipped Leading (RASL) picture associated with the CRA picture may be discarded by the decoder or system or may not be successfully decoded. Unavailable reference pictures may be generated with default values ​​for pixels and parameters (46).

[0337] In the same or another embodiment, when the current picture is an IDR picture with sps_idr_rpl_present_flag equal to 1 or pps_rpl_info_in_ph_flag equal to 1, a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, or a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, at least one of the following decoding processes for generating an unavailable reference picture (46) is called, which may need to be called only for the first slice of the picture.

[0338] A. General Decoding Process for Generating Unavailable Reference Pictures

[0339] This process may be called once for each coded picture when the current picture is an IDR picture with sps_idr_rpl_present_flag equal to 1 or pps_rpl_info_in_ph_flag equal to 1, a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, or a GDR picture with NoOutputBeforeRecoveryFlag equal to 1. When this process is called, the following may apply: For each RefPicList[i][j] equal to "no reference picture", where i is in the range of 0 to 1 (inclusive) and j is in the range of 0 to num_ref_entries[i][RplsIdx[i]]-1 (inclusive), generate the picture in "Generate an unavailable picture" as described below, and the following apply: (1) Set the value of nuh_layer_id of the generated picture to be equal to the nuh_layer_id of the current picture. (2) If st_ref_pic_flag[i][RplsIdx[i]][j] is equal to 1 and inter_layer_ref_pic_flag[i][RplsIdx[i]][j] is equal to 0, set the value of PicOrderCntVal of the generated picture equal to RefPicPocList[i][j] and mark the generated picture as "used for short-term reference". (3) Otherwise, when st_ref_pic_flag[i][RplsIdx[i]][j] is equal to 0 and inter_layer_ref_pic_flag[i][RplsIdx[i]][j] is equal to 0, the value of PicOrderCntVal of the generated picture is set equal to RefPicLtPocList[i][j], the value of ph_pic_order_cnt_lsb of the generated picture is inferred to be equal to (RefPicLtPocList[i][j] & (MaxPicOrderCntLsb-1)), and the generated picture is marked as "used for long-term reference". (4) Set the value of the PictureOutputFlag of the generated reference picture to 0. (5) Set RefPicList[i][j] to the generated reference picture. (6) Set the value of the TemporalId of the generated image to be equal to the TemporalId of the current image. (7) Set the value of ph_non_ref_pic_flag of the generated picture to be equal to 0. (8) Set the value of ph_pic_parameter_set_id of the generated picture to be equal to ph_pic_parameter_set_id of the current picture.

[0340] The flag ph_non_ref_pic_flag equal to 1 may specify that the picture associated with the PH is never used as a reference picture. The flag ph_non_ref_pic_flag equal to 0 may specify that the picture associated with the PH may or may not be used as a reference picture.

[0341] B. Generation of an unusable image

[0342] When this process is called, the unavailable image is generated as follows: (1) Set the value of each element in the sample array SL of the picture to be equal to 1<<(BitDepth-1). (2) When sps_chroma_format_idc is not equal to 0, the sample array S of the picture Cb and S Cr The value of each element in is set equal to 1<<(BitDepth-1). (3) Set the prediction mode CuPredMode[0][x][y] to be equal to MODE_INTRA, where x ranges from 0 to pps_pic_width_in_luma_samples-1 (including 0 and pps_pic_width_in_luma_samples-1), and y ranges from 0 to pps_pic_height_in_luma_samples-1 (including 0 and pps_pic_height_in_luma_samples-1).

[0343] In the same or another embodiment, after generating the unavailable reference pictures in the RPL list, a bitstream consistency check for all reference pictures or activated reference pictures in the RPL list may be invoked, for example, by a decoder. For example, the decoder may check that the following constraints apply to the bitstream consistency: (1) For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] should not be less than NumRefIdxActive[i]. (2) The picture referred to by each active entry in RefPicList[0] or RefPicList[1] shall be present in the DPB and shall have a TemporalId less than or equal to the TemporalId of the current picture. (3) The picture referred to by each entry in RefPicList[0] or RefPicList[1] shall not be the current picture and shall have ph_non_ref_pic_flag equal to 0. (4) A short-term reference picture (STRP) entry in RefPicList[0] or RefPicList[1] of a slice of a picture and a long-term reference picture (LTRP) entry in RefPicList[0] or RefPicList[1] of an identical slice or a different slice of the same picture shall not refer to the same picture. (5) There shall be no LTRP entry in RefPicList[0] or RefPicList[1] for which the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture referred to by the entry is greater than or equal to 2 24 . (6) Let setOfRefPics be the set of unique pictures referred to by all entries in RefPicList[0] that have the same nuh_layer_id as the current picture and all entries in RefPicList[1] that have the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics shall be less than or equal to MaxDpbSize-1 (inclusive), where MaxDpbSize and setOfRefPics are the same for all slices of the picture. (7) When the nal_unit_type of the current slice is equal to STSA_NUT, there will be no activated entry in RefPicList[0] or RefPicList[1] with TemporalId equal to the TemporalId of the current picture and nuh_layer_id equal to the nuh_layer_id of the current picture. (8) When the current picture is a picture that follows, in decoding order, a step-wise temporal sub-layer access (STSA) picture whose TemporalId is equal to the TemporalId of the current picture and whose nuh_layer_id is equal to the nuh_layer_id of the current picture, there shall not be a picture that precedes the STSA picture in decoding order, whose TemporalId is equal to the TemporalId of the current picture, and whose nuh_layer_id is equal to the nuh_layer_id of the current picture (included as an active entry in RefPicList[0] or RefPicList[1]). (9) When the current sub-picture (wherein TemporalId is equal to a specific value tId, nuh_layer_id is equal to a specific value layerId, and sub-picture index is equal to a specific value subpicIdx) is a sub-picture that follows, in decoding order, a STSA sub-picture whose TemporalId is equal to tId, nuh_layer_id is equal to layerId, and sub-picture index is equal to subpicIdx, there will be no picture whose TemporalId is equal to tId and nuh_layer_id is equal to layerId that precedes, in decoding order, the picture containing the STSA sub-picture (which is included as an activated entry in RefPicList[0] or RefPicList[1]). (10) When the current picture (where nuh_layer_id is equal to a particular value layerId) is an IRAP picture, there shall be no picture referred to by an entry in RefPicList[0] or RefPicList[1] that precedes any previous IRAP picture in output order or decoding order, any previous IRAP picture having nuh_layer_id equal to layerId in decoding order (when present). (11) When the current sub-picture (where nuh_layer_id is equal to a particular value layerId and the sub-picture index is equal to a particular value subpicIdx) is an IRAP sub-picture, there shall be no picture referred to by an entry in RefPicList[0] or RefPicList[1] that precedes any previous picture in output order or decoding order, and any previous picture in decoding order (when present) contains an IPAP sub-picture where nuh_layer_id is equal to layerId and the sub-picture index is equal to subpicIdx. (12) When the current picture is not a RASL picture associated with a CRA picture in which NoOutputBeforeRecoveryFlag is equal to 1, there will be no pictures generated by the decoding process used to generate unavailable reference pictures for the CRA picture associated with the current picture (which is referred to by an activated entry in RefPicList[0] or RefPicList[1]). (13) When the current sub-picture is not a RASL sub-picture associated with a CRA sub-picture in a CRA picture in which NoOutputBeforeRecoveryFlag is equal to 1, there will be no picture generated by the decoding process used to generate an unavailable reference picture for the CRA picture (which is referred to by an activated entry in RefPicList[0] or RefPicList[1]), and the CRA picture contains the CRA sub-picture associated with the current sub-picture. (14) When the current picture (where nuh_layer_id is equal to a specific value layerId) is not any of the following pictures, there shall be no pictures (which are referred to by entries in RefPicList[0] or RefPicList[1]) generated by the decoding process for generating unavailable reference pictures for IRAP or GDR pictures associated with the current picture: (a) IDR pictures with sps_idr_rpl_present_flag equal to 1 or pps_rpl_info_in_ph_flag equal to 1. (b) CRA picture with NoOutputBeforeRecoveryFlag equal to 1. (c) A picture associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1 that precedes, in decoding order, the leading picture associated with the same CRA picture. (d) Leading pictures associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1. (e) GDR picture with NoOutputBeforeRecoveryFlag equal to 1. (f) Recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId. (15) There shall be no picture (which is referred to by an entry in RefPicList[0] or RefPicList[1]) that has been processed by decoding for generating an unusable reference picture for an IRAP picture or a GDR picture when the current sub-picture (where nuh_layer_id is equal to a specific value layerId and the sub-picture index is equal to a specific value subpicIdx) is not any of the following pictures, the IRAP or GDR picture containing the IRAP sub-picture or GDR sub-picture associated with the current sub-picture includes: (a) An IDR sub-picture in an IDR picture with sps_idr_rpl_present_flag equal to 1 or pps_rpl_info_in_ph_flag equal to 1. (b) A CRA sub-picture in a CRA picture where NoOutputBeforeRecoveryFlag is equal to 1. (c) A sub-picture associated with a CRA sub-picture in a CRA picture with NoOutputBeforeRecoveryFlag equal to 1 that precedes, in decoding order, the leading picture associated with the same CRA picture. (d) Leading sub-pictures associated with a CRA sub-picture in a CRA picture with NoOutputBeforeRecoveryFlag equal to 1. (e) A GDR sub-picture in a GDR picture in which NoOutputBeforeRecoveryFlag is equal to 1. (f) A sub-picture in a recovery picture of a GDR picture in which NoOutputBeforeRecoveryFlag is equal to 1 and nuh_layer_id is equal to layerId. (16) When the current picture follows an IRAP picture with the same nuh_layer_id value in both decoding order and output order, there shall be no pictures (which are referred to by active entries in RefPicList[0] or RefPicList[1]) that precede the IRAP picture in output order or decoding order. (17) When the current sub-picture follows an IRAP sub-picture with the same nuh_layer_id value and the same sub-picture index value in both decoding order and output order, there shall be no picture (which is referred to by an active entry in RefPicList[0] or RefPicList[1]) that precedes the picture containing the IRAP sub-picture in output order or decoding order. (18) When the current picture follows an IRAP picture with the same nuh_layer_id value and the leading picture associated with the IRAP picture (if any) in both decoding order and output order, there shall be no pictures (which are referred to by entries in RefPicList[0] or RefPicList[1]) that precede the IRAP picture in output order or decoding order. (19) When the current sub-picture follows an IRAP sub-picture with the same nuh_layer_id value and the same sub-picture index value and the leading sub-picture associated with the IRAP sub-picture (if any) in both decoding order and output order, there shall be no picture (which is referred to by an entry in RefPicList[0] or RefPicList[1]) that precedes the picture containing the IRAP sub-picture in output order or decoding order. (20) When the current picture is a random access decodable leading (RADL) picture, none of the following active entries will exist in RefPicList[0] or RefPicList[1]: (a) A RASL picture with pps_mixed_nalu_types_in_pic_flag equal to 0. This means that the activation entry of the RPL of a RADL picture may refer to a RASL picture with pps_mixed_nalu_types_in_pic_flag equal to 1. However, when decoding starts from the associated CRA picture, such a RADL picture may still be correctly decoded because the RADL sub-picture in the referenced RASL picture will be correctly decoded because the RADL picture will only reference RADL sub-pictures in the referenced RASL picture, as imposed by the next constraint that RADL sub-pictures are not allowed to reference RASL sub-pictures. (b) A picture that precedes the associated IRAP picture in decoding order. (21) When the current sub-picture (where nuh_layer_id is equal to a specific value layerId and the sub-picture index is equal to a specific value subpicIdx) is a RADL sub-picture, none of the following active entries shall exist in RefPicList[0] or RefPicList[1]: (a) A picture with nuh_layer_id equal to layerId containing a RASL sub-picture with sub-picture index equal to subpicIdx. (b) A picture that precedes, in decoding order, the picture containing the associated IRAP sub-picture. (22) The following constraints apply to pictures referenced by each ILRP entry (when present) in RefPicList[0] or RefPicList[1] of the slice of the current picture: (a) The picture will be in the same AU as the current picture. (b) The picture will exist in the DPB. (c) The nuh_layer_id refPicLayerId of the picture will be smaller than the nuh_layer_id of the current picture. (d) Either of the following constraints applies: the picture shall be an IRAP picture; or the TemporalId of the picture shall be less than or equal to Max(0,vps_max_tid_il_ref_pics_plus1[currLayerIdx][refLayerIdx]-1), where currLayerIdx and refLayerIdx are equal to GeneralLayerIdx[nuh_layer_id] respectively. and GeneralLayerIdx[refpicLayerId]. (23) Each ILRP entry in the slice's RefPicList[0] or RefPicList[1] (when present) shall be the active entry. (24) When vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 0 and sps_num_subpics_minus1 is greater than 0, either (but not both) of the following two conditions shall be true: (a) The picture referred to by each active entry in RefPicList[0] or RefPicList[1] has the same sub-picture layout as the current picture (i.e., the SPS referenced by the picture has the same value of sps_num_subpics_minus1 as the current picture and the same values ​​of sps_subpic_ctu_top_left_x[j], sps_subpic_ctu_top_left_y[j], sps_subpic_width_minus1[j], and sps_subpic_height_minus1[j] for each value of j in the range of 0 to sps_num_subpics_minus1, inclusive). (b) The picture referred to by each active entry in RefPicList[0] or RefPicList[1] is an ILRP with the value of sps_num_subpics_minus1 equal to 0.

[0344] C. Decoding Process for Reference Picture Marking

[0345] In the same or another embodiment, when a reference picture is marked, the following decoding process may be invoked.

[0346] This process can be called once for each picture after decoding of the slice header and decoding process of construction of reference picture lists for the slice, but before decoding of the slice data. This process can cause one or more reference pictures in the DPB to be marked as "unused for reference" or "used for long-term reference".

[0347] A decoded picture in the DPB may be marked as "unused for reference", "used for short-term reference", or "used for long-term reference", but only one of these three at any given moment during the operation of the decoding process. Assigning one of these markings to a picture may implicitly remove another of these markings when applicable. When a picture is referred to as being marked as "used for reference", this collectively refers to the picture being marked as either "used for short-term reference" or "used for long-term reference" (but not both).

[0348] STRP and ILRP can be identified by their nuh_layer_id and PicOrderCntVal values. LTRP can be identified by their nuh_layer_id value and Log2(MaxLtPicOrderCntLsb)LSB of their PicOrderCntVal value.

[0349] If the current picture is a CLVSS picture, all reference pictures (if any) currently in the DPB that have the same nuh_layer_id as the current picture may be marked as "unused for reference".

[0350] Otherwise, the following may apply: (1) For each LTRP entry in RefPicList[0] or RefPicList[1], when the picture is a LTRP with the same nuh_layer_id as the current picture, mark the picture as "used for long-term reference". (2) Mark each reference picture with the same nuh_layer_id as the current picture that is not referred to by any entry in RefPicList[0] or RefPicList[1] in the DPB as “unused for reference”. (3) For each ILRP entry in RefPicList[0] or RefPicList[1], mark the picture as “used for long-term reference”.

[0351] In the same or another embodiment, for each LTRP entry in RefPicList[0] or RefPicList[1], the picture is marked as "used for long-term reference" when a picture with the same nuh_layer_id as the current picture is marked as "used for short-term reference".

[0352] In the same or another embodiment, when constructing one or more RPL lists, the following may apply: For each RefPicList[i][j] that is equal to "no reference picture", where i is in the range of 0 to 1 (including 0 and 1) and j is in the range of 0 to num_ref_entries[i][RplsIdx[i]]-1 (including 0 and num_ref_entries[i][RplsIdx[i]]-1), generate a picture.

[0353] In the same or another embodiment, when constructing one or more RPL lists, the following may apply: For each RefPicList[i][j] equal to "no reference picture" (where i is in the range of 0 to 1 (including 0 and 1) and j is in the range of 0 to NumRefIdxActive[i]-1 (including 0 and NumRefIdxActive[i]-1)), generate a picture.

[0354] In the decoding process for reference picture list construction, the reference picture lost in the DPB can be set equal to "no reference picture". The unavailable picture equal to "no reference picture" is generated by the decoding process for generating unavailable reference pictures for consistency check purposes. The problem is that not only the unavailable reference pictures in the same layer, but also the unavailable inter-layer reference pictures in the reference layer are set equal to "no reference picture", as shown below:

[0355] However, in the decoding process for generating unavailable reference pictures, all unavailable pictures equal to "no reference picture" may be processed as reference pictures in the same layer, so that the nuh_layer_id value and PicOrderCntVal value of the unavailable inter-layer reference picture are set to incorrect values. Moreover, the unavailable inter-layer reference pictures are not correctly marked as "used for long-term reference". These erroneous values ​​lead to errors in the decoding process of subsequent pictures and bitstream consistency checks. Therefore, according to an embodiment, the unavailable inter-layer reference picture equal to "no reference picture" should be correctly generated as follows:

[0356] D. Improved Decoding Process for Generating Unavailable Reference Pictures

[0357] This process is called once for each coded picture when the current picture is an IDR picture with sps_idr_rpl_present_flag equal to 1 or pps_rpl_info_in_ph_flag equal to 1, a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, or a GDR picture with NoOutputBeforeRecoveryFlag equal to 1.

[0358] When this process is called, for each RefPicList[i][j] equal to "no reference picture", where i is in the range of 0 to 1 (inclusive), and j is in the range of 0 to num_ref_entries[i][RplsIdx[i]]-1 (inclusive), pictures may be generated as previously described in this disclosure, and the following may apply: (1) If inter_layer_ref_pic_flag[i][RplsIdx[i]][j] is equal to 0, the value of nuh_layer_id of the generated picture is set equal to nuh_layer_id of the current picture. (2) Otherwise (inter_layer_ref_pic_flag[i][RplsIdx[i]][j] is equal to 1), the value of nuh_layer_id of the generated picture is set equal to vps_layer_id[DirectRefLayerIdx[GeneralLayerIdx[nuh_layer_id]][ilrp_idx[i][RplsIdx][j]]]. (3) If st_ref_pic_flag[i][RplsIdx[i]][j] is equal to 1 and inter_layer_ref_pic_flag[i][RplsIdx[i]][j] is equal to 0, the value of PicOrderCntVal of the generated picture is set equal to RefPicPocList[i][j], and the generated picture is marked as "used for short-term reference". (4) If st_ref_pic_flag[i][RplsIdx[i]][j] is equal to 0 and inter_layer_ref_pic_flag[i][RplsIdx[i]][j] is equal to 0, the value of PicOrderCntVal of the generated picture is set equal to RefPicLtPocList[i][j], the value of ph_pic_order_cnt_lsb of the generated picture is inferred to be equal to (RefPicLtPocList[i][j] & (MaxPicOrderCntLsb-1)), and the generated picture is marked as "used for long-term reference". (5) Otherwise, if inter_layer_ref_pic_flag[i][RplsIdx[i]][j] is equal to 1, the value of PicOrderCntVal of the generated picture is set equal to PicOrderCntVal of the current picture, the value of ph_pic_order_cnt_lsb of the generated picture is inferred to be equal to ph_pic_order_cnt_lsb of the current picture, and the generated picture is marked as "used for long-term reference". (6) Set the value of PictureOutputFlag of the generated reference picture to 0. (7) Set RefPicList[i][j] to the generated reference picture. (8) Set the value of the TemporalId of the generated image to be equal to the TemporalId of the current image. (9) Set the value of ph_non_ref_pic_flag of the generated picture to be equal to 0. (10) Set the value of ph_pic_parameter_set_id of the generated picture to be equal to ph_pic_parameter_set_id of the current picture.

[0359] In the above decoding process for reference picture marking, to clarify the marking process, modify the following statement: For each LTRP entry in RefPicList[0] or RefPicList[1], when the picture is a STRP with the same nuh_layer_id as the current picture, mark the picture as "for long-term reference".

[0360] As a first option, the original sentence is modified as follows: For each LTRP entry in RefPicList[0] or RefPicList[1], when the picture is currently marked as "used for short-term reference" with the same nuh_layer_id as the current picture, mark the picture as "used for long-term reference".

[0361] As a second option, the original sentence is modified as follows: For each LTRP entry in RefPicList[0] or RefPicList[1], when the picture is a LTRP with the same nuh_layer_id as the current picture, mark the picture as "used for long-term reference".

[0362] The original sentence is a bit confusing because it says that the STRP is marked as "for long-term reference". If the sentence means to mark the LTRP picture of the current picture in RefPicList[0] or RefPicList[1] as "for long-term reference", then option 1 can be used when the LTRP is currently marked as "for short-term reference". Otherwise, option 2 can be used.

[0363] E. Improved Decoding Process for Reference Picture List Construction

[0364] In HEVC, the reference picture list is constructed after the unavailable reference pictures are generated in the RPS. In VVC (current draft), the reference picture list is constructed before the unavailable reference pictures are generated in the RPS. If the bitstream consistency check is performed before the unavailable reference pictures are generated as previously specified, the bitstream cannot comply with the specified constraints. To clarify the order, the decoding process can be modified as follows to have the following constraints on bitstream consistency: (1) For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] shall not be less than NumRefIdxActive[i]. (2) The picture referred to by each active entry in RefPicList[0] or RefPicList[1] shall exist in the DPB and shall have a TemporalId less than or equal to the TemporalId of the current picture. (3) The picture referred to by each entry in RefPicList[0] or RefPicList[1] shall not be the current picture and shall have ph_non_ref_pic_flag equal to 0. (4) A STRP entry in RefPicList[0] or RefPicList[1] of a slice of a picture and an LTRP entry in RefPicList[0] or RefPicList[1] of the same slice or a different slice of the same picture shall not refer to the same picture. (5) There shall be no LTRP entry in RefPicList[0] or RefPicList[1] for which the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture referred to by the entry is greater than or equal to 2 24 . (6) Let setOfRefPics be the set of unique pictures referred to by all entries in RefPicList[0] that have the same nuh_layer_id as the current picture and all entries in RefPicList[1] that have the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics shall be less than or equal to MaxDpbSize-1 (inclusive), where MaxDpbSize and setOfRefPics shall be the same for slices of a picture. (7) When the nal_unit_type of the current slice is equal to STSA_NUT, there will be no activated entry in RefPicList[0] or RefPicList[1] with TemporalId equal to the TemporalId of the current picture and nuh_layer_id equal to the nuh_layer_id of the current picture. (8) When the current picture is after an STSA picture in decoding order in which TemporalId is equal to the TemporalId of the current picture and nuh_layer_id is equal to the nuh_layer_id of the current picture, there shall be no picture in decoding order before the STSA picture in which TemporalId is equal to the TemporalId of the current picture and in which nuh_layer_id is equal to the nuh_layer_id of the current picture, and the current picture is included as an active entry in RefPicList[0] or RefPicList[1]. (9) When the current sub-picture (wherein TemporalId is equal to a specific value tId, nuh_layer_id is equal to a specific value layerId, and the sub-picture index is equal to a specific value subpicIdx) is a sub-picture that follows, in decoding order, a STSA sub-picture in which TemporalId is equal to tId, nuh_layer_id is equal to layerId, and the sub-picture index is equal to subpicIdx, there will be no picture in decoding order with TemporalId equal to tId and nuh_layer_id equal to layerId that precedes the picture containing the STSA sub-picture (included as an active entry in RefPicList[0] or RefPicList[1]). (10) When the current picture (where nuh_layer_id is equal to a particular value layerId) is an IRAP picture, there shall be no picture referred to by an entry in RefPicList[0] or RefPicList[1] that precedes any previous IRAP picture in output order or decoding order, any previous IRAP picture having nuh_layer_id equal to layerId in decoding order (when present). (11) When the current sub-picture (nuh_layer_id equals a particular value layerId and the sub-picture index equals a particular value subpicIdx) is an IRAP sub-picture, there shall be no picture referred to by an entry in RefPicList[0] or RefPicList[1] that precedes any previous picture in output order or decoding order, and any previous picture in decoding order (when present) contains an IPRP sub-picture in which nuh_layer_id is equal to layerId and the sub-picture index is equal to subpicIdx. (12) When the current picture is not a RASL picture associated with a CRA picture in which NoOutputBeforeRecoveryFlag is equal to 1, there will be no pictures generated by the decoding process used to generate unavailable reference pictures for the CRA picture associated with the current picture (refer to by active entries in its RefPicList[0] or RefPicList[1]). (13) When the current sub-picture is not a RASL sub-picture associated with a CRA sub-picture in a CRA picture in which NoOutputBeforeRecoveryFlag is equal to 1, there will be no picture generated by the decoding process used to generate an unavailable reference picture for the CRA picture (which is referred to by an activated entry in RefPicList[0] or RefPicList[1]), and the CRA picture contains the CRA sub-picture associated with the current sub-picture. (14) When the current picture (where nuh_layer_id is equal to a specific value layerId) is not any of the following pictures, there shall be no pictures (which are referred to by entries in RefPicList[0] or RefPicList[1]) generated by the decoding process for generating unavailable reference pictures for IRAP or GDR pictures associated with the current picture: (a) IDR pictures with sps_idr_rpl_present_flag equal to 1 or pps_rpl_info_in_ph_flag equal to 1. (b) CRA picture with NoOutputBeforeRecoveryFlag equal to 1. (c) A picture associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1 that precedes, in decoding order, the leading picture associated with the same CRA picture. (d) Leading picture associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1 (e) GDR picture with NoOutputBeforeRecoveryFlag equal to 1. (f) Recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id e equal to layerId. (15) There shall be no picture (which is referred to by an entry in RefPicList[0] or RefPicList[1]) that has passed the decoding process for generating an unusable reference picture for an IRAP picture or a GDR picture when the current sub-picture (nuh_layer_id is equal to a specific value layerId and the sub-picture index is equal to a specific value subpicIdx) is not any of the following pictures, the IRAP picture or the GDR picture containing the IRAP sub-picture or the GDR sub-picture associated with the current sub-picture: (a) An IDR sub-picture in an IDR picture with sps_idr_rpl_present_flag equal to 1 or pps_rpl_info_in_ph_flag equal to 1. (b) A CRA sub-picture in a CRA picture where NoOutputBeforeRecoveryFlag is equal to 1. (c) A sub-picture associated with a CRA sub-picture in a CRA picture with NoOutputBeforeRecoveryFlag equal to 1 that precedes, in decoding order, the leading picture associated with the same CRA picture. (d) Leading sub-pictures associated with a CRA sub-picture in a CRA picture with NoOutputBeforeRecoveryFlag equal to 1. (e) A GDR sub-picture in a GDR picture in which NoOutputBeforeRecoveryFlag is equal to 1. (f) A sub-picture in a recovery picture of a GDR picture in which NoOutputBeforeRecoveryFlag is equal to 1 and nuh_layer_id is equal to layerId. (16) When the current picture follows an IRAP picture with the same nuh_layer_id value in both decoding order and output order, there shall be no pictures (which are referred to by active entries in RefPicList[0] or RefPicList[1]) that precede the IRAP picture in output order or decoding order. (17) When the current sub-picture follows an IRAP sub-picture with the same nuh_layer_id value and the same sub-picture index value in both decoding order and output order, there shall be no picture (which is referred to by an active entry in RefPicList[0] or RefPicList[1]) that precedes the picture containing the IRAP sub-picture in output order or decoding order. (18) When the current picture follows an IRAP picture with the same nuh_layer_id value and the leading picture associated with the IRAP picture (if any) in both decoding order and output order, there shall be no pictures (which are referred to by entries in RefPicList[0] or RefPicList[1]) that precede the IRAP picture in output order or decoding order. (19) When the current sub-picture follows an IRAP sub-picture with the same nuh_layer_id value and the same sub-picture index value and the leading sub-picture associated with the IRAP sub-picture (if any) in both decoding order and output order, there shall be no picture (which is referred to by an entry in RefPicList[0] or RefPicList[1]) that precedes the picture containing the IRAP sub-picture in output order or decoding order. (20) When the current picture is a RADL picture, none of the following active entries will exist in RefPicList[0] or RefPicList[1]: (a) A RASL picture with pps_mixed_nalu_types_in_pic_flag equal to 0. This means that an activated entry of the RPL of a RADL picture may reference a RASL picture with pps_mixed_nalu_types_in_pic_flag equal to 1. However, such a RADL picture may still be correctly decoded when decoding starts from the associated CRA picture, because the RADL sub-picture in the referenced RASL picture will be correctly decoded, because the RADL picture will only reference RADL sub-pictures in the referenced RASL picture, as imposed by the next constraint that does not allow reference to RADL sub-pictures in RASL pictures. (b) A picture that precedes the associated IRAP picture in decoding order. (21) When the current sub-picture (where nuh_layer_id is equal to a specific value layerId and the sub-picture index is equal to a specific value subpicIdx) is a RADL sub-picture, none of the following active entries shall exist in RefPicList[0] or RefPicList[1]: (a) A picture with nuh_layer_id equal to layerId containing a RASL sub-picture with sub-picture index equal to subpicIdx. (b) A picture that precedes, in decoding order, the picture containing the associated IRAP sub-picture. (22) The following constraints apply to the pictures referred to by each ILRP entry (when present) in RefPicList[0] or RefPicList[1] of the slice of the current picture: (a) The picture will be in the same AU as the current picture. (b) The picture will exist in the DPB. (c) The nuh_layer_id refPicLayerId of the picture will be smaller than the nuh_layer_id of the current picture. (d) Any of the following constraints applies: the picture shall be an IRAP picture; the TemporalId of the picture shall be less than or equal to Max(0,vps_max_tid_il_ref_pics_plus1 [currLayerIdx][refLayerIdx]–1), where currLayerIdx and refLayerIdx are equal to GeneralLayerIdx[nuh_layer_id] and GeneralLayerIdx[refpicLayerId], respectively. (23) Each ILRP entry in the slice's RefPicList[0] or RefPicList[1] (when present) shall be the active entry. (24) When vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 0 and sps_num_subpics_minus1 is greater than 0, either (but not both) of the following two conditions shall be true: (a) The picture referred to by each active entry in RefPicList[0] or RefPicList[1] has the same sub-picture layout as the current picture (i.e., the SPS referenced by the picture has the same value of sps_num_subpics_minus1 as the current picture and the same values ​​of sps_subpic_ctu_top_left_x[j], sps_subpic_ctu_top_left_y[j], sps_subpic_width_minus1[j], and sps_subpic_height_minus1[j] for each value of j in the range of 0 to sps_num_subpics_minus1, inclusive). (b) The picture referred to by each active entry in RefPicList[0] or RefPicList[1] is an ILRP with the value of sps_num_subpics_minus1 equal to 0.

[0365] According to an embodiment, the decoder may check whether the above constraints are satisfied to achieve bitstream consistency.The consistency check may be performed after invoking the decoding process for generating unavailable reference pictures as described above.

[0366] The embodiments of the present disclosure may include at least one processor and a memory storing computer code. The computer code, when executed by the at least one processor, may be configured to cause the at least one processor to perform the functions of the embodiments of the present disclosure.

[0367] For example, refer to Fig.33, a decoder of the present disclosure may include at least one processor and a memory storing computer code (80). The decoder may be configured to receive a bitstream including at least one coded picture and parameter sets (e.g., SPS and VPS), headers (e.g., picture headers and slice headers), and AU delimiters. The computer code may be configured to cause at least one processor to perform any number of decoding processes as described in the present disclosure (e.g., constructing a reference picture list, generating unavailable reference pictures, and checking bitstream consistency) and aspects related to decoding (e.g., signaling flags and other syntax elements in picture headers, slice headers, and access unit delimiters). For example, the computer code (80) may include a plurality of signaling codes (81) and decoding codes (82).

[0368] The plurality of signaling codes (81) may include various signaling codes configured to cause at least one processor to signal (and / or infer) flags and other syntax elements in picture headers, slice headers, and access unit delimiters.

[0369] The decoding code (82) may be configured to cause at least one processor to decode one or more pictures. According to an embodiment, the decoding code (82) may include a construction code (83), a generation code (84) and a check code (85). The construction code (83) may be configured to cause at least one processor to construct a reference picture list. The generation code (84) may be configured to cause at least one processor to generate an unavailable reference picture in the reference picture list. The check code (85) may be configured to cause at least one processor to check the bitstream consistency of the following constraints for the reference pictures in the reference picture list: (a) the number of entries indicated as being in the reference picture list is not less than the number of active entries indicated as being in the reference picture list, (b) each picture referred to by the active entry in the reference picture list is present in a decoded picture buffer (DPB) and has a temporal identifier value less than or equal to the temporal identifier value of the current picture, and (c) each picture referenced by an entry in the reference picture list is not the current picture, and each picture is indicated by a picture header flag as a potential reference picture.

[0370] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Fig.34 A computer system (900) suitable for implementing embodiments of the disclosed subject matter is shown.

[0371] Computer software may be encoded using any suitable machine code or computer language. Computer software may be subjected to mechanisms such as assembly, compilation, and linking to create code comprising instructions that may be directly executed by a computer central processing unit (CPU), a graphics processing unit (GPU), etc., or executed through interpretation, microcode execution, etc.

[0372] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0373] Fig.34 The components shown in the computer system (900) are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement related to any one or combination of components shown in the exemplary embodiment of the computer system (900).

[0374] The computer system (900) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not shown). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to the following conscious inputs from a person: for example, audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0375] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard (901), mouse (902), touch pad (903), touch screen (910), data glove, joystick (905), microphone (906), scanner (907), camera (908).

[0376] The computer system (900) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback provided by a touch screen (910), a data glove, or a joystick (905), but there may also be tactile feedback devices that are not used as input devices). For example, such devices may be audio output devices (e.g., speakers (909), headphones (not depicted)), visual output devices (e.g., screens (910), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of these screens may be able to output two-dimensional visual output or more than three-dimensional output by means such as stereo output; virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)) and printers (not depicted).

[0377] The computer system (900) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (920) with CD / DVD etc. media (921), thumb drives (922), removable hard disk drives or solid-state drives (923), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), etc.

[0378] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0379] The computer system (900) may also include an interface to one or more communication networks. The network may be, for example, wireless, wired, optical. The network may also be local, wide area, urban, vehicle-mounted and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicle-mounted and industrial networks (including CANBus), etc. Some networks typically require an external network interface adapter (such as, for example, a USB port of the computer system 900) attached to some universal data port or peripheral bus (949); other networks are typically integrated into the core of the computer system 900 by attaching to the system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). The computer system (900) can communicate with other entities by using any of these networks. Such communications may be one-way receive-only (e.g., broadcast television), one-way send-only (e.g., CANbus to certain CANbus devices), or bidirectional, such as to other computer systems using local or wide area digital networks. Such communications may include communications to a cloud computing environment (955). As described above, certain protocols and protocol stacks may be used on each of these networks and network interfaces.

[0380] The above-mentioned human-machine interface device, human-accessible storage device, and network interface ( 954 ) may be attached to the core ( 940 ) of the computer system ( 900 ).

[0381] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (943), hardware accelerators (944) for certain tasks, etc. These devices, along with read-only memory (ROM) (945), random access memory (946), internal mass storage devices (e.g., internal non-user accessible hard drives, SSDs, etc.) (947) may be connected via a system bus (948). In some computer systems, the system bus (948) may be accessed in the form of one or more physical plugs to enable expansion through additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (948) directly or via a peripheral bus (949). The architecture of the peripheral bus includes PCI, USB, etc. A graphics adapter 950 may be included in the core 940.

[0382] The CPU (941), GPU (942), FPGA (943), and accelerator (944) can execute certain instructions, which can be combined to form the aforementioned computer code. The computer code can be stored in ROM (945) or RAM (946). Transient data can also be stored in RAM (946), while permanent data can be stored in, for example, an internal mass storage device (947). Fast storage and retrieval of any storage device in the storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (941), GPUs (942), mass storage devices (947), ROMs (945), RAMs (946), etc.

[0383] The computer readable medium may have computer code thereon, which is used to perform various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0384] As an example and not a limitation, a computer system having an architecture (900), in particular a core (940), can provide functionality by executing software contained in one or more tangible computer-readable media through a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable medium can be a medium associated with the following storage devices: a user-accessible mass storage device as described above, and certain storage devices of the core (940) with non-transient properties, such as a core mass storage device (947) or ROM (945). Software that implements various embodiments of the present disclosure can be stored in such a device and executed by the core (940). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (940), in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform specific processing or specific parts of specific processing described herein, including defining data structures stored in RAM (946) and modifying such data structures according to processing defined by the software. Additionally or alternatively, the computer system may provide functionality in the form of hardwired logic or otherwise represented by circuitry (e.g., accelerator (944)) that may operate in place of or in conjunction with software to perform a particular process or particular portion of a particular process described herein. Reference to software may include logic, and vice versa, where appropriate. Reference to a computer-readable medium may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry implementing logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software. An embodiment of the present disclosure also provides a system for decoding a video stream, the system comprising: a first signaling unit, configured to signal a first flag in an access unit delimiter of an encoded video stream, the first flag indicating whether the access unit includes any one of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; a second signaling unit, configured to signal a second flag in a picture header of the encoded video stream, the second flag indicating whether the picture is an IRAP picture; and a decoding unit, configured to decode the picture into a current picture based on the first flag and the second flag, wherein the value of the first flag is equal to the value of the second flag. In some examples, the system also includes: a third signaling unit configured to signal a third flag in the picture header of the encoded video stream, the third flag indicating whether the picture is the GDR picture, wherein the value of the first flag is equal to the value of the third flag. In some examples, the third flag is signaled based on the second flag indicating that the picture is not the IRAP picture. In some examples, the first flag has a value indicating that the picture is any one of the IRAP picture and the GDR picture, the second flag has a value indicating that the picture is the IRAP picture, and the system also includes a fifth signaling unit, which is configured to signal a fifth flag in a slice header of a slice of the picture of the encoded video stream, the fifth flag indicating whether any picture before the IRAP picture is output. In some examples, the system further includes: a determination unit configured to determine a network abstraction layer (NAL) unit type of the slice, wherein the fifth flag is signaled based on the determined NAL unit type. In some examples, the fifth flag is signaled based on the NAL unit type being determined to be equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT. In some examples, the system also includes: a fourth signaling unit configured to signal a fourth flag in a picture header of the encoded video stream, the fourth flag indicating whether the picture is the GDR picture, wherein the value of the first flag is equal to the value of the fourth flag. In some examples, the fifth flag is signaled based on the second flag indicating that the picture is not the IRAP picture. In some examples, the decoding unit includes: a construction unit configured to construct a reference picture list; a generation unit configured to generate unavailable reference pictures in the reference picture list; and a checking unit configured to check bitstream consistency, for reference pictures in the reference picture list, the following constraints apply: the number of entries indicated in the reference picture list is not less than the number of active entries indicated in the reference picture list, each picture referred to by an active entry in the reference picture list is present in a decoded picture buffer (DPB) and has a temporal identifier value that is less than or equal to the temporal identifier value of the current picture, and Each picture referred to by an entry in the reference picture list is not the current picture and is indicated by a picture header flag as being a potential reference picture. The embodiment of the present disclosure also provides a computer device. The device includes: a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute according to the instructions in the program code: receiving an encoded video stream including an access unit, the access unit including a picture; signaling a first flag in the access unit delimiter of the encoded video stream, the first flag indicating whether the access unit includes any one of an intra-frame random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; signaling a second flag in the picture header of the encoded video stream, the second flag indicating whether the picture is an IRAP picture; based on signaling the first flag and the second flag, decoding the picture into a current picture, wherein the value of the first flag and the value of the second flag are equal.

[0385] Although the present disclosure has described several non-limiting embodiments, there are changes, permutations, and various replacement equivalents that fall within the scope of the present disclosure. It will therefore be appreciated that those skilled in the art will be able to conceive of many systems and methods that, although not explicitly shown or described herein, implement the principles of the present disclosure and are therefore within its spirit and scope.

Claims

1. A method for decoding a video stream, characterized in that: The method comprises: receiving an encoded video stream comprising access units, the access units comprising pictures; signaling a first flag in an access unit delimiter of the coded video stream, the first flag indicating whether the access unit includes any one of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; signaling a second flag in a picture header of the coded video stream, the second flag indicating whether the picture is the IRAP picture; and Decoding the picture into a current picture based on the first flag and the second flag, wherein a value of the first flag is equal to a value of the second flag; The method further comprises: determining a network abstraction layer (NAL) unit type of a slice of a picture of the coded video stream; Signaling a fifth flag based on the determined NAL unit type comprises: The fifth flag is signaled based on the NAL unit type being determined to be equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT, wherein the fifth flag indicates whether any picture before the IRAP picture is output.

2. The method according to claim 1, characterized in that The method further comprises: signaling a third flag in a picture header of the coded video stream, the third flag indicating whether the picture is the GDR picture, The value of the first flag is equal to the value of the third flag.

3. The method according to claim 2, characterized in that Based on the second flag indicating that the picture is not the IRAP picture, the third flag is signaled.

4. The method according to claim 1, characterized in that: the first flag has a value indicating that the picture is any one of the IRAP picture and the GDR picture, The second flag has a value indicating that the picture is the IRAP picture, and The method further comprises signaling the fifth flag in a slice header of the slice.

5. The method according to claim 1, characterized in that The method further comprises: signaling a fourth flag in a picture header of the coded video stream, the fourth flag indicating whether the picture is the GDR picture, The value of the first flag is equal to the value of the fourth flag.

6. The method according to claim 5, characterized in that Based on the second flag indicating that the picture is not the IRAP picture, the fifth flag is signaled.

7. A method for decoding a video stream, characterized in that: The method comprises: receiving an encoded video stream comprising access units, the access units comprising pictures; signaling a first flag in an access unit delimiter of the coded video stream, the first flag indicating whether the access unit includes either or both of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; and signaling a second flag in a picture header of the coded video stream, the second flag indicating whether the picture is the IRAP picture; wherein the value of the first flag is equal to the value of the second flag; The video data is decoded based on: Build a list of reference images; generating an unavailable reference picture in the reference picture list; and Check bitstream conformance. For reference pictures in the reference picture list, the following constraints apply: the number of entries indicated in the reference picture list is not less than the number of active entries indicated in the reference picture list, each picture referred to by an active entry in the reference picture list is present in a decoded picture buffer (DPB) and has a temporal identifier value that is less than or equal to the temporal identifier value of the current picture, and Each picture referred to by an entry in the reference picture list is not the current picture and is indicated by a picture header flag as being a potential reference picture.

8. The method according to claim 7, characterized in that The checking bitstream consistency is performed based on determining that the current picture is an independent decoder refresh (IDR) picture, a clean random access (CRA) picture, or a progressive decoding refresh (GDR) picture.

9. A method for encoding a video stream, characterized in that: The method comprises: receiving video data; encoding the video data into a coded video stream including access units and pictures; Setting a first flag in an access unit delimiter, the first flag indicating whether the access unit includes any one of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; setting a second flag in a picture header of the picture, the second flag indicating whether the picture is the IRAP picture or the GDR picture; and encoding the picture as a current picture based on the first flag and the second flag, wherein a value of the second flag is equal to a value of the first flag; The method further comprises: determining a network abstraction layer (NAL) unit type for a slice of the picture; Setting a fifth flag based on the determined NAL unit type comprises: The fifth flag is set based on the NAL unit type being determined to be equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT, wherein the fifth flag indicates whether any picture before the IRAP picture is output.

10. The method according to claim 9, characterized in that The method further comprises: setting a third flag in the picture header, the third flag indicating whether the picture is the GDR picture, The value of the first flag is equal to the value of the third flag.

11. The method according to claim 10, characterized in that Based on the second flag indicating that the picture is not the IRAP picture, the third flag is set.

12. The method according to claim 9, characterized in that the first flag has a value indicating that the picture is any one of the IRAP picture and the GDR picture, The second flag has a value indicating that the picture is the IRAP picture, and The method further comprises: setting the fifth flag in a slice header of the slice.

13. The method according to claim 9, characterized in that The method further comprises: setting a fourth flag in the picture header, the fourth flag indicating whether the picture is the GDR picture, The value of the first flag is equal to the value of the fourth flag.

14. The method according to claim 13, characterized in that Based on the second flag indicating that the picture is not the IRAP picture, the fifth flag is set.

15. A method for encoding a video stream, characterized in that: The method comprises: receiving video data; encoding the video data into a coded video stream including access units and pictures; signaling a first flag in an access unit delimiter of the coded video stream, the first flag indicating whether the access unit includes either or both of an intra random access point (IRAP) picture and a progressive decoding refresh (GDR) picture; and signaling a second flag in a picture header of the coded video stream, the second flag indicating whether the picture is the IRAP picture; wherein the value of the first flag is equal to the value of the second flag; The video data is encoded based on: Build a list of reference images; generating an unavailable reference picture in the reference picture list; and Check bitstream conformance. For reference pictures in the reference picture list, the following constraints apply: the number of entries indicated in the reference picture list is not less than the number of active entries indicated in the reference picture list, each picture referred to by an active entry in the reference picture list is present in a decoded picture buffer (DPB) and has a temporal identifier value that is less than or equal to the temporal identifier value of the current picture, and Each picture referred to by an entry in the reference picture list is not the current picture and is indicated by a picture header flag as being a potential reference picture.

16. The method according to claim 15, characterized in that The checking bitstream consistency is performed based on determining that the current picture is an independent decoder refresh (IDR) picture, a clean random access (CRA) picture, or a progressive decoding refresh (GDR) picture.

17. A video stream decoding device, characterized in that: include: A memory, used for storing program codes and transmitting the program codes to the processor; A processor, configured to execute the method according to any one of claims 1 to 8 according to the instructions in the program code.

18. A video stream encoding device, characterized in that: include: A memory, used for storing program codes and transmitting the program codes to the processor; A processor, configured to execute the method according to any one of claims 9 to 16 according to the instructions in the program code.

19. A non-transitory computer-readable storage medium, characterized in that: It stores a bit stream obtained by a method according to any one of claims 9-16.

20. A method for processing a video bitstream, characterized in that: The video bit stream is generated according to the encoding method according to any one of claims 9 to 16, or is decoded based on the decoding method according to any one of claims 1 to 8.

Images