Method for referencing set of parameters in encoded video bitstream

Abstract

FIELD: video encoding and decoding.SUBSTANCE: invention relates to references to sets of parameters and to their scopes of action in the bitstream of encoded video. Video data is received that comprises a coded video sequence (CVS), a network abstraction layer (NAL) block of a picture header (PH) included in the picture header is obtained, at least one coded slice NAL block corresponding to the PH NAL block is obtained, a picture parameter set (PPS) included in the PPS NAL block is obtained from the coded video sequence, wherein said PPS NAL block is referenced by said PH NAL block and the coded slice NAL block and this PPS NAL block includes a first identifier of the PPS NAL block level that is greater than the value of a second identifier of the network abstraction layer (NAL) block level of a sequence parameter set (SPS), a sequence parameter set (SPS) included in the SPS NAL block of a coded video sequence of a level (CLVS) obtained from the CVS is obtained, wherein the SPS NAL block that includes the SPS is directly referenced by the PPS NAL block and the value of the parameter_set_id syntax of the SPS is the same in all SPSs that are referenced by the CLVS in CVS, and decoding the coded image based on the PPS NAL unit and the SPS NAL unit.EFFECT: increasing video encoding efficiency.20 cl, 28 dwg, 1 tbl

Description

Cross-references to related applications

[0001] This application is separated from the Russian Federation patent application No. 2021129261 / 07, filed on October 7, 2021, which is the national stage transfer of the international application PCT / US 2020 / 055757, filed on October 15, 2020, which refers to the priority of the provisional patent application US No. 62 / 954099, filed on December 27, 2019, and the patent application US No. 17 / 063060, filed on October 5, 2020, and the mentioned applications are fully incorporated herein by reference. Technical Field

[0002] The present invention relates to video coding and video decoding, namely, to references to parameter sets and their scopes in a bitstream of encoded video.Background of the invention

[0003] Video coding and decoding methods using inter-image prediction with motion compensation exist. Uncompressed digital video may contain a sequence of images, each of which has a given spatial dimension, for example, 1920x1080 luminance samples and corresponding chrominance samples. Such an image sequence may have a fixed or variable image rate (informally also called a frame rate), equal to, for example, 60 images per second, or 60 Hz. Uncompressed video places high demands on the bit rate. For example, 1080p60 4:2:0 video with 8-bit sample depth (resolution of 1920x1080 luminance samples at a frame rate of 60 Hz) requires a bandwidth of approximately 1.5 Gbps. An hour of such video requires over 600 GB of storage.

[0004] One of the goals of video coding and decoding is to reduce redundancy in the input video signal through compression. Compression can reduce bandwidth or storage requirements by up to two orders of magnitude or more in some cases. Both lossless and lossy compression, as well as combinations of both, can be used. Lossless compression refers to methods that allow an exact copy of the original signal to be reconstructed from the compressed signal. When lossy compression is used, the reconstructed signal may not be identical to the original, but the discrepancy between the original and reconstructed signals is small enough that the reconstructed signal is suitable for the intended use. Lossy compression is widely used for video. The amount of acceptable distortion depends on the specific application; for example, users of commercial streaming applications may be more tolerant of distortion than users of broadcast applications.The degree of compression is subject to the following rule: the greater the permissible distortion, the greater the achievable degree of compression.

[0005] A video encoder and video decoder may apply a number of techniques under different categories, including, for example, motion compensation, transforms, quantization, and entropy coding, some of which will be discussed below.

[0006] Historically, video encoders and decoders have worked with a predetermined picture size, which in most cases was known and remained constant throughout a coded video sequence (CVS), a Group of Pictures (GOP), or similar time frames containing multiple pictures. For example, the MPEG-2 standard is designed to allow for changes in horizontal resolution (and therefore picture size) based on certain factors, such as activity in a video scene, but only within I-pictures, and therefore typically only within a GOP. ITU-T Rec. H.263 Annex P defines resampling of reference pictures to allow for the use of different resolutions within a CVS.However, in this case, only the size of the reference images changed, while the size of the images themselves remained unchanged. This resulted in the ability to use only parts of the entire image canvas (with lower resolution) or capture only part of the scene (with higher resolution). Also, H.263 Annex Q allows upsampling and downsampling of individual macroblocks by a power of two (along any axis). However, again, the image size remains unchanged. The macroblock size in H.263 is fixed and therefore does not need to be signaled.

[0007] In modern video coding, changes in the size of predicted pictures are more common. For example, the VP9 standard allows resampling of reference pictures and changing the resolution of the picture as a whole. Similarly, corresponding proposals have been made for the VVC standard (including, for example, the Joint Video Experts Group document JVET-M0135-v1, “On Adaptive Resolution Change (ARC) for VVC”, January 9-19, 2019, Hendry et al., incorporated herein by reference in its entirety), allowing resampling of reference pictures as a whole, to different, higher or lower, resolutions. The mentioned document proposes encoding different candidate (possible) resolutions in a set of sequence parameters and referencing them in syntax elements, for each picture, in the set of picture parameters. Summary of the invention

[0008] In one embodiment of the present invention, a method is proposed for decoding a bitstream of an encoded video using at least one processor, comprising obtaining an encoded video sequence from the bitstream of the encoded video; obtaining an image block from the encoded video sequence; obtaining a network abstraction layer (NAL) block of a picture header (PH) contained in said image block; obtaining at least one NAL block of a coded slice contained in said image block;decoding an encoded picture based on said PH NAL unit, at least one coded slice NAL unit, a picture parameter set (PPS) contained in the PPS NAL unit obtained from the coded video sequence, and a sequence parameter set (SPS) contained in the SPS NAL unit obtained from the coded video sequence; and outputting the decoded picture, wherein said SPS NAL unit is available to said at least one processor before said PPS NAL unit, and said PPS NAL unit is available to said at least one processor before said PH NAL unit and at least one coded slice NAL unit.

[0009] In one embodiment of the present invention, a device is proposed for decoding a bitstream of an encoded video, including at least one memory configured to store a program code; at least one processor configured to read the computer code and to perform operations according to the instructions of the program code, wherein the program code includes: a first acquisition code configured to ensure that said at least one processor obtains an encoded video sequence from the bitstream of the encoded video; a second acquisition code configured to ensure that said at least one processor obtains an image block from the encoded video sequence;a third acquisition code configured to ensure that said at least one processor obtains a network abstraction layer (NAL) unit of a picture header (PH) contained in said image block; a fourth acquisition code configured to ensure that said at least one processor obtains a coded slice NAL unit contained in said image block; a decoding code configured to ensure that said at least one processor decodes an encoded image based on said PH NAL unit, at least one coded slice NAL unit, a picture parameter set (PPS) contained in a PPS NAL unit obtained from an encoded video sequence, and a sequence parameter set (SPS) contained in an SPS NAL unit obtained from an encoded video sequence;and an output code configured to provide output by said at least one processor of a decoded image, wherein said SPS NAL unit is available to said at least one processor before said PPS NAL unit, and said PPS NAL unit is available to said at least one processor before said PH NAL unit and at least one coded slice NAL unit.

[0010] In one embodiment of the present invention, a computer-readable medium is provided that stores instructions comprising: one or more instructions that, when executed by one or more processors of a device for decoding an encoded video bitstream, cause said one or more processors to perform the following: obtaining an encoded video sequence from the encoded video bitstream; obtaining an image block from the encoded video sequence; obtaining a network abstraction layer (NAL) block of a picture header (PH) contained in said image block; obtaining at least one NAL block of a coded slice contained in said image block;decoding an encoded image based on said PH NAL unit, at least one coded slice NAL unit, a picture parameter set (PPS) contained in the PPS NAL unit obtained from the coded video sequence, and a sequence parameter set (SPS) contained in the SPS NAL unit obtained from the coded video sequence; and outputting the decoded image, wherein said SPS NAL unit is available to said one or more processors before said PPS NAL unit, and said PPS NAL unit is available to said one or more processors before said PH NAL unit and at least one coded slice NAL unit. Brief description of the drawings;

[0011] Further features, nature and various advantages of the proposed invention can be understood in more detail with the help of the following detailed description and the attached drawings, where:

[0012] Fig. 1 shows a schematic diagram of a simplified block diagram of a communication system in accordance with one embodiment of the present invention;

[0013] Fig. 2 shows a schematic diagram of a simplified block diagram of a communication system in accordance with one embodiment of the present invention;

[0014] Fig. 1 shows a schematic diagram of a simplified block diagram of a decoder in accordance with one embodiment of the present invention;

[0015] Fig. 4 is a schematic diagram of a simplified block diagram of an encoder in accordance with one embodiment of the present invention;

[0016] Fig. 5A-5E schematically illustrate embodiments of signaling ARC parameters in accordance with one embodiment of the present invention;

[0017] Fig. 6A-6B schematically illustrate examples of syntax tables according to one embodiment of the present invention;

[0018] Fig. 7 shows an example of a prediction structure for adaptive resolution scaling according to one embodiment of the present invention;

[0019] Fig. 8 shows an example of a syntax table according to one embodiment of the present invention;

[0020] Fig. 9 is a schematic diagram of a simplified block diagram of the analysis and decoding of the ROS cycle for each access block and each value of the access block sequence number, in accordance with one embodiment of the present invention;

[0021] Fig. 10 is a schematic illustration of a structure of a video bitstream including multi-level sub-images in accordance with one embodiment of the present invention;

[0022] Fig. 11 is a schematic illustration of a display of a selected sub-image with increased resolution in accordance with one embodiment of the present invention;

[0023] Fig. 12 shows a flow chart of a procedure for decoding and displaying a video bitstream including multi-level sub-images, in accordance with one embodiment of the present invention;

[0024] Fig. 13 is a schematic illustration of a display of a 360-degree video with a sub-image enhancement layer in accordance with one embodiment of the present invention;

[0025] Fig. 14 shows an example of sub-image arrangement information and a corresponding layer and image prediction structure according to one embodiment of the present invention;

[0026] Fig. 15 shows an example of sub-image arrangement information and the corresponding layer and image prediction structure, with the possibility of spatial scaling of a local area, in accordance with one embodiment of the present invention;

[0027] Fig. 16A-16B show examples of syntax tables for sub-image arrangement information, according to one embodiment of the present invention;

[0028] Fig. 17 shows examples of syntax tables of an SEI message for sub-image arrangement information, according to one embodiment of the present invention;

[0029] Fig. 18 shows an example of a syntax table for specifying output levels and profile / tier / standard level information for each set of output levels, according to one embodiment of the present invention;

[0030] Fig. 19 shows an example of a syntax table for indicating the activation of the output layer mode for each set of output layers, according to one embodiment of the present invention;

[0031] Fig. 20 shows an example of a syntax table for indicating the presence of a sub-image for each level, for each set of output levels, in accordance with one embodiment of the present invention;

[0032] Fig. 21 shows a flow chart of an example of a procedure for decoding a bitstream of an encoded video in accordance with one embodiment of the present invention;

[0033] Fig. 22 is a schematic illustration of a computer system in accordance with one embodiment of the present invention. Detailed description of the invention

[0034] Fig. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present invention. The system (100) may include at least two terminals (110-120) connected to each other by a network (150). In unidirectional data transmission, the first terminal (110) may encode video data locally for transmission to the second terminal (120) over the network (150). The second terminal (120) may receive encoded video data from the first terminal from the network (150), decode the encoded data, and display the reconstructed video data. Unidirectional data transmission is widely used in media service applications and similar applications.

[0035] Fig. 1 illustrates a second pair of terminals (130, 140) configured to support bidirectional transmission of encoded video, which may be required, for example, during videoconferencing. During bidirectional data transmission, both terminals (130, 140) can encode locally captured video data for transmission to another terminal over a network (150). Both terminals (130, 140) can also receive encoded video data transmitted by the other terminal, can decode the encoded data, and display the reconstructed video data on a local display device.

[0036] In the example of Fig. 1, the terminals (110-140) may be servers, personal computers or smartphones, but the concept of the present invention is not limited to them. Embodiments of the present invention may be used in laptop computers, tablet computers, media players and / or specialized videoconferencing equipment. The network (150) may be any number of networks transmitting encoded video data between the terminals (110-140), including, for example, wired and / or wireless communication networks. The communication network (150) may provide data exchange over circuit-switched and / or packet-switched communication lines. Examples of such networks may be telecommunication networks, local area networks, wide area networks and / or the Internet. In the present description, the architecture and topology of the network (150) do not play any role in the operation of the proposed invention unless explicitly stated.

[0037] Fig. 2 illustrates, as an example of the application of the present invention, the placement of a video encoder and decoder in a streaming environment. The proposed invention can be applied with equal effectiveness in other areas where video is used, including, for example, videoconferencing, digital television, storage of compressed video on digital media, including CDs, DVDs, memory cards, etc.

[0038] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a video camera, which generates a stream (202) of uncompressed samples. The stream (202) of samples, shown in bold to emphasize the larger amount of data compared to encoded video streams, may be processed by an encoder (203) associated with the camera (201). The encoder (203) may include hardware, software, or a combination thereof that enable aspects of the proposed invention to be implemented, in accordance with the following more detailed description. The bitstream (204) of the encoded video, shown in thin to emphasize the smaller amount of data compared to the uncompressed stream of video samples, may be stored on the streaming server (205) for subsequent use.One or more streaming clients (206, 208) may access a streaming server (205) to obtain copies (207, 209) of an encoded video bitstream (204). A client device (206) may include a video decoder (210) that decodes a received copy of the encoded video bitstream (207) and generates an output stream (211) of video samples that may be displayed on a display (212) or other display device (not shown in the drawing). In some streaming systems, the video bitstreams (204, 207, 209) may be encoded in accordance with specified video coding / video compression standards. An example of such standards may be ITU-T Recommendation H.265. A video coding standard that is informally referred to as Versatile Video Coding (VVC) is currently being developed. The invention described in this document can be applied in the context of the VVC standard.

[0039] Fig. 3 shows a functional block diagram of a video decoder (210) according to one embodiment of the present invention.

[0040] The receiver (310) can receive one or more encoded video sequences for decoding by the decoder (210). In the same embodiment or in alternative embodiments of the present invention, the reception of video sequences can be performed one by one, wherein the decoding of each of the encoded video sequences is independent of the decoding of the remaining video sequences. The encoded video sequence can be received from the channel (312), which can be a hardware and / or software communication line with a memory device where the encoded video data is stored. The receiver (310) can receive the encoded video data together with other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to the corresponding elements using them (not shown in the drawing). The receiver (310) can separate the encoded video sequence from the remaining data.To combat network jitter, a buffer memory (315) can be installed between the receiver (310) and the entropy decoder / analyzer (320) (hereinafter referred to as the "analyzer"). When the receiver (310) receives data from a storage or transmission device with sufficient bandwidth and controllability, or from a network with isosynchronous transmission, the buffer (315) may not be used or have a small volume. In the case of using packet networks with non-guaranteed delivery, such as the Internet, the buffer (315) is necessary, can be relatively large and, preferably, have an adaptive size.

[0041] The video decoder (210) may include an analyzer (320) for recovering symbols (321) from the entropy-coded video sequence. The types of these symbols may include, for example, information used to control the operation of the decoder (210), and also, possibly, information for controlling a display device, such as a display (212), which may be associated with the decoder, but is not an integral part of it, as shown in Fig. 3. The control information for display devices may, for example, have the form of Supplementary Enhancement Information (SEI) messages or fragments of Video Usability Information (VUI) parameter sets (not shown in the drawing). The analyzer (320) may perform analysis / entropy decoding of the received encoded video sequence.The encoded video sequence may be encoded in accordance with some video coding technology or standard, and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, context-dependent or context-independent arithmetic coding, and the like. The analyzer (320) may extract from the encoded video sequence a set of subgroup parameters for at least one subgroup of pixels in the video decoder, based on at least one parameter corresponding to the group. Subgroups may include Groups of Picture_s (GOP), pictures, subpictures, tiles, slices, macroblocks, Coding Units (CU), blocks, bricks, Transform Units (TU), Prediction Units (PU), and the like.A tile may be a rectangular region of a CU / CTU block in a given column and row of tiles in an image. A "brick" may be a rectangular region of CU / CTU rows within a tile. A slice may be one or more image bricks contained in a NAL unit. A subpicture may be a rectangular region of one or more slices in an image. The analyzer may also extract information such as transform coefficients, quantizer parameter values, motion vectors, and the like from the encoded video sequence.

[0042] The analyzer (320) can perform entropy decoding / analysis operations on the video sequence received from the buffer (315) and generate symbols (321).

[0043] When restoring symbols (321), various device units may be used depending on the type of encoded video images or parts thereof (e.g., intra- and inter-predicted images, intra- and inter-predicted blocks), as well as other factors. Which device units will be used and how may be determined by the subgroup control information extracted from the encoded video sequence by the analyzer (320). For simplicity, the flow of such subgroup control information between the analyzer (320) and the plurality of device units described below is not shown in the drawing.

[0044] In addition to the functional blocks already mentioned, decoder 210 can be conceptually subdivided into a set of functional blocks described below. In practical implementations used in commercial settings, many of these blocks closely interact with each other and can be, at least partially, mutually integrated. However, for the purposes of describing the proposed invention, the subdivision into functional blocks described below is suitable.

[0045] The first of such blocks may be the scaling / inverse transform block (351). The scaling / inverse transform block (351) receives quantized transform coefficients, as well as control information, including which transform to use, block size, quantization coefficient, quantization scaling matrices, etc., in the form of symbols (321) from the analyzer (320). It outputs blocks containing sample values ​​that can be input into the aggregator (355).

[0046] In some cases, the samples at the output from the scaling / inverse transform unit (351) may be of the intra-coded block type; that is, blocks for which prediction information from previously reconstructed images is not used, but may use information from previously reconstructed portions of the current image. Such prediction information may be provided by the intra-image prediction unit (352). In some cases, the intra-image prediction unit (352) may form blocks of the same size and shape as the block being reconstructed, using already reconstructed information of its surroundings obtained from the current (partially reconstructed) image in the current image memory (358).The aggregator (355) may, in some cases, add the prediction information generated by the intra prediction unit (352) to the output sample information provided by the scaling / inverse transform unit (351), individually for each sample.

[0047] In other cases, the samples at the output of the scaling / inverse transform block (351) may relate to inter-prediction, possibly with motion compensation. In such cases, the motion-compensated prediction block (353) may access the reference image memory (357) and obtain the samples used for prediction. After motion compensation, the obtained samples, in accordance with the symbols (321) related to this block, may be added by the aggregator (355) to the output data of the scaling / inverse transform block (in this case, they are called difference samples or a difference signal), thereby forming the output sample information.The addresses in the reference image memory from which the motion-compensated prediction unit obtains predicted samples may be determined by motion vectors accessible to the motion-compensated prediction unit in the form of symbols (321), which may have, for example, an X-component, a Y-component, and a reference image component. Motion compensation may also include interpolation of sample values ​​obtained from the reference image memory (457), when motion vectors, motion vector prediction mechanisms, etc., with subpixel accuracy are used.

[0048] The samples at the output of the aggregator (355) may be processed using various loop filtering methods in the loop filtering unit (356). Video compression technologies may include in-loop filtering technologies that are controlled by parameters contained in the encoded video bitstream and provided to the loop filtering unit (356) in the form of symbols (321) from the analyzer (320). They may also depend on metainformation obtained during decoding of preceding (in decoding order) portions of the encoded image or encoded video sequence, as well as on previously reconstructed and loop-filtered sample values.

[0049] The output data of the loop filtering unit (356) may be a stream of samples that is fed to the display device (212) and also stored in the reference image memory for use in future external image prediction.

[0050] Individual encoded images, after their full restoration, can be used as reference images for future prediction. After the encoded image is fully restored, and if it has been determined to be a reference image (e.g., by the analyzer (320)), the current reference image (358) can be placed in the reference image buffer (357), and new memory for the current images can be allocated before the next encoded image is restored.

[0051] Video decoder 210 may perform decoding operations in accordance with a predetermined video compression technology, which may be documented in a standard, such as ITU-T Recommendation H.265. The encoded video sequence may satisfy the syntax defined by the applicable video compression technology or standard, in the sense that it satisfies the syntax defined in the document or standard of the video compression technology, and in particular, the syntax of the specified profiles of the standard. At the same time, in order to comply with some of the video compression technologies or standards, the complexity of the encoded video sequence must be within the limitations determined by the level of the video compression technology or standard. In some cases, the standard levels limit the maximum image size, the maximum frame rate, the maximum sample rate (measured, for example, in millions of samples per second), the maximum size of the reference image, etc.n. The restrictions imposed by the levels can in some cases be further limited by the specifications of the Hypothetical Reference Decoder (HRD) and the metadata for controlling the HRD decoder buffer, signaled in the encoded video sequence.

[0052] In one embodiment of the present invention, the receiver (310) may receive additional (redundant) data along with the encoded video. This additional data may be a component of the encoded video sequence (or video sequences). The additional data may be used by the video decoder (210) to correctly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0053] Fig. 4 shows a functional block diagram of a video encoder (203) according to one embodiment of the present invention.

[0054] The encoder (203) can receive video samples from a video source (201) (which is not part of the encoder), capturing video images for encoding by the encoder (203).

[0055] The video source (201) may supply a source video sequence for encoding by the video encoder (203) in the form of a digital stream of video samples having any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601 Y CrCb, RGB, ...) and any suitable report structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (201) may be a storage device storing pre-prepared video. In a videoconferencing system, the video source (203) may be a video camera that captures image information locally in the form of a video sequence. The video data may be in the form of a plurality of individual images that convey a sense of motion when viewed sequentially.The images themselves can be organized as a spatial matrix of pixels, where each pixel can include one or more samples, depending on the sample structure used, the color space, etc. Those skilled in the art will understand the relationship between pixels and samples. Further in this description, samples will be discussed.

[0056] According to one embodiment of the present invention, the video encoder (203) can encode and compress images of the original video sequence into the encoded video sequence (443) in real time, or in accordance with other time constraints imposed by practical application. One of the functions of the controller (450) can be to ensure a suitable encoding rate. The controller controls the remaining functional blocks described below and is functionally connected with these blocks. This connection is not shown in the drawing for simplicity. The parameters set by the controller can include parameters related to rate control (picture skipping, quantizer, X value for rate-distortion optimization methods, ...), picture size, group of pictures (GOP) arrangement, maximum search range of motion vectors, etc.Other functions of the controller (450) will be apparent to those skilled in the art, as they may correspond to a video encoder (203) optimized for a particular system design.

[0057] Some video encoders operate in a configuration that those skilled in the art call a "coding loop." In very simplified terms, the coding loop may consist of an encoding engine in the encoder (430) (hereinafter referred to as the "source encoder"), responsible for generating symbols based on the input images to be encoded, as well as reference images, and a (local) decoder (433), built into the encoder (203) and restoring the symbols, generating sample data that would be identically generated by the (remote) decoder (since the compression of symbols into an encoded video bitstream is lossless compression in the video compression technologies considered in the present invention). This reconstructed sample stream is input into a reference image memory (434).Since decoding a symbol stream yields the same bit-perfect results regardless of the decoder (local or remote), the contents of the reference picture buffer are also bit-perfectly identical in both the local and remote encoder. In other words, the prediction subsystem in the encoder "sees" as reference picture samples exactly the same sample values ​​that the decoder would see using prediction during decoding. This fundamental principle of reference picture synchronicity (and the resulting drift if synchronicity cannot be ensured, for example, due to channel errors) should be well known to those skilled in the art.

[0058] The operation of the "local" decoder (433) is essentially identical to the "remote" decoder (210), which has already been described in detail above in connection with Fig. 3. However, returning to Fig. 4, since the symbols are available, and the encoding / decoding of the symbols into the encoded video sequence by the entropy encoder (445) and the analyzer (320) can be performed without loss, the local decoder (433) may not fully implement the entropy decoding subsystems of the decoder (210), including the channel (312), the receiver (310), the buffer (315), and the analyzer (320).

[0059] It should be noted here that any decoding technology other than analysis / entropy decoding present in a decoder must be present in a substantially identical functional form in the corresponding encoder. For this reason, the description of the present invention focuses on the operation of the decoder. A description of the encoding technologies may be omitted, as they may be inverse to the decoding technologies described in detail. Only in a few places is a more detailed description necessary, and this will be provided below.

[0060] Among its operations, the source encoder (430) may perform motion-compensated predictive coding, in which input frames are predictively encoded based on one or more previously encoded frames of the video sequence, which have been marked as "reference frames." Thus, the encoding engine (432) encodes differences between blocks of pixels in the input frame and blocks of pixels in the reference frame (or frames), which can be selected as reference for predicting the input frame.

[0061] The local video decoder (433) can decode the encoded video data of the frames marked as reference frames, depending on the symbols generated by the source encoder (430). The operations of the encoding machine (432) are preferably lossy data processing. When the encoded video data is decoded in the video decoder (not shown in Fig. 4), the reconstructed video sequence is usually a replica of the original video sequence with some errors. The local video decoder (433) exactly reproduces the decoding process that could be performed by the remote video decoder on the reference frames and places the reconstructed reference frames in the reference picture cache (434). In this way, the encoder (203) can locally store copies of the reconstructed reference frames, the contents of which match the reconstructed reference frames received by the remote video decoder (in the absence of transmission errors).

[0062] The predictor (435) can search for predictions for the coding engine (432). That is, for a new frame to be encoded, the predictor (435) can search the reference image memory (434) to find sample data (as candidate reference pixel blocks) or metadata, such as motion vectors of reference images, block shapes, etc., which can serve as reference for new images. The predictor (435) can find suitable reference data for each individual pixel block. In some cases, depending on the search results obtained by the predictor (435), the reference data for predicting the input image can be extracted from multiple reference images stored in the reference image memory (434).

[0063] The controller (450) may control encoding operations in the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[0064] The output data of all the above-described functional blocks can be entropy-encoded in the entropy encoder (445). The entropy encoder converts the symbols generated by various functional blocks into an encoded video sequence by losslessly compressing these symbols in accordance with technologies known to those skilled in the art, such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0065] The transmitter (440) can buffer the encoded video sequence (or video sequences) generated by the entropy encoder (445) to prepare it for transmission over the communication channel (460), which can be a hardware and / or software communication line with a storage device where the encoded video data is stored. The transmitter (440) can combine the encoded video data from the video encoder (430) with other transmitted data, for example, streams of encoded audio data and / or service data (their sources are not shown in the drawing).

[0066] The controller (450) can control the operation of the encoder (203). During encoding, the controller (450) can assign each encoded image a certain encoded image type, which can influence the encoding methods applied to it. For example, images can be assigned one of the frame types described below.

[0067] An intra-predicted picture (I-picture) is a picture that is encoded and decoded without using any other frames in the video sequence as a source for prediction. Some video codecs support various types of intra-predicted pictures, such as Independent Decoder Refresh (IDR) pictures. Those skilled in the art should be aware of these types of I-pictures, as well as their properties and applicability.

[0068] A predictable image (P-image) is an image that can be encoded and decoded using intra-prediction or inter-prediction using at most one motion vector and a pointer to a reference image to predict the sample values ​​of each block.

[0069] A bidirectionally predicted image (B-image) is an image that can be encoded and decoded using intra- or inter-prediction using a maximum of two motion vectors and reference image pointers to predict the sample values ​​of each block. Similarly, in the case of multidirectionally predicted images, more than two reference images and corresponding metadata can be used to reconstruct a single block.

[0070] Source images are typically spatially partitioned into multiple blocks (e.g., 4x4, 8x8, 4x8, or 16x16 sample blocks) and encoded block by block. Blocks can be predictively encoded based on other (already encoded) blocks, depending on the coding types assigned to the images corresponding to these blocks. For example, blocks in I-pictures can be encoded without prediction or with prediction based on already encoded blocks of the same image (“spatial prediction” or “intra prediction”). Blocks of pixels in P-pictures can be encoded without prediction, using spatial prediction, or using temporal prediction based on one previously encoded reference image. Blocks in B-pictures can be encoded without prediction, using spatial prediction, or using temporal prediction based on one or two previously encoded reference images.

[0071] The video encoder (203) may perform encoding operations in accordance with a predetermined video coding technology or standard, which may be documented in a standard such as ITU-T Recommendation H.265. During operation, the video encoder (203) may perform various compression operations, including predictive encoding operations that utilize temporal and spatial redundancy in the input video sequence. The encoded video data may accordingly satisfy the syntax specified by the video coding technology or standard used.

[0072] In one embodiment of the present invention, the transmitter (440) may transmit additional data along with the encoded video. The video encoder (430), for example, may provide such data as a fragment of the encoded video sequence. The additional data may include data of temporal, spatial, or SNR enhancement layers, redundant pictures or slices, supplemental enhancement information (SEI) messages, or fragments of video usability information (VUI) parameter sets, etc.

[0073] Recently, more attention has been paid to the aggregation or extraction of multiple semantically independent image parts into a compressed form into a single video image. In particular, in the context of, for example, 360-degree coding for some video surveillance applications, multiple semantically independent source images (e.g., six cube surfaces in a cubic projection of a 360-degree video scene, or inputs from individual cameras in a multi-camera video surveillance system) may require independent adaptive resolution parameters to adapt to different scene activity at any given time. In other words, encoders, at any given time, must be able to use different resampling rates for the different semantically independent images that make up a 360-degree video scene or the video surveillance scene as a whole.Combining into a single image, in turn, requires resampling of the reference images and the presence of adaptive resolution coding signaling for parts of the coded image.

[0074] Below, several terms used later in this description will be introduced.

[0075] A sub-image, in some cases, can be understood as a rectangular structure of samples, blocks, macroblocks, coding units, or similar elements, grouped semantically, which can be coded independently at different resolutions. One or more sub-images can form an image. One or more coded sub-images can form a coded image. One or more sub-images can be combined into an image, and one or more sub-images can be extracted from the image. In some embodiments of the present invention, one or more sub-images can be combined in compressed form, without sample-level re-coding, into a single coded image, and in this case, as well as in other cases, one or more coded sub-images can be extracted from the coded image in compressed form.

[0076] Adaptive Resolution Change (ARC) is a mechanism that allows for changing the resolution of an image or subimage within an encoded video sequence, for example, by resampling reference images. From here on, ARC parameters are the control information required to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, output and / or reference image resolutions, various control flags, etc.

[0077] In one embodiment of the present invention, encoding and decoding may be performed on a single, semantically independent encoded video image. Before describing the essence of encoding / decoding multiple sub-images with independent ARC parameters and the additional complications they introduce, we will consider options for signaling ARC parameters.

[0078] FIGS. 5A-5E illustrate several embodiments of ARC parameter signaling. As noted in each embodiment, they have their own advantages and disadvantages in terms of coding efficiency and complexity, as well as from an architectural point of view. In a video coding standard or technology, one or more of the proposed embodiments of ARC parameter signaling, as well as embodiments known in the existing art, may be selected. These embodiments are not necessarily mutually exclusive and may be interchangeable depending on the application requirements, the technologies and standards used, and the choices made by the encoder.

[0079] ARC parameter classes may include the following:

[0080] - upsampling or downsampling coefficients, separately for the X and Y axes, or in combination;

[0081] - upsampling or downsampling coefficients, with the addition of a time dimension indicating a constant rate of increase / decrease in scaling for a given number of images;

[0082] - either of the previous two may include encoding one or more as short as possible syntactic elements that may point to a table containing the mentioned coefficients;

[0083] - the resolution, along the X or Y axis, measured in samples, blocks, macroblocks, coding units (CUs), or any other suitable precision, for the input image, output image, reference image, or encoded image, in combination or individually; If more than one resolution is present (e.g., one for the input image and one for the reference image), then in some cases one set of values ​​may be calculated based on the other set of values. This can be controlled, for example, using special flags. A more detailed example will be discussed below.

[0084] - warping coordinates similar to those used in H.263 Annex P, again, as mentioned above, with suitable precision. H.263 Annex P defines one efficient method for encoding warping coordinates, but more efficient methods could potentially be proposed. For example, the reversible coding, such as Huffman coding, with a variable-length codeword of the warping coordinates in Annex P could be replaced by binary coding with a codeword of an appropriate length, which could, for example, be calculated based on the maximum image size, possibly multiplied by a coefficient and offset by a specified value to allow "warping" beyond the boundaries defined by the maximum image size.

[0085] —up- or downsampling filter parameters. In embodiments of the present invention, only one filter may be present for up- and / or downsampling. However, in embodiments of the present invention, it is desirable to provide greater flexibility in filter design, and this may require signaling of the filter parameters. Such parameters may be selected using an index in a list of possible filter designs, the filter may be fully specified (e.g., using a list of filter coefficients, using suitable entropy encoding methods), the filter may be implicitly selected using up- or downsampling coefficients, which, in turn, are signaled in accordance with the mechanisms described above, etc.

[0086] In the following description, we assume the encoding of a finite set of upsampling or downsampling coefficients (the same coefficient is used for both the X-axis and Y-axis), specified using a codeword. Such a codeword can have a variable length, for example, using Ext-Golomb coding for the corresponding syntactic elements in video coding standards such as H.264 and H.265. One possible correspondence table between the values ​​of upsampling and downsampling coefficients is illustrated in Table 1.

[0087] Multiple similar lookup tables can be proposed, depending on application requirements and the capabilities of upsampling or downsampling mechanisms available in a standard or video coding technology. The table can be expanded to include more values. Values ​​can also be represented using coding mechanisms other than Ext-Golomb codes, for example, using binary coding. This can provide certain advantages when upsampling factors are used outside of the video processing subsystems themselves (primarily in the encoder and decoder), for example, in MANE. It should be noted that in cases where resolution changes are not required, a short Ext-Golomb code, such as the one in the table above, which is only one bit long, can be selected. This allows for an advantage in coding efficiency, in general, compared to using binary codes.

[0088] The number of table entries, as well as their semantics, may be partially or fully configurable. For example, the basic framework of a table may be conveyed in a high-level parameter set, such as a sequence parameter set or a decoder parameter set. In embodiments of the present invention, multiple such tables may be defined in a video coding technology or standard, and selected, for example, using a decoder parameter set or sequence.

[0089] Below, we will discuss how upsampling or downsampling coefficients (ARC information), encoded as described above, can be incorporated into the syntax of a video coding technology or standard. Similar considerations can be applied to one or more codewords controlling upsampling or downsampling filters. The case where filters or other data structures require relatively large amounts of data will be discussed below.

[0090] According to the illustration of Fig. 5A, Annex P of the H.263 specification (H.263 Annex P) includes ARC information (502) in the form of four warp coordinates in the image header (501), namely, in the PLUSPTYPE header extension (503) of the H.263 standard. This seems to be a rational solution for cases where a) the image header is available, and b) frequent changes in the ARC information are expected. However, the amount of service information when applying the signaling type proposed in H.263 can be quite high, and the scaling factors between image boundaries may be inconsistent, since image headers are not necessarily stored for a long time.

[0091] According to the illustration of Fig. 5B, the JVCET-M135-v1 specification includes ARC reference information (505) (pointer) located in the sequence parameter set (504), which points to a table (506) including target resolutions, which, in turn, is located inside the sequence parameter set (507). The placement of available resolutions in the table (506) in the sequence parameter set (507) (507), according to the authors, is due to the use of the SPS set as a compatibility agreement point in the exchange of capability information. The resolution can vary within the range specified by the values ​​of the table (506) from image to image by referring to the appropriate image parameter set (504).

[0092] Figs. 5C-5E show possible embodiments of transmitting ARC information in the encoded video bitstream proposed in the present invention. Each of these embodiments has advantages over the implementation embodiments discussed above. The proposed embodiments may be present simultaneously in a video coding technology or standard.

[0093] In embodiments of the present invention, such as the embodiment shown in Fig. 5C, the ARC information (509), such as an oversampling (scaling) factor, may be present in a slice header, a GOP header, a tile header, or a tile group header. Fig. 5C illustrates an embodiment of the present invention that uses a tile group header (508). This may be an adequate solution if the ARC information is small in size, such as a single variable-length ue(v) value or a fixed-length codeword of a few bits, as shown above. Placing the ARC information directly in the tile group header has the additional advantage that the ARC information can be applied to a sub-picture represented by, for example, this tile group, and not only to the entire picture. This will be described in more detail below.Furthermore, even if a video coding technology or standard only allows for adaptive resolution change for images as a whole (as opposed to, for example, adaptive resolution change at the tile group level), placing ARC information in the tile group header, as opposed to placing it in the image header, like H.263, provides certain advantages in terms of error resilience.

[0094] In embodiments of the present invention, for example, in the embodiment shown in Fig. 5D, the ARC information (512) can be directly represented in a corresponding parameter set, for example, in a picture parameter set, a header parameter set, a tile parameter set, an adaptation parameter set, etc. Fig. 5D shows an embodiment of the present invention, in which an adaptation parameter set (511) is used. The scope of this parameter set may preferably be no larger than a picture, for example, it can be a group of tiles. The use of ARC information is specified implicitly by activating the corresponding parameter set. For example, if only ARC at the picture level is defined in a video coding technology or standard, then a picture parameter set or an equivalent set may be suitable.

[0095] In embodiments of the present invention, such as the embodiment shown in Fig. 5E, the ARC reference information (513) may be present in the tile group header (514) or in a similar data structure. Such reference information (513) may reference a subset of the ARC information (515) available in a parameter set (516) with a scope exceeding an individual picture, such as in a sequence parameter set or a decoder parameter set.

[0096] The additional layer of implicit activation of the PPS set from the tile group header, PPS, SPS, used in JVET-M0135-v1 seems redundant, since image parameter sets, like sequence parameter sets, can (and should in some standards, such as RFC3984) be used for interoperability agreements or declarations. However, if the ARC information should also apply to a subimage, such as that represented by a tile group, a better choice might be a parameter set with a scope limited to the tile group, such as an adaptation parameter set or a header parameter set.Also, if the ARC information is non-negligible in size, such as containing filter control information, i.e., multiple filter coefficients, then from a coding efficiency standpoint, the parameter may be a better solution than using the header (508) directly, since these settings can be reused for future images or sub-images by referencing the same set of parameters.

[0097] When using a sequence parameter set or higher-level parameter sets with a scope spanning multiple images, the following considerations may be relevant:

[0098] 1. The parameter set for storing the table (516) with the ARC information may in some cases be a sequence parameter set, but in other cases, it is preferably a decoder parameter set. The scope of the decoder parameter set may cover several CVSs, namely, an encoded video stream, i.e., all the bits of the encoded video, from the beginning to the end of the session. Such a scope may be more suitable since the ARC coefficients can be programmed in the decoder, and possibly implemented in hardware, so that the hardware parameters remain largely constant within a single CVS (which, at least in some multimedia systems, is a group of pictures of one second or less in length). In this case, placing the table in a sequence parameter set is explicitly listed among the placement options discussed in this document, namely, in connection with paragraph 2 below.

[0099] 2. The ARC reference information (513) can be preferably placed directly in the picture / slice / tile / GOP / tile group header, such as the tile group header (514), instead of the picture parameter set as in JVCET-M0135-v1. For example, when an encoder needs to change one value in the picture parameter set, such as the ARC reference information, it needs to create a new PPS and reference this new PPS. Suppose that only the ARC reference information is changed, and the other information, such as the quantization matrix information, in the PPS remains unchanged. This information may be quite large, and it needs to be retransmitted to make the new PPS complete. Since the ARC reference information (513) can be a single codeword, such as a table pointer, and this is the only value that changes, it is inconvenient and wasteful to retransmit all the quantization matrix information.Therefore, from a coding efficiency perspective, it would be significantly better to eliminate the indirect reference via the PPS set proposed in JVET-M0135-v1. Also, placing the ARC reference information in the PPS set has the additional disadvantage that the ARC information referenced by the (513) ARC reference information may apply to the entire image, rather than to sub-images, since the scope of the image parameter set is the image as a whole.

[0100] In this or an alternative embodiment of the present invention, the signaling of the ARC parameters may correspond to the detailed example shown in Figs. 6A-6B. Figs. 6A-6B show syntax trees represented using a notation roughly corresponding to the C programming language, which, for example, has been used in video coding standards since at least 1993. Lines in bold indicate syntactic elements present in the bitstream. Lines not in bold generally indicate control commands or the assignment of variable values.

[0101] Referring to the illustration of Fig. 6A, the header (601) of a tile group, as an example of the syntactic structure of a header applicable to a (possibly rectangular) portion of an image, may conditionally contain a variable-length, Exp-Golomb-encoded syntax element dec_pic_size_idx (602) (highlighted in bold). The presence of this syntax element in the header of a tile group may be controlled depending on the use of adaptive resolution (603). In this example, the flag value is not highlighted in bold, meaning that the flag is present in the bitstream at the time it appears in the syntax diagram. Whether adaptive resolution is applied to a given image, or a portion thereof, may be signaled in any high-level syntax structure inside or outside the bitstream. In the example considered, this is signaled in the sequence parameter set, as will be shown below.

[0102] Fig. 6B shows a fragment of the sequence parameter set (610). The first syntax element shown is the adaptive_pic_resolution_change_flag flag (611). When it has the value "TRUE", the flag indicates the use of adaptive resolution, which, in turn, may require corresponding control information. In this example, such control information is included conditionally depending on the flag value, using the if() statement in the parameter set (612) and in the tile group header (601).

[0103] When adaptive resolution is applied, in this example, the output resolution is encoded in samples (613). The position 613 refers to both output_pic_width_in_luma_samples and output_pic_width_in_luma_samples, which together define the resolution of the output image. A specific video coding technology or standard may elsewhere define some restrictions on any of these values. For example, the standard-level resolution may limit the total number of output samples, which may be equal to the product of the values ​​of these two syntax elements. Also, specific video coding technologies or standards, or external technologies or standards, such as system standards, may limit the numerical range (for example, one or both dimensions must be a multiple of a power of two) or the aspect ratio (for example, the height and width must have a specified ratio, such as 4:3 or 16:9).Such limitations may be imposed by hardware implementation or other reasons and should be known to those skilled in the art.

[0104] In some applications, it is desirable for the encoder to instruct the decoder to use a specific image size, rather than implicitly specifying the output image size. In this example, the reference_pic_size_present_flag (614) specifies the conditional presence of the reference image dimensions (615) (again, the notations shown refer to both the width and height).

[0105] Finally, a table of possible values ​​for the width and height of the decoded image is shown. Such a table can be referenced, for example, using a pointer (616) to a table, (num_dec_pic_size_in_luma_samples_minus1). "Minus 1" can indicate the interpretation of the value of this syntax element. For example, if the encoded value is zero, there is only one table entry. If the value is five, there are six table entries. For each "row" of the table, the width and height of the decoded image are then included in the syntax structure (617).

[0106] The table entries designated by (617) can be referenced by the syntax element (602) dec_pic_size_idx in the tile group header, which allows for different decoded image sizes, essentially a scaling effect, in each individual tile group.

[0107] Some video coding technologies or standards, such as VP9, ​​support spatial scaling by performing some form of reference image resampling (the corresponding signaling differs significantly from that proposed in the present invention) in combination with temporal scaling to enable spatial scalability. Specifically, individual reference images can be upsampled to higher resolutions using methods such as ARC, forming the basis of a spatial enhancement layer. The upsampled images can be refined using standard high-resolution prediction mechanisms, allowing for greater detail.

[0108] The proposed embodiments of the present invention can be applied in such an environment. In some cases, in the same or an alternative embodiment of the present invention, a value in the NAL unit header, for example, the Temporal ID field, can be used to indicate not only the temporal but also the spatial layer. This can be advantageous in some systems; for example, for environments where scalability is applied, existing Selected Forwarding Units (SFUs) can be used without modification; they are created and optimized for selectively forwarding temporal layers depending on the Temporal ID value in the NAL unit header. To utilize this functionality, a correspondence between the coded picture size and the temporal layer referenced by the Temporal ID field in the NAL unit header may be necessary.

[0109] In some video coding technologies, an Access Unit (AU) may refer to coded images, slices, tiles, NAL units, etc., that were captured and composed into the corresponding image / slice / tile / NAL unit bitstream at a given point in time. Such a point in time could be, for example, composition time.

[0110] In the HEVC standard and some other video coding technologies, the picture order count (POC) value can be used to indicate a selected reference picture among multiple reference pictures stored in the decoded picture buffer (DPB). When an access unit (AU) contains one or more pictures, slices, or tiles, each picture, slice, or tile belonging to the same AU can have the same POC value, which indicates that they were formed based on content with the same composition time. In other words, if two pictures / slices / tiles contain the same specified POC value, these two pictures / slices / tiles belong to the same AU and have the same composition time.Conversely, two images / slices / tiles with different POC values ​​indicate that these images / slices / tiles belong to different access blocks and have different composition times.

[0111] In embodiments of the present invention, this strict requirement may be relaxed, i.e., an access unit may include pictures, slices, or tiles with different POC values. By allowing different POC values ​​within a single access unit, the POC value can be used to identify potential independently decodable pictures / slices / tiles with the same display time. This, in turn, allows for support of multiple zoom levels without changing the reference picture selection signaling, such as the reference picture set signaling or reference picture list signaling, described in more detail below.

[0112] However, it is still desirable to be able to identify the access block to which an image / slice / tile belongs among other images / slices / tiles with different POC values, based solely on the POC value. This can be achieved as described below.

[0113] In embodiments of the present invention, the access unit count (AUC) may be signaled in a high-level syntax structure, such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or access unit delimiter. The AUC value may be used to determine which NAL units, pictures, slices, or tiles belong to a given access unit. The AUC value may correspond to a specific point in time of the composition. The AUC value may be a multiple of the POC value. The AUC value may be calculated by dividing the POC value by an integer value. In some embodiments, division operations may impose a significant load on the decoder. In such cases, a slight limitation on the number space for AUC values ​​allows replacing the division operation with a shift operation.For example, the AUC value may be equal to the value of the Most Significant Bit (MSB) in the range of POC values.

[0114] In the embodiments of the present invention, the POC cycle value for an access unit (poc_cycle_au) may be signaled in a top-level syntax structure, such as in a NAL unit header, a slice header, a tile group header, an SEI message, a parameter set, or an access unit delimiter. The poc_cycle_au value may indicate how many consecutive different POC values ​​may be associated with one access unit. For example, if the poc_cycle_au value is 4, pictures, slices, or tiles with a POC value of 0-3, inclusive, may belong to an access unit with an AUC value of 0, and pictures, slices, or tiles with a POC value of 4-7, inclusive, may belong to an access unit with an AUC value of 1. Therefore, the AUC value can be calculated by dividing the POC value by the poc_cycle_au value.

[0115] In embodiments of the present invention, the poc_cyle_au value may be calculated based on information located, for example, in a video parameter set (VPS), which defines the number of spatial or SNR levels in the encoded video sequence. One example of such a potential correspondence is briefly described below. The calculations described above allow saving several bits in the VPS, and therefore increasing the coding efficiency, however, in some embodiments of the present invention, poc_cycle_au may be explicitly encoded in the corresponding upper-level syntax structure hierarchically subordinate to the video parameter set, which allows minimizing poc_cycle_au for a given small portion of the bitstream, for example, for an image.This optimization allows saving more bits than the calculation procedure described above, since the values ​​of the POC and / or the values ​​of syntactic elements indirectly based on the POC can be encoded in lower-level syntactic structures.

[0116] Fig. 8 shows an example of syntax tables, according to embodiments of the present invention, for signaling the vps_poc_cycle_au syntax element in a VPS (or SPS), which indicates the poc_cycle_au value used for all pictures / slices in a coded video sequence in a slice header. If the POC value increases uniformly in each access unit, vps_contant_poc_cycle_per_au in the VPS may be set to 1, and vps_poc_cycle_au may be signaled in the VPS. In this case, slice_poc_cycle_au may not be explicitly signaled, and the AUC value for each access unit may be calculated by dividing the POC value by vps_poc_cycle_au. If the POC value does not increase uniformly in each access unit, vps_contant_poc_cycle_per_au in the VPS may be set to 0.In this case, vps_access_unit_cnt may not be signaled, while slice_access_unit_cnt may be signaled in the slice header for each image slice. Each image slice may have a different slice_access_unit_cnt value. The AUC value for each access unit can be calculated by dividing the POC value by slice_poc_cycle_au.

[0117] Fig. 9 is a flowchart illustrating an example of the above-discussed procedure. For example, in operation S910, a VPS (or SPS) analysis may be performed, and in operation S920, it may be determined whether the POC cycle in each access unit is constant within the coded video sequence. If the POC cycle in each access unit is constant (Yes, in operation S920), then in operation S930, a value of the access unit sequence number for a specific access unit may be calculated based on the poc_cycle_au signaled for the coded video sequence and the POC value of this specific access unit. If the POC cycle in each access unit is not constant (No, in operation S920), then in operation S940, a value of the access unit sequence number for a specific access unit may be calculated based on the poc_cycle_au signaled at the picture level and the POC value of this specific access unit. In operation S950, a new VPS (or SPS) set can be analyzed.

[0118] In embodiments of the present invention, although the POC values ​​in images, slices, or tiles may be different, these images, slices, or tiles corresponding to an access unit with the same AUC values ​​may be related to the same decoding or output time. Therefore, in the absence of any analysis or decoding dependencies between images, slices, or tiles in one access unit, all images, slices, or tiles related to one access unit, or a subset thereof, can be decoded in parallel and can be output at the same time.

[0119] In embodiments of the present invention, although the POC values ​​in images, slices, or tiles may be different, these images, slices, or tiles corresponding to an access unit with the same AUC values ​​may belong to the same composition / display time. When the composition time is contained in a container format, even if images correspond to different access units but have the same composition time, these images can be displayed at the same time.

[0120] In embodiments of the present invention, each image, slice, or tile may have the same temporal identifier (temporal_id) in a single access unit. All images, slices, or tiles corresponding to a single moment in time, or a subset thereof, may correspond to a single temporal sublevel. In embodiments of the present invention, all images, slices, or tiles may have the same or different spatial layer identifiers (layer_id) in a single access packet. All images, slices, or tiles corresponding to a single moment in time, or a subset thereof, may correspond to a single or different spatial levels.

[0121] Fig. 7 shows an example of the structure of a video sequence with a combination of temporal_id, layer_id, POC, and AUC values ​​during adaptive resolution change. In this example, a picture, slice, or tile in the first access unit with AUC=0 may have temporal_id=0 and layer_id=0 or 1, while a picture, slice, or tile in the second access unit with AUC=1 may have temporal_id=1 and layer_id=0 or 1, respectively. The POC value is increased by 1 in each picture regardless of the values ​​of temporal_id and layer_id. In this example, the poc_cycle_au value may be equal to 2. In embodiments of the present invention, the poc_cycle_au value may be set equal to the number of levels (spatial scaling). In this example, accordingly, the POC value is increased by 2, while the AUC value is increased by 1.

[0122] In the above embodiments of the present invention, the entire inter-layer prediction and reference picture indication structure, or a subset thereof, may be supported using the existing reference picture set (RPS) signaling in HEVC or using the reference picture list (RPL) signaling. In the RPS or RPL, the indication of the selected reference picture may be performed by signaling the POC value or the POC difference value between the current picture and the selected reference picture. In one embodiment of the present invention, the RPS and RPL may be used to indicate the inter-picture prediction or inter-layer prediction structure without changing the signaling, but with the limitations described below.If the temporal_id of a reference image is greater than the temporal_id of the current image, this reference image cannot be used for motion compensation in the current image or for any other prediction. If the layer_id of a reference image is greater than the layer_id of the current image, this reference image cannot be used for motion compensation in the current image or for any other prediction.

[0123] In embodiments of the present invention, scaling motion vectors based on the difference in POC for temporal motion vector prediction may be prohibited between different images within an access unit. Therefore, although each value may have different POC values ​​within an access unit, the motion vector is not scaled and is not used for temporal motion vector prediction within the access unit. This is necessary because a reference image with a different POC in the same access unit is considered a reference image having the same time instance. Accordingly, in this embodiment of the present invention, the motion vector scaling function may return 1 when the reference image belongs to the access unit to which the current image belongs.

[0124] In embodiments of the present invention, motion vector scaling based on the POC difference for temporal motion vector prediction may optionally be disabled over several images when the spatial resolution of the reference image differs from the spatial resolution of the current image. When motion vector scaling is enabled, it is performed based on both the POC difference and the spatial resolution ratio between the current image and the reference image.

[0125] In embodiments of the present invention, motion vector scaling may be performed based on the AUC difference, instead of the POC difference, for temporal prediction of motion vectors, particularly when poc_cycle_au has non-uniform values ​​(e.g., when vps_contant_poc_cycle_per_au=0). Otherwise (e.g., when vps_contant_poc_cycle_per_au=1), motion vector scaling based on the AUC difference may be identical to motion vector scaling based on the POC difference.

[0126] In embodiments of the present invention, when a motion vector is scaled based on an AUC difference, a reference motion vector in the same access block (with the same AUC value) as the current image is not scaled based on the AUC difference and is used to predict motion vectors without scaling or with scaling based on the spatial resolution ratio between the current image and the reference image.

[0127] In embodiments of the present invention, the AUC value can be used to determine the boundaries of an access unit, as well as for the operation of a hypothetical reference decoder (HRD), which requires input and output time values ​​with access unit accuracy. In embodiments of the present invention, the decoded image in the upper layer of the access unit can be output for display. The AUC value and the layer_id value can be used to identify the output image.

[0128] In embodiments of the present invention, an image may include one or more sub-images. Each sub-image may cover a local region or the entire image region. The region occupied by a sub-image may or may not overlap with regions of other sub-images. The region covered by one or more sub-images may cover either the entire image region or a portion thereof. If an image includes a sub-image, the region occupied by the sub-image may be identical to the region occupied by the image.

[0129] In embodiments of the present invention, a sub-image may be encoded using a coding method similar to the coding method used for the encoded image. A sub-image may be encoded independently or coding dependent on another sub-image or the encoded image. A sub-image may be either dependent, with respect to its parsing, on another sub-image or the encoded image, or independent.

[0130] In embodiments of the present invention, an encoded sub-image may be contained in one or more layers. The encoded sub-images in different layers may have different spatial resolutions. The original sub-image may be spatially resampled (e.g., upsampled or downsampled), encoded using different spatial resolution parameters, and included in a bitstream corresponding to one of the layers.

[0131] In embodiments of the present invention, a sub-image with parameters (W, H), where W is the width of the image and H is the height of the sub-image, respectively, may be encoded and included in the encoded bitstream corresponding to level 0, while a sub-image with increased (or decreased) resolution, obtained based on the sub-image with the original spatial resolution, with parameters (W*S w,k , H*S h,k), can be encoded and included in the encoded bit stream corresponding to level k, where S w,k , S h,k - horizontal and vertical resampling coefficients. If the value of S w,k , S h,k greater than 1, the resampling will be upward. Whereas if the values ​​of S w,k , S h,k less than 1, resampling will be downsampling.

[0132] In embodiments of the present invention, an encoded image in one of the layers may have a different visual quality from an encoded sub-image in another layer for the same sub-image or a different sub-image. For example, sub-image i in layer n may be encoded with a parameter Q i,n quantization, while subimage j in level m can be encoded with parameter Q j , m quantization.

[0133] In embodiments of the present invention, an encoded sub-picture in one of the layers can be decoded independently, without any analysis or decoding dependency on encoded sub-pictures in other layers for the same local area. A sub-picture layer that can be independently decoded without reference to other sub-picture layers of the same local area can be called an independent sub-picture layer. An encoded sub-picture in an independent sub-picture layer may or may not have a decoding or analysis dependency on previously encoded sub-pictures in the same sub-picture layer; however, an encoded sub-picture cannot have any dependencies on encoded pictures in another sub-picture layer.

[0134] In embodiments of the present invention, an encoded sub-picture in one of the layers can be decoded dependently, with any dependencies, analysis or decoding, on encoded sub-pictures in other layers for the same local area. A sub-picture layer that can be decoded dependently, with references to other sub-picture layers of the same local area, can be called a dependent sub-picture layer. An encoded sub-picture in a dependent sub-picture can reference an encoded sub-picture belonging to the same sub-picture, a previously encoded sub-picture in the same sub-picture layer, or both of these reference sub-pictures.

[0135] In embodiments of the present invention, an encoded sub-picture may include one or more independent sub-picture layers and one or more dependent sub-picture layers. However, at least one independent sub-picture layer must be present in each encoded sub-picture. The layer identifier (layer_id), which may be located in the NAL unit header or other high-level syntax structure, may be equal to 0 for an independent sub-picture layer. A sub-picture layer with a layer_id equal to 0 may be a base sub-picture layer.

[0136] In embodiments of the present invention, an image may include one or more background sub-images and one foreground sub-image. The area covered by a background sub-image may be equal to the image area. The area occupied by a foreground sub-image may overlap with the area covered by a background sub-image. The background sub-image may be a base layer of the sub-image, while the foreground sub-image may be a non-base (enhancement) layer of the sub-image. One or more non-base layers of the sub-image may reference the same base layer for decoding. Each non-base layer of the sub-image with a layer_id equal to a may reference a non-base layer of the sub-image with a layer_id equal to b, where a is greater than b.

[0137] In embodiments of the present invention, an image may include one or more foreground subimages, with or without background subimages. Each subimage may have its own base subimage layer and one or more non-base (enhancement) layers. Each base subimage layer may be referenced by one or more non-base subimage layers. Each non-base subimage layer with a layer_id equal to a may reference a non-base subimage layer with a layer_id equal to b, where a is greater than b.

[0138] In embodiments of the present invention, an image may include one or more foreground sub-images, with or without background sub-images. Each encoded sub-image in a (base or non-base) sub-image layer may be referenced by one or more non-base sub-image layers belonging to the same sub-image, and one or more non-base sub-image layers not belonging to the same sub-image.

[0139] In embodiments of the present invention, an image may include one or more foreground sub-images, with or without background sub-images. A sub-image in one of the layers may be further split into multiple sub-images in the same layer. One or more encoded sub-images in layer b may reference a split sub-image in layer a.

[0140] In embodiments of the present invention, a coded video sequence (CVS) may be a group of coded pictures. The CVS may include one or more coded sub-picture sequences (CSPS), where a CSPS may be a group of coded sub-pictures covering the same local image region. The CSPS may have the same or different temporal resolution as the coded video sequence.

[0141] In embodiments of the present invention, a CSPS may be encoded and included in one or more layers. A CSPS may include one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS allows for the reconstruction of a sequence of sub-images corresponding to the same local region.

[0142] In embodiments of the present invention, the number of CSPS levels corresponding to a CSPS may be equal to or different from the number of CSPS levels corresponding to another CSPS.

[0143] In embodiments of the present invention, a CSPS layer may have a temporal resolution (e.g., frame rate) different from another CSPS layer. The original (uncompressed) sub-picture sequence may be temporally resampled (e.g., up- or down-sampled), encoded using different temporal resolution parameters, and included in a bitstream corresponding to one of the layers.

[0144] In embodiments of the present invention, a sub-picture sequence with a frame rate of F may be encoded and included in an encoded bitstream corresponding to level 0, while a sub-picture sequence with increased (or decreased) temporal resolution, compared to the original sub-picture sequence, with F*S t,k , can be encoded and included in the encoded bitstream corresponding to level k, where S t,k - the coefficient of temporal resampling for level k. If the value of S t,k greater than 1, the temporal resampling procedure can be a frame rate upsampling conversion. Whereas if the value of S t,k less than 1, the temporal resampling procedure may be a frame rate downsampling conversion.

[0145] In embodiments of the present invention, when a sub-picture in a CSPS layer is referenced by a sub-picture from layer b, for motion compensation or any inter-layer prediction, if the spatial resolution of this CSPS layer differs from the spatial resolution of CSPS layer b, then the decoded pixels in the CSPS layer are resampled and used as reference pixels. The resampling procedure may use up-sampling or down-sampling filtering.

[0146] Fig. 10 shows an example of a video stream including a background video CSPS with a layer_id of 0 and multiple foreground CSPS layers. While an encoded subimage may include one or more CSPS layers, a background region that does not belong to any of the foreground CSPS layers may include a base layer. The base layer may include the background region and the foreground region, while an enhancement CSPS layer may include the foreground region. The enhancement CSPS layer may have higher visual quality than the base layer in the same region. The enhancement CSPS layer may reference the reconstructed pixels and motion vectors of the base layer corresponding to the same region.

[0147] In embodiments of the present invention, a video bitstream corresponding to a base layer is contained in one track in a video file, while the CSPS layers corresponding to each sub-picture are contained in a separate track.

[0148] In embodiments of the present invention, the video bitstream corresponding to a layer is contained in a single track, while the CSPS layers with the same layer_id are contained in a separate track. In this example, the track corresponding to layer k includes only the CSPS layers corresponding to layer k.

[0149] In embodiments of the present invention, each CSPS level of each sub-image is stored in a separate track. Each track may or may not have an analysis or decoding dependency on one or more other tracks.

[0150] In embodiments of the present invention, each track may contain bitstreams corresponding to levels i to j of the CSPS levels of all sub-images, or a subset thereof, where 0 <i=<j=<k, а к верхний уровень CSPS.

[0151] In embodiments of the present invention, an image comprises corresponding media data, including one or more of the following: a depth map, an alpha channel value map, 3D geometry data, an occupancy map, etc. These time-stamped media data can be divided into one or more sub-data streams, each of which corresponds to one sub-image.

[0152] Fig. 11 shows an example of videoconferencing based on the multi-level sub-image method. The video stream contains one base-layer video bitstream corresponding to the background image and one or more enhancement-layer video bitstreams corresponding to the foreground sub-images. Each enhancement-layer video bitstream may correspond to a CSPS level. By default, the display shows an image corresponding to the base layer. It contains images of one or more users placed within the image using the picture-in-picture (PIP) method. When the client selects a specific user, the enhancement CSPS level corresponding to the selected user and having increased quality or spatial resolution can be decoded and displayed.

[0153] Fig. 12 is a flowchart illustrating an example of the above-discussed procedure. For example, in operation S1210, a video bitstream with a plurality of layers may be decoded. In operation S1220, a background region and one or more background sub-pictures may be determined. In operation S1230, it may be determined whether a specific region of a sub-picture, for example, one of the foreground sub-pictures, is selected. If a specific region of a sub-picture is selected (Yes, in operation S1240), an enhanced sub-picture may be decoded and displayed. If a specific region of a sub-picture is not selected (No, in operation S1240), the background region may be decoded and displayed.

[0154] ​​In one embodiment of the present invention, network intermediate equipment (e.g., a router) can select a subset of layers to transmit to a user based on its bandwidth. To adapt to bandwidth, images can be divided into sub-images. For example, if the user lacks bandwidth, the router can discard layers or select sub-images based on their importance or the applicable configuration, and this can be done dynamically to adapt to the bandwidth.

[0155] Fig. 13 shows an embodiment of the present invention related to an application scenario in the case of 360-degree video. When a spherical 360-degree image, for example, image 1310, is projected onto a flat image, the projected 360-degree image can be split into a plurality of sub-images as a base layer. For example, this plurality of sub-images can include a back sub-image, an upper sub-image, a right sub-image, a left sub-image, a front sub-image, and a bottom sub-image. An enhancement layer for a specific sub-image, for example, the front sub-image, can be encoded and transmitted to the client. The decoder can be capable of decoding both the base layer, including all sub-images, and the enhancement layer of a selected sub-image.When the current viewport is identical to the selected subimage, the displayed image may have higher quality due to the decoded subimage in the enhancement layer. Otherwise, the decoded base-layer image may be displayed at lower quality.

[0156] In embodiments of the present invention, display arrangement information may be present in the file as auxiliary information (e.g., an SEI message or metadata). One or more decoded sub-pictures may be moved and displayed depending on the signaled arrangement information. The arrangement information may be signaled by a streaming server or a broadcasting station, or may be re-generated by a network entity or a cloud server, or may be determined using user-selectable settings.

[0157] In embodiments of the present invention, when an input image is divided into one or more (rectangular) sub-regions, each such sub-region may be encoded as an independent layer. Each independent layer corresponding to a local region may have a unique layer_id value. For each independent layer, the sub-image size and sub-image location information may be signaled. For example, the image size (width, height), offset information relative to the upper left corner (x_offset, y_offset). Fig. 14 shows an example of an arrangement of sub-images, the corresponding sub-image size and position information, and the corresponding image prediction structure. The arrangement information, including the sub-image sizes and positions, may be signaled in a high-level syntax structure, such as parameter sets, a slice or tile group header, or in an SEI message.

[0158] In embodiments of the present invention, each sub-picture corresponding to an independent layer may have a unique POC value within an access block. When a reference picture among the pictures stored in the DPB is referenced using a syntax element(s) in the RPS or RPL structure, the POC value(s) of each sub-picture corresponding to one of the layers may be used.

[0159] In the embodiments of the present invention, to indicate the (inter-layer) prediction structure, the layer_id may not be used, but the POC (difference) value may be used.

[0160] In embodiments of the present invention, a sub-image with a POC value of N corresponding to a level (or local region) may (but need not) be used as a reference image for a sub-image with a POC value of N+K corresponding to the same level (or the same local region) for motion-compensated prediction. In most cases, the value of K may be equal to the maximum number of (independent) levels, which can coincide with the number of sub-regions.

[0161] Fig. 15 illustrates the case of Fig. 14 in more detail, according to embodiments of the present invention. When an input image is divided into multiple subregions (e.g., four), each local region can be encoded using one or more layers. In this case, the number of independent layers may be equal to the number of subregions, and each subregion will correspond to one or more layers. Thus, each subregion can be encoded using one or more independent layers and zero or more dependent layers.

[0162] In the embodiments of the present invention, according to Fig. 15, the input image can be divided into four sub-regions. As an example, the upper-right sub-region can be encoded as two layers, layer 1 and layer 4, while the lower-right sub-region can be encoded as two layers, layer 3 and layer 5. In such a case, layer 4 can refer to layer 1 for motion-compensated prediction, while layer 5 can refer to layer 3 for motion-compensated prediction.

[0163] In embodiments of the present invention, in-loop filtering (e.g., deblocking filtering, adaptive in-loop filtering, shape restoration, bilateral filtering, or any deep learning-based filtering) with layer boundary crossing may be (optionally) prohibited.

[0164] In embodiments of the present invention, motion-compensated prediction or intra-block copying with intersection of block boundaries may be (optionally) prohibited.

[0165] In embodiments of the present invention, boundary padding or in-loop filtering at sub-image boundaries may be optionally performed. A flag indicating whether or not to perform boundary padding may be signaled in a high-level syntax structure, such as a parameter set (or sets) (VPS, SPS, PPS, or APS), a slice or tile group header, or an SEI message.

[0166] In embodiments of the present invention, sub-region (or sub-picture) arrangement information may be signaled in a VPS or an SPS. Fig. 16A shows an example of syntax elements in a VPS, and Fig. 16B shows an example of syntax elements in an SPS. In this example, the vps_sub_picture_dividing_flag flag is signaled in the VPS. This flag may indicate whether the input picture (or pictures) is divided into a plurality of sub-regions or not. When the value of the vps_sub_picture_dividing_flag flag is 0, the input picture (or pictures) in the coded video sequence (or sequences) corresponding to the current VPS cannot be divided into a plurality of sub-regions. In this case, the input image size can be equal to the encoded image size, (pic_width_in_luma_samples, pic_height_in_luma_samples), which is signaled in the SPS set.When the value of the vps_sub_pic_ture__dividing_flag flag is 1, the input image (or images) can be divided into multiple subregions. In this case, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples signal in the VPS set. The values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples can be equal to the width and height of the input image (or images), respectively.

[0167] In the embodiments of the present invention, the values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples cannot be used for decoding, but can be used for layout and display.

[0168] In embodiments of the present invention, when the value of the vps_sub_picture_dividing_flag flag is 1, the syntax elements pic_offset_x and pic_offset_y may be signaled in the SPS corresponding to a specific level (or levels). In this case, the coded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to the specified level. The position (pic_offset_x, pic_offset_y) of the upper-left corner of the sub-region may also be signaled in the SPS.

[0169] In the embodiments of the present invention, the information (pic_offset_x, pic_offset_y) about the position of the upper left corner of the sub-region cannot be used for decoding, but can be used for layout and display.

[0170] In embodiments of the present invention, information on the arrangement (size and position) of all subregions or a subset of subregions of an input image (or images) and information on the dependencies between layers may be signaled in a parameter set or in an SEI message. Fig. 17 shows examples of syntax elements for indicating information on the arrangement of subregions, on the dependencies between layers, and on the relationship between subregions and one or more layers. In this example, the num_sub_region syntax element indicates the number of (rectangular) subregions in the current coded video sequence. The num_layers syntax element indicates the number of layers in the current coded video sequence. The num_layers value may be greater than or equal to the num_sub_region value. When all subregions are coded as single layers, the num_layers value may be equal to the num_sub_region value.When one or more subregions are encoded as multiple levels, the value of num_layers must be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] specifies the dependence of the ith level on the jth level. The value of num_layers_for_region[i] specifies the number of levels belonging to the ith subregion. The value of sub_region_layer_id[i][j] specifies the layer_id of the jth level belonging to the ith subregion. The values ​​of sub_region_offset_x[i] and sub_region_offset_y[i] specify the horizontal and vertical locations of the upper-left corner of the ith subregion, respectively. The values ​​of sub_region_width[i] and sub_region_height[i] specify the width and height of the ith subregion, respectively.

[0171] In embodiments of the present invention, one or more syntax elements that define an output layer set for indicating one or more layers to be output, with or without information on the level, tier, and standard profile, may be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS set, or in an SEI message. According to the illustration of FIG. 18, the num_output_layer_sets syntax element indicating the number of an output layer set (OLS) in an encoded video sequence referring to the VPS may be signaled in the VPS. For each output layer set, the output_layer_flag flag may be signaled as many times as there are output layers.

[0172] In embodiments of the present invention, the output_layer_flag[i] flag, equal to 1, specifies that the i-th layer should be output. The vps_output_layer_flag[i] flag, equal to 0, specifies that the i-th layer should not be output.

[0173] In embodiments of the present invention, one or more syntax elements that define the level, tier, and profile information for each set of output layers may be signaled in a high-level syntax structure, for example, a VPS, DPS, SPS, PPS, APS set, or in an SEI message. Also, according to the illustration of FIG. 18, a num_profile_tile_level syntax element indicating the number of the level, tier, and profile information for each OLS in the coded video sequence referring to the VPS set may be signaled in the VPS. For each set of output layers, a set of syntax elements for the level, tier, and profile information or a pointer pointing to the level, tier, and profile information among the entries in the level, tier, and profile information may be signaled (as many times as there are output layers).

[0174] In embodiments of the present invention, the value profile_tier_level_idx[i][j] specifies a pointer, in the list of profile_tier_level() syntax structures in the VPS set, to a profile_tier_level() syntax structure applicable to the j-th level of the i-th OLS set.

[0175] In the embodiments of the present invention, according to the illustration of Fig. 19, the syntax element num_profile_tile_level and / or num_output_layer_sets may be signaled when the maximum number of layers is greater than 1 (vps_max_layers_minus1>0).

[0176] In the embodiments of the present invention, according to the illustration of Fig. 19, the syntax element vps_output_layers_mode[i] indicating the signaling mode of the output layer for the i-th output layer may be present in the VPS set.

[0177] In one embodiment of the present invention, a vps_output_layers_mode[i] value of 0 specifies that only the top layer is output using the i-th set of output layers. A vps_output_layer_mode[i] value of 1 specifies that all layers are output using the i-th set of output layers. A vps_output_layer_mode[i] value of 2 specifies that the output layers are the layers with the vps_output_layer_flag[i][j] flag equal to 1, using the i-th set of output layers. More values ​​may be reserved.

[0178] In embodiments of the present invention, output_layer_flag[i][j] may be signaled or not signaled depending on the value of vps_output_layers_mode[i] for the i-th set of output layers.

[0179] In the embodiments of the present invention, according to the illustration of Fig. 19, the flag vps_ptl_signal_flag[i] may be present for the i-th set of output levels. Depending on the value of vps_ptl_signal_flag[i], the level, tier, and profile information of the standard for the i-th set of output levels may or may not be signaled.

[0180] In embodiments of the present invention, according to the illustration of Fig. 20, the number of sub-pictures, max_subpic_sminusl, in the current CVS may be signaled in a high-level syntax structure, such as a set of VPS, DPS, SPS, PPS, APS, or in an SEI message.

[0181] In the embodiments of the present invention, according to the illustration of Fig. 20, the sub-picture identifier, sub_pic_id[i], for the i-th sub-picture may be signaled when the number of sub-pictures is greater than 1 (max_subpic_sminus1>0).

[0182] In embodiments of the present invention, one or more syntax elements indicating the identifier of a sub-picture belonging to each layer of each set of output layers may be signaled in the VPS. According to the illustration of Fig. 20, the value sub_pic_id_layer[i][j][k] indicates that the k-th sub-picture is present in the j-th layer of the i-th set of output layers. Using this information, the decoder can recognize which of the sub-pictures can be decoded and output for each layer of a given set of output layers.

[0183] In embodiments of the present invention, a picture header (PH) is a syntactic structure that contains syntax elements applicable to all slices of a coded picture. A picture unit (PU) is a set of NAL units that are linked to each other according to a specific classification rule, follow each other in decoding order, and contain exactly one coded picture. A picture unit may comprise a picture header and one or more coded slice NAL units or VCL NAL units that make up the coded picture.

[0184] In embodiments of the present invention, the SPS set (RBSP) may be available in the decoding procedure before it is referenced, it may be included in at least one access block with a Temporalid equal to 0, or provided by external means.

[0185] In embodiments of the present invention, the SPS set (RBSP) may be available in the decoding procedure before it is referenced, it may be included in at least one access unit with Temporalid equal to 0 in a CVS that contains one or more PPS sets referencing this SPS, or provided by external means.

[0186] In embodiments of the present invention, the SPS set (RBSP) may be available in the decoding procedure before it is referenced in one or more PPS sets, it may be included in at least one image block with a nuh_layer_id equal to the smallest nuh_layer_id value among the PPS NAL units referencing the SPS NAL unit in the CVS that contains one or more PPS sets referencing this SPS, or provided by external means.

[0187] In embodiments of the present invention, an SPS set (RBSP) may be available in the decoding procedure before being referenced in one or more PPS sets, it may be included in at least one image block with a Temporalid equal to 0 and a nuh_layer_id equal to the smallest nuh_layer_id value among the PPS NAL units referencing the SPS NAL unit in a CVS that contains one or more PPS sets referencing this SPS, or provided by external means.

[0188] In embodiments of the present invention, an SPS set (RBSP) may be available in the decoding procedure before being referenced in one or more PPS sets, it may be included in at least one image block with a Temporalid equal to 0 and with a nuh_layer_id equal to the smallest nuh_layer_id value among the PPS NAL units referencing the SPS NAL unit in a CVS that contains one or more PPS sets referencing this SPS, or provided by external means.

[0189] In embodiments of the present invention, the pps_seq_parameter_set_id value determines the sps_seq_parameter_set_id value for the referenced SPS set. The pps_seq_parameter_set_id value may be the same in all PPS sets referenced by coded images in CLVS.

[0190] In embodiments of the present invention, all SPS NAL units having the same sps_seq_parameter_set_id value in the CVS may have the same content.

[0191] In embodiments of the present invention, regardless of the nub_layer_id values, NALSPS blocks may have a common space of sps_seq_tarameterset_id values.

[0192] In embodiments of the present invention, the value of nuh_layer_id in an SPS NAL unit may be equal to the smallest value of nuh_layer_id among the PPS NAL units that refer to the given SPS NAL unit.

[0193] In embodiments of the present invention, when an SPS set with nuh_layer_id equal to m is referenced in one or more PPS sets with nuh_layer_id equal to n, a layer with nuh_layer_id equal to m may be the same as a layer with nuh_layer_id equal to n, or a (direct or indirect) reference layer for a layer with nuh_layer_id equal to m.

[0194] In embodiments of the present invention, the PPS set (RBSP) is available in the decoding procedure before it is referenced, it may be included in at least one access unit with a Temporalid equal to the Temporalid of the PPS NAL unit, or provided by external means.

[0195] In embodiments of the present invention, the PPS set (RBSP) may be available in the decoding procedure before referencing it, it may be included in at least one access unit with a Temporalid equal to the Temporalid of the PPS NAL unit in the CVS that contains one or more sets of picture headers, PHs (or coded slice NAL units) referencing this PPS, or provided by external means.

[0196] In embodiments of the present invention, a PPS set (RBSP) may be available in the decoding procedure before it is referenced in one or more picture headers, PHs (or coded slice NAL units), it may be included in at least one picture block with a nuh_layer_id equal to the smallest nuh_layer_id value among the coded slice NAL units referencing the PPS NAL unit in a CVS that contains one or more sets of picture headers, PHs (or coded slice NAL units) referencing this PPS, or provided by external means.

[0197] In embodiments of the present invention, a PPS set (RBSP) may be available in the decoding procedure before being referenced in one or more picture headers, PHs (or coded slice NAL units), it may be included in at least one picture block with a Temporalid equal to the Temporalid of the PPS NAL unit and a nuh_layer_id equal to the smallest nuh_layer_id value among the coded slice NAL units referencing the PPS NAL unit in a CVS that contains one or more sets of picture headers, PHs (or coded slice NAL units) referencing this PPS, or provided by external means.

[0198] In embodiments of the present invention, the ph_pic_parameterset_id value in the image header (PH) determines the pps_pic_parameter_set_id value for the referenced PPS used. The pps_seq_parameter_set_id value may be the same in all PPS sets referenced by encoded images in CLVS.

[0199] In embodiments of the present invention, all PPS NAL units with the same pps_pic_parameter_set_id value within a picture unit must have the same content.

[0200] In embodiments of the present invention, regardless of the nuh_layer_id values, the PPS NAL units may have a common pps_seq_parameter_set_id value space.

[0201] In embodiments of the present invention, the value of nuh_layer_id in the SPS NAL unit may be equal to the smallest value of nuh_layer_id among the coded slice NAL units referencing the given PPS NAL unit.

[0202] In embodiments of the present invention, when a PPS set with nuh_layer_id equal to m is referenced in one or more NAL units of a coded slice with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or a (direct or indirect) reference layer for the layer with nuh_layer_id equal to m.

[0203] In embodiments of the present invention, the PPS set (RBSP) is available in the decoding procedure before it is referenced, it may be included in at least one access unit with a Temporalid equal to the Temporalid of the PPS NAL unit, or provided by external means.

[0204] In embodiments of the present invention, the PPS set (RBSP) may be available in the decoding procedure before referencing it, it may be included in at least one access unit with a Temporalid equal to the Temporalid of the PPS NAL unit in the CVS that contains one or more sets of picture headers, PHs (or coded slice NAL units) referencing this PPS, or provided by external means.

[0205] In embodiments of the present invention, a PPS set (RBSP) may be available in the decoding procedure before it is referenced in one or more picture headers, PHs (or coded slice NAL units), it may be included in at least one picture block with a nuh_layer_id equal to the smallest nuh_layer_id value among the coded slice NAL units referencing the PPS NAL unit in a CVS that contains one or more sets of picture headers, PHs (or coded slice NAL units) referencing this PPS, or provided by external means.

[0206] In embodiments of the present invention, a PPS set (RBSP) may be available in the decoding procedure before being referenced in one or more picture headers, PHs (or coded slice NAL units), it may be included in at least one picture block with a Temporalid equal to the Temporalid of the PPS NAL unit and a nuh_layer_id equal to the smallest nuh_layer_id value among the coded slice NAL units referencing the PPS NAL unit in a CVS that contains one or more sets of picture headers, PHs (or coded slice NAL units) referencing this PPS, or provided by external means.

[0207] In embodiments of the present invention, the ph_pic_parameterset_id value in the picture header (PH) determines the pps_pic_parameter_set_id value for the referenced PPS used. The pps_seq_parameter_set_id value may be the same in all PPS sets referenced by encoded pictures in CLVS.

[0208] In embodiments of the present invention, all PPS NAL units with the same pps_pic_parameter_set_id value within a picture unit must have the same content.

[0209] In embodiments of the present invention, regardless of the nuh_layer_id values, the PPS NAL units may have a common pps seq_parameter_set_id value space.

[0210] In embodiments of the present invention, the value of nuh_layer_id in the SPS NAL unit may be equal to the smallest value of nuh_layer_id among the coded slice NAL units referencing the given PPS NAL unit.

[0211] In embodiments of the present invention, when a PPS set with nuh_layer_id equal to m is referenced in one or more NAL units of a coded slice with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or a (direct or indirect) reference layer for the layer with nuh_layer_id equal to m.

[0212] In embodiments of the present invention, when the no_temporal_sublayer_switching_flag flag is signaled in a DPS, VPS, or SPS set, the Temporalid value of the PPS set referencing the parameter set containing this flag equal to 1 may be equal to 0, while the Temporalid value of the PPS set referencing the parameter set containing this flag equal to 1 may be greater than or equal to the Temporalid value of the parameter set.

[0213] In embodiments of the present invention, each PPS set (RBSP) may be available in the decoding procedure before referring to it, it may be included in at least one access unit with a Temporalid less than or equal to the Temporalid of the coded slice NAL unit (or PH NAL unit) that refers to it, or provided by external means. When a PPS NAL unit is included in an access unit before an access unit containing a coded slice NAL unit referring to said PPS NAL set, a VCL NAL unit providing up-switching of temporal layers, or a VCL NAL unit with nal_unit_type equal to STSA NUT, meaning that a picture in this VCL NAL unit may be an incremental temporal sublayer access (STSA) picture, it cannot be present after said PPS NAL unit and before the coded slice NAL unit referring to this PPS set.

[0214] In embodiments of the present invention, a PPS NAL unit and a coded slice NAL unit (as well as its PH NAL unit) referring to one PPS set may be contained in one access unit, AU.

[0215] In embodiments of the present invention, a PPS NAL unit and an STSA NAL unit referring to one PPS set may be contained in one access unit, AU, which precedes the coded slice NAL unit (and its PH NAL unit) referring to said PPS set.

[0216] In embodiments of the present invention, the STSA NAL unit, the PPS NAL unit, and the coded slice NAL unit (as well as its PH NAL unit) may be present in a single access unit, AU.

[0217] In the same embodiment of the present invention, the Temporalid value of the VCL NAL unit contained in the PPS may be equal to the Temporalid value of the preceding STSA NAL unit.

[0218] In the same embodiment of the present invention, the value of the picture order number (POC) in the PPS NAL unit may be greater than or equal to the value of the POC in the STSA NAL unit.

[0219] In the same embodiment of the present invention, the value of the picture order number (POC) in the coded slice NAL unit or in the PH NAL unit that refers to the PPS NAL unit may be greater than or equal to the value of the POC in the PPS NAL unit that is referred to.

[0220] Fig. 21 shows a flow chart of an example of a procedure 2100 for decoding a bitstream of an encoded video. In some implementations, one or more blocks of the procedure shown in Fig. 21 may be performed by a decoder 210. In some implementations, one or more blocks of the procedure shown in Fig. 21 may be performed by another device or group of devices separate from the decoder 210 or including the decoder 210, for example, an encoder 203.

[0221] As illustrated in Fig. 21, procedure 2100 may include obtaining an encoded video sequence from an encoded video bitstream (block 2110).

[0222] Also, in accordance with the illustration of Fig. 21, the procedure 2100 may include obtaining an image block from the encoded video sequence (block 2120).

[0223] Also, in accordance with the illustration of Fig. 21, the procedure 2100 may include obtaining a PH NAL block contained in said image block (block 2130).

[0224] Also, in accordance with the illustration of Fig. 21, the procedure 2100 may include obtaining at least one NAL unit of the coded slice contained in said image block (block 2140).

[0225] Also, according to the illustration of Fig. 21, the procedure 2100 may include decoding the coded picture based on said PH NAL unit, at least one coded slice NAL unit, a set of PPS contained in a PPS NAL unit obtained from the coded video sequence, and a set of SPS contained in an SPS NAL unit obtained from the coded video sequence, wherein said SPS NAL unit is available to at least one processor executing the procedure 2100 before said PPS NAL unit, and the PPS NAL unit is available to said at least one processor before said PP NAL unit and said at least one coded slice NAL unit (block 2150).

[0226] Also, as illustrated in Fig. 21, procedure 2100 may include outputting a decoded image (block 2160).

[0227] In embodiments of the present invention, the temporary identifier of said SPS NAL unit may be zero.

[0228] In embodiments of the present invention, the layer identifier in the PPS NAL unit is greater than or equal to the layer identifier in the SPS NAL unit.

[0229] In embodiments of the present invention, the identifier of said at least one coded slice NAL unit may be greater than or equal to a layer identifier in a PPS NAL unit.

[0230] In embodiments of the present invention, said PH NAL unit, at least one coded slice NAL unit and PPS NAL unit may be contained in a single access unit.

[0231] In embodiments of the present invention, the coded video sequence may also include a stepwise temporal sublayer access (STSA) NAL unit corresponding to the STSA picture, wherein the STSA NAL unit is not located between said PPS NAL unit and at least one coded slice NAL unit.

[0232] In embodiments of the present invention, said at least one coded slice NAL unit, PPS NAL unit and STSA NAL unit may be contained in a single access unit.

[0233] In embodiments of the present invention, the temporary identifier of said PPS NAL unit may be equal to the temporary identifier of said STSA NAL unit.

[0234] In embodiments of the present invention, the value of the picture order number (POC) of said PPS NAL unit is greater than or equal to the value of the POC of said STSA NAL unit.

[0235] Fig. 21 shows examples of blocks of the procedure (2100), however, in some implementations, the procedure 2100 may include more blocks, fewer blocks, different blocks, or blocks in a different order, compared to the illustration of Fig. 21. Moreover, in addition or alternatively, two or more blocks of the procedure (2100) may be executed in parallel.

[0236] The proposed methods may be implemented using data processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors may execute a program stored on a machine-readable medium to perform one or more of the proposed methods.

[0237] The above-described methods may be implemented as computer software that utilizes machine-readable instructions and is physically stored on one or more machine-readable media. For example, Fig. 22 shows a computer system 2200 suitable for implementing some of the embodiments of the present invention.

[0238] Computer software may be encoded using any suitable machine code or programming language that can be processed by assembly, compilation, linking, or similar mechanisms to produce code that includes instructions that are executed directly or through interpretation, microcode execution, etc., by computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0239] The instructions can be executed on computers, or computer components, of various types, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0240] The components of the computer system 2200 shown in Fig. 22 are provided for exemplary purposes only and do not imply any limitations on the scope of application or functionality of the computer software implementing embodiments of the present invention. Likewise, the illustrated configuration of components should not be interpreted as having any dependency or requirement associated with any component illustrated in the example embodiment of the computer system 2300, or a combination of such components.

[0241] The computer system 2200 may include input devices as part of a user interface. The input devices in the user interface may respond to input from one or more users using, for example, tactile input (e.g., keystrokes, finger swipes, cyberglove movements), audio input (e.g., voice, hand claps), visual input (e.g., gestures), olfactory input (not shown in the drawing). The user interface devices may also be used to capture various types of media data not necessarily associated with conscious input from a person, such as audio (e.g., voice, music, environmental sounds), images (e.g., scanned images, photographic images obtained from a camera), video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0242] The user interface devices may include one or more of the following (only one device of each type is shown in the drawing): keyboard 2201, mouse 2202, trackpad 2203, touch screen 2210 and associated graphics adapter 2250, cyber glove, joystick 2205, microphone 2206, scanner 2207, camera 2208.

[0243] The computer system 2200 may also include user interface output devices. The user interface output devices may act on the senses of one or more users, for example, via tactile output, sound, light, and / or smell / taste. The user interface output devices may include haptic output devices (e.g., haptic feedback from a touchscreen 2210, a cyber glove (not shown in the drawing), or a joystick 2205), but haptic output devices that are not input devices may also be present.For example, such devices may be audio output devices (e.g., speakers 2209, headphones (not shown in the drawing)), visual output devices (e.g., screens 2210, including screens on cathode ray tubes (cathode ray tubes, CRT), screens on liquid crystals (liquid-crystal display, LCD), plasma screens, screens on organic light-emitting diodes (OLED), both touch and without touch input functions, both with and without haptic feedback functions, and some of the screens may be capable of outputting two-dimensional visual information or more than three-dimensional visual information, using such means as, for example, stereographic output; virtual reality glasses (not shown in the drawing), holographic displays, smoke machines, as well as printers (not shown in the drawing).

[0244] The computer system 2200 may also include storage devices accessible to users and associated storage media, such as optical media, including CD / DVD ROM / RW 2220, with CD / DVD media 2221 or the like, a flash drive 2222, a removable hard disk or solid state disk 2223, previously used magnetic media, such as tapes or floppy disks (not shown in the drawing), specialized ROM / ASIC / PLD-based devices, such as hardware keys (not shown in the drawing), etc.

[0245] Those skilled in the art will appreciate that the expression "machine-readable storage medium" as used in connection with the present invention does not include data transmission media, carrier waves or other transient signals.

[0246] The computer system 2200 may also have an interface (or interfaces) with one or more communication networks (955). The networks may be, for example, wireless, wired, or optical. The networks may also be local, wide area, metropolitan, located on vehicles or industrial facilities, real-time networks, delay-tolerant networks, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks, including the global system for mobile communications (GSM), third-generation (3G), fourth-generation (4G), fifth-generation (5G), long-term evolution (LTE) networks, etc., wired or wireless global digital television broadcasting networks, including cable television, satellite television, and terrestrial broadcasting networks, vehicle and industrial facility networks, including CANBus, etc.Some networks require external network interface adapters (954) connected to general-purpose data ports or peripheral buses (949) (e.g., universal serial bus (USB) ports) of computer system 2200. Other networks can be integrated into the internal structure of computer system 2200 by connecting to a system bus, as described below (e.g., an Ethernet interface in a personal computer system or a cellular network interface in a smartphone-based computer system). For example, network 2255 can be connected to peripheral bus 2249 using network interface 2254. The use of any of such networks allows computer system 2200 to communicate with other objects.Such communication can be unidirectional, receiving only (e.g., television broadcasting), unidirectional, transmitting only (e.g., a CANbus network to certain CANbus devices), or bidirectional, such as to other computer systems using local or wide area networks. Any of the networks and network interfaces (954) described above can utilize the appropriate protocols and protocol stacks.

[0247] The user interface devices, user-accessible storage devices, and network interfaces described above can be connected to the basic internal structure 2240 of the computer system 2200.

[0248] The basic internal structure 2240 may include one or more central processing units (Central Processing Units, CPU) 2241, graphic processing units (Graphics Processing Units, GPU) 2242, specialized programmable data processing units in the form of electrically programmable gate matrices (Field Programmable Gate Areas, FPGA) 2243, hardware accelerators 2244 for specific tasks, etc. These devices, together with the read-only memory (ROM) 2245 945, the random access memory 2246, the internal high-capacity memory such as internal, user-inaccessible hard drives, solid-state drives (SSD) and similar memory 2247, may be connected by a system bus 2248. In some computer systems, the system bus 2248 may be accessible in the form of one or more physical slots, allowing the system to be expanded with additional CPUs, GPUs, etc.Peripheral devices can be connected either directly to the 2248 base system bus or to the 2249 peripheral bus. Examples of peripheral bus architectures include peripheral component interconnect (PCI) buses, USB, and the like.

[0249] CPU 2241, GPU 2242, FPGA 2243 and accelerators 2244 can execute instructions that, taken together, can constitute the computer code described above. The computer code can be stored in ROM 2245 or RAM 2246. Temporary data can be stored in RAM 2246, while persistent data can be stored, for example, in internal high-capacity memory 2247. High speed of storing data in memory devices and retrieving data from them can be ensured by using cache memory, which can be closely connected with one or more CPU 2241, GPU 2242, high-capacity memory 2247, ROM 2245, RAM 2246 and the like.

[0250] Machine-readable storage media may store computer code for performing various computer-executable operations. Such media and computer code may be specially designed and manufactured for the purposes of the present invention or may be widely available and known to those skilled in the art of computer software.

[0251] As a non-limiting example, a computer system with architecture 2200, and in particular, base structure 2240, can provide the required functionality as a result of execution by a processor (or processors) (including CPUs, GPUs, FPGAs, accelerators, etc.) of software implemented on one or more tangible computer-readable storage media. Such computer-readable storage media can be media associated with the above-described mass storage devices that are accessible to users, or non-volatile storage devices in the base structure 2240, for example, a built-in mass storage device 2247 or ROM 2245. Software that implements various embodiments of the present invention can be stored in such devices and executed by the internal structure 2240 of the computer system.The computer-readable storage medium, depending on the specific requirements, may include one or more memory devices or memory chips. The software may ensure that the base structure 2240, and in particular the processors included therein (including CPUs, GPUs, FPGAs, etc.), execute the necessary procedures or portions of the necessary procedures described in this document, including the creation of data structures stored in RAM 2246 and the modification of these data structures in accordance with the procedures defined by the software. In addition or alternatively, the computer system may provide the required functionality as a result of the operation of logic hard-coded or otherwise embodied in the electrical circuit (e.g., accelerator 2244), which may operate in conjunction with the software, or instead of it, to perform the required procedures or portions of the required procedures described in this document.References to software, where appropriate, may include such logic, and vice versa. References to a machine-readable medium, where appropriate, may include an electrical circuit (e.g., an integrated circuit) storing executable software, an electrical circuit implementing executable logic, or both. The scope of the present invention includes all appropriate combinations of hardware and software.

[0252] While several embodiments of the present invention have been described herein, modifications, variations, and equivalent substitutions are possible and fall within the scope of the present invention. Accordingly, it should be understood that those skilled in the art are capable of creating numerous systems and methods that, although not explicitly described herein, embody the concept of the present invention and, accordingly, are within its scope and spirit.

Claims

1. A decoding method using at least one processor, comprising: receiving video data containing a coded video sequence (CVS); obtaining a network abstraction layer (NAL) block of a picture header (PH) included in the picture header; obtaining at least one coded slice NAL unit corresponding to the PH NAL unit; obtaining a picture parameter set (PPS) included in a PPS NAL unit from a coded video sequence, wherein this PPS NAL unit is referenced by said PH NAL unit and coded slice NAL unit and this PPS NAL unit includes a first identifier of the PPS NAL unit level that is greater than a value of a second identifier of the network abstraction layer (NAL) unit level of the sequence parameter set (SPS); obtaining a sequence parameter set (SPS) included in an SPS NAL unit of a coded level video sequence (CLVS) obtained from a CVS, wherein the SPS NAL unit that includes the SPS is directly referenced by the PPS NAL unit and the value of the parameter_set_id syntax of the SPS is the same in all SPSs referenced by the CLVS in the CVS; and decoding the encoded image based on the PPS NAL unit and the SPS NAL unit.

2. The method according to claim 1, wherein the temporary identifier of the SPS NAL block is equal to zero.

3. The method according to claim 1, wherein the SPS NAL unit is available to said at least one processor before the PPS NAL unit and the PPS NAL unit is accessible to said at least one processor before the PH NAL unit and at least one coded slice NAL unit.

4. The method according to claim 1 or 2, wherein said PH NAL unit, at least one coded slice NAL unit and PPS NAL unit are contained in one access unit.

5. The method of claim 1, wherein the CVS also includes a stepwise temporal sublayer access (STSA) NAL unit corresponding to the STSA image, wherein the STSA NAL unit is not located between said PPS NAL unit and at least one coded slice NAL unit and said PH NAL unit, at least one coded slice NAL unit, PPS NAL unit and STSA NAL unit are contained in one access unit.

6. The method according to claim 5, wherein the temporary identifier of the PPS NAL unit is equal to the temporary identifier of the STSA NAL unit and the picture order number (POC) value of the PPS NAL unit is greater than or equal to the POC value of the STSA NAL unit.

7. A method of encoding using at least one processor, comprising: receiving video data containing at least one image; and encoding video data and at least one image to obtain encoded video data, wherein the encoded video data is transmitted for the implementation of: obtaining a coded video sequence (CVS) and a network abstraction layer (NAL) block of a picture header (PH) included in the code picture header; obtaining at least one coded slice NAL unit corresponding to the PH NAL unit and a picture parameter set (PPS) included in the PPS NAL unit from the coded video sequence, wherein said PPS NAL unit is referenced by said PH NAL unit and coded slice NAL unit and this PPS NAL unit includes a first PPS NAL unit level identifier that is greater than a value of a second network abstraction layer (NAL) unit level identifier of a sequence parameter set (SPS); and obtaining a sequence parameter set (SPS) included in an SPS NAL unit of a coded level video sequence (CLVS) obtained from a CVS, wherein the SPS NAL unit that includes the SPS is directly referenced by the PPS NAL unit and the value of the parameter_set_id syntax of the SPS is the same in all SPSs referenced by the CLVS in the CVS.

8. The method of claim 7, wherein the temporary identifier of the SPS NAL block is zero.

9. The method according to claim 7, wherein the SPS NAL unit is available to said at least one processor before the PPS NAL unit and the PPS NAL unit is accessible to said at least one processor before the PH NAL unit and at least one coded slice NAL unit.

10. The method according to claim 7 or 8, wherein said PH NAL unit, at least one coded slice NAL unit and PPS NAL unit are contained in a single access unit.

11. The method of claim 7, wherein the CVS also includes a stepwise temporal sublevel access (STSA) NAL unit corresponding to the STSA image, wherein the STSA NAL unit is not located between said PPS NAL unit and at least one coded slice NAL unit and said PH NAL unit, at least one coded slice NAL unit, PPS NAL unit and STSA NAL unit are contained in one access unit.

12. The method according to claim 11, wherein the temporary identifier of the PPS NAL unit is equal to the temporary identifier of the STSA NAL unit and the picture order number (POC) value of the PPS NAL unit is greater than or equal to the POC value of the STSA NAL unit.

13. A method for processing visual media data using at least one processor, comprising: performing a conversion between a visual media file and a visual media bitstream according to a format rule, where the bitstream specifies to perform: obtaining a coded video sequence (CVS) and a network abstraction layer (NAL) block of a picture header (PH) included in the code picture header; obtaining at least one coded slice NAL unit corresponding to the PH NAL unit and a picture parameter set (PPS) included in the PPS NAL unit from the coded video sequence, wherein said PPS NAL unit is referenced by said PH NAL unit and coded slice NAL unit and this PPS NAL unit includes a first PPS NAL unit level identifier that is greater than a value of a second network abstraction layer (NAL) unit level identifier of a sequence parameter set (SPS); and obtaining a sequence parameter set (SPS) included in an SPS NAL unit of a coded level video sequence (CLVS) obtained from a CVS, wherein the SPS NAL unit that includes the SPS is directly referenced by the PPS NAL unit and the value of the parameter_set_id syntax of the SPS is the same in all SPSs referenced by the CLVS in the CVS; and storing and transmitting a bit stream.

14. The method of claim 13, wherein the temporary identifier of the SPS NAL block is zero.

15. The method of claim 13, wherein the SPS NAL unit is available before the PPS NAL unit and The PPS NAL unit is accessible before the PH NAL unit and at least one coded slice NAL unit.

16. The method according to claim 13 or 14, wherein said PH NAL unit, at least one coded slice NAL unit and PPS NAL unit are contained in a single access unit.

17. The method of claim 13, wherein the CVS also includes a stepwise temporal sublayer access (STSA) NAL unit corresponding to the STSA image, wherein the STSA NAL unit is not located between said PPS NAL unit and at least one coded slice NAL unit and said PH NAL unit, at least one coded slice NAL unit, PPS NAL unit and STSA NAL unit are contained in one access unit.

18. A device for video decoding, comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and to perform operations according to the instructions of the program code for implementing the method according to any one of claims 1-6.

19. A device for video encoding, comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and to perform operations according to the instructions of the program code for implementing the method according to any one of claims 7-12.

20. A machine-readable medium that stores instructions that include one or more instructions that, when executed by one or more processors of a video decoding device, cause the one or more processors to perform the method according to any one of paragraphs 1-6.

Images