Method, computer system and computer readable medium for encoding video data

Abstract

A method, computer program, and computer system for encoding video data are provided. Video data is received, the video data including one or more sub-pictures having one or more virtual boundaries. Information corresponding to the virtual boundaries is signaled in a picture parameter set, a picture header, or a slice header associated with the received video data. The video data is decoded based on the signaled information corresponding to the virtual boundaries.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 003,148, filed March 31, 2020, with the United States Patent and Trademark Office, and U.S. Patent Application No. 17 / 095,366, filed November 11, 2020, with the United States Patent and Trademark Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the field of data processing, and more specifically to video encoding and decoding. Background Technology

[0004] Video encoding and decoding using inter-frame or intra-frame prediction with motion compensation has been known for decades. Uncompressed digital video can comprise a series of pictures, each with a spatial dimension of, for example, 1920×1080 luma samples and associated chroma samples. This series of pictures can have a fixed or variable picture rate (also informally referred to as the frame rate), such as 60 pictures per second or 60 Hz. Uncompressed video has very high bitrate requirements. For example, at 8 bits per sample, 1080p60 4:2:0 video (with a 1920×1080 luma sample resolution at a 60Hz frame rate) requires close to 1.5 Gbit / s of bandwidth. One hour of such video could require more than 600 GB of storage space.

[0005] One objective of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements, in some cases by two orders of magnitude or more. Lossless compression, lossy compression, and combinations thereof can be employed. Lossless compression refers to a technique that reconstructs an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may differ from the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. In the case of video, lossy compression is widely used. The tolerable amount of distortion depends on the application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television contribution applications. The achievable compression ratio reflects this: higher permissible / acceptable distortion results in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding, some of which will be discussed below.

[0007] Historically, video encoders and decoders tended to operate on a given picture size, which in most cases was defined for a coded video sequence (CVS), Group of Pictures (GOP), or similar multi-picture time frame and remained constant. For example, in MPEG-2, it was known that system design would change the horizontal resolution (and thus the picture size) depending on factors such as scene activity, but only on I-pictures, thus typically targeting a GOP. In Annex P of ITU-T Recommendation H.263, for example, it is known to resample reference pictures for different resolutions in a CVS. However, in that annex, the picture size is not changed, only the reference pictures are resampled, potentially resulting in only part of the picture canvas being used (in the case of downsampling), or only part of the scene being captured (in the case of upsampling). Further, Annex Q of H.263 allows individual macroblocks to be up- or down-resampled by a factor of 2 (in each dimension). Again, the picture size remains constant. The size of the macroblocks is fixed in H.263, so there is no need to signal it.

[0008] Changes in picture size in predicted pictures have become more mainstream in modern video coding. For example, VP9 allows reference picture resampling and changing resolution for entire pictures. Similarly, certain proposals made for VVC (including, for example, Hendry et al., “On adaptive resolution change (ARC) for VVC,” Joint Video Team document JVET-M0135-vl, January 9-19, 2019, the entire contents of which are incorporated by reference herein) allow resampling of entire reference pictures to different - higher or lower - resolutions. In the above document, it is proposed that different candidate resolutions are encoded in the sequence parameter set and referenced by each picture syntax element in the picture parameter set. SUMMARY

[0009] Embodiments relate to methods, systems, and computer-readable media for encoding video data. According to one aspect, a method for encoding video data is provided. The method can include receiving video data including one or more sub-pictures having one or more virtual boundaries; signaling, in a picture parameter set, a picture header, or a slice header associated with the received video data, information corresponding to the virtual boundaries; and decoding the video data based on the signaled information corresponding to the virtual boundaries.

[0010] According to another aspect, a computer system for encoding video data is provided. The computer system can include one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored in the one or more storage devices which are executable by at least one of the one or more processors via at least one of the one or more memories, whereby the computer system is capable of performing a method. The method can include receiving video data including one or more sub-pictures having one or more virtual boundaries, signaling information corresponding to the virtual boundaries in a picture parameter set, a picture header, or a slice header associated with the received video data, and decoding the video data based on the signaled information corresponding to the virtual boundaries.

[0011] According to yet another aspect, a computer-readable medium for encoding video data is provided. The computer-readable medium can include one or more computer-readable storage devices and program instructions stored in the one or more tangible storage devices which are executable by a processor. The program instructions are executable by the processor to perform a method which can accordingly include receiving video data including one or more sub-pictures having one or more virtual boundaries, signaling information corresponding to the virtual boundaries in a picture parameter set, a picture header, or a slice header associated with the received video data, and decoding the video data based on the signaled information corresponding to the virtual boundaries. BRIEF DESCRIPTION OF DRAWINGS

[0012] These and other objects, features, and advantages will become apparent in light of the following detailed description of illustrative embodiments thereof, described in connection with the following drawings. The various features of the drawings are not to scale as the drawings are intended to convey a clear understanding of the embodiments described herein. In the drawings:

[0013] Figure 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0014] Figure 2 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0015] Figure 3 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0016] Figure 4 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0017] Figure 5 is a schematic diagram of options for signaling ARC parameters according to prior art or embodiments indicated.

[0018] Figure 6 is an example of a syntax table according to an embodiment.

[0019] Figure 7 is a schematic diagram of a computer system according to an embodiment.

[0020] Figure 8 is an example of a prediction structure for scalability with adaptive resolution change.

[0021] Figure 9 is an example of a syntax table according to an embodiment.

[0022] Figure 10 is a schematic diagram of a simplified block diagram that parses and decodes the poc cycle and access unit count values for each access unit.

[0023] Figure 11 is a schematic diagram of a video bitstream structure including multiple layers of sub-pictures.

[0024] Figure 12 is a schematic diagram of a display of selected sub-pictures with enhanced resolution.

[0025] Figure 13 is a block diagram of a decoding and display process for a video bitstream including multiple layers of sub-pictures.

[0026] Figure 14 is a schematic diagram of a 360 video display with sub-picture enhancement layers.

[0027] Figure 15 is an example of sub-picture layout information and its corresponding layer and picture prediction structure.

[0028] Figure 16 is an example of sub-picture layout information and its corresponding layer and picture prediction structure, with a spatial scalability mode of local regions.

[0029] Figure 17 is an example of a syntax table for sub-picture layout information.

[0030] Figure 18 is an example of a syntax table for a SEI message of sub-picture layout information.

[0031] Figure 19 is an example of a syntax table for indicating output layers and tier / level / horizontal information for each output layer set.

[0032] Figure 20 is an example of a syntax table for indicating output layer mode on for each output layer set.

[0033] Figure 21is an example of a syntax table for indicating the current subpicture of each layer of each output layer set.

[0034] Figure 22 is an example of a syntax table for indicating a subpicture identifier.

[0035] Figure 23 is an example of a syntax table for indicating subpicture partition information.

[0036] Figure 24 is an example of a syntax table for virtual boundary information in SPS.

[0037] Figure 25 is an example of a syntax table for virtual boundary information in picture header.

[0038] Figure 26 is an example of a syntax table for virtual boundary information in PPS. DETAILED DESCRIPTION

[0039] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it is understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that can be practiced in various forms. The claimed structures and methods can be embodied in a multitude of different ways and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope to those skilled in the art. In the description, details of well-known features and techniques can be omitted to avoid unnecessarily obscuring the presented embodiments.

[0040] As described previously, video encoders and decoders tend to operate on a given picture size, which is defined and remains constant for a coded video sequence in most cases. However, a picture can be partitioned into one or more subpictures. Each subpicture can be further divided into one or more slices. Two or more independently coded subpictures can be merged into a coded picture, decoded by a decoder, and displayed as a single output picture. When a coded picture is composed of two or more coded subpictures, there can be virtual boundaries. It is therefore most desirable to treat virtual boundaries specially.

[0041] The numerous aspects described herein with respect to various embodiments are described in reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer-readable media. It will be understood that each block of the flow diagram and / or block diagram illustrations, and combinations of blocks in the flow diagram and / or block diagram illustrations, can be implemented by computer-readable program instructions.

[0042] Figure 1A simplified block diagram of a communication system (100) according to one embodiment of the present disclosure is shown. The system (100) can include at least two terminals (110, 120) interconnected via a network (150). For unidirectional data transmission, a first terminal (110) can code video data at a local location for transmission to the other terminal (120) via the network (150). The second terminal (120) can receive the coded video data of the other terminal from the network (150), decode the coded data and display the recovered video data. Unidirectional data transmission can be common in the application of media serving, etc.

[0043] Figure 1 A second pair of terminals (130, 140) is shown arranged to support bidirectional transmission of coded video that can occur, for example, during a video conference. For bidirectional data transmission, each terminal (130, 140) can code video data captured at a local location for transmission to the other terminal via the network (150). Each terminal (130, 140) also can receive coded video data transmitted by the other terminal, can decode the coded data and can display the recovered video data at a local display device.

[0044] In Figure 1 The terminals (110-140) can be illustrated as servers, personal computers and smartphones, although the principles of the present disclosure can not be limited to this. Embodiments of the present disclosure find application with laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network (150) represents any number of networks that convey coded video data among the terminals (110-140), including, for example, wireline and / or wireless communication networks. The communication network (150) can exchange data in circuit- switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks and / or the Internet. For the purposes of the present discussion, the architecture and topology of the network (150) can be immaterial to the operation of the present disclosure unless explained in the following.

[0045] As an example of the application of the disclosed subject matter, Figure 2 Video encoders and decoders in a streaming system are shown. The disclosed subject matter can be equally applicable to other video enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0046] A streaming system can include a capture subsystem (213) that can include a video source (201), for example a digital camera, that creates, for example, an uncompressed video sample stream (202). In comparison to an encoded video bitstream, the video sample stream (202), depicted as a bold line to emphasize a high data volume, can be processed by an encoder (203) coupled to the camera (201). The encoder (203) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in greater detail below. In comparison to the sample stream, the encoded video bitstream (204), depicted as a thin line to emphasize the lower data volume, can be stored on a streaming server (205) for future use. One or more streaming clients (206, 208) can access the streaming server (205) to retrieve copies (207, 209) of the encoded video bitstream (204). The client (206) can include a video decoder (210) that decodes the incoming copy (207) of the encoded video bitstream and can create an outgoing video sample stream (211) that can be rendered on a display (212) or another rendering device (not depicted). In some streaming systems, the video bitstream (204, 207, 209) can be encoded according to certain video coding / compression standards. Examples of those standards include ITU-T Recommendation H.265. Under study is a video coding standard informally known as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC.

[0047] Figure 3 A functional block diagram of a video decoder (210), which can be in accordance with an embodiment of the present application.

[0048] The receiver (310) can receive one or more coded video sequences to be decoded by the decoder (210). In the same or another embodiment, one coded video sequence at a time can be received, where the decoding of each coded video sequence is independent from the decoding of the other coded video sequences. The coded video sequences can be received from a channel (312), which can be a hardware / software link into a storage device storing the encoded video data. The receiver (310) can receive coded video data that can be forwarded to their respective using entities (not depicted) along with other data such as coded audio data and / or ancillary data streams. The receiver (310) can separate the coded video sequences from the other data. To protect against network jitter, a buffer memory (315) can be coupled in between the receiver (310) and the entropy decoder / parsen (320) (hereinafter “parsen”). When the receiver (310) receives data from a store / forward device with sufficient bandwidth and controllability, or from an isosychronous network, the buffer (315) can not be needed, or can be smaller. To strive for usage over a service packet network like the Internet, the buffer (315) can be needed, can be relatively large, and can advantageously be of adaptive size.

[0049] The video decoder (210) can include a parser (310) to reconstruct symbols (421) from the entropy coded video sequence. Categories of those symbols include information to manage operation of the decoder (210), and potentially information to control a display device (212) that is not a part of the decoder, but can be coupled to it, as in Figure 2The control information for the display device can be Supplemental Enhancement Information (SEI messages) or Parameter Sets fragments (not depicted) of Video Usability Information (VUI). The parser (320) can parse / entropy-decode the received coded video sequence. The coding of the coded video sequence can be in accordance with video coding technologies or standards as are known to those skilled in the art and can follow principles as are known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The parser (320) can extract from the coded video sequence, a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based upon at least one parameter corresponding to the group. The subgroups can include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, Coding Units (CUs), blocks, Transform Units (TUs), Prediction Units (PUs) and so on. The entropy decoder / parser can also extract from the coded video sequence, information such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0050] The parser (320) can perform entropy-decoding / parsing operations on the video sequence received from the buffer (315), creating symbols (321).

[0051] The reconstruction of the symbols (321) can involve a number of different units, depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks), and other factors. Which units are involved, and how, can be controlled by subgroup control information that the parser (320) parses from the coded video sequence. For the sake of brevity, such subgroup control information flows between the parser (320) and the units below are not described.

[0052] In addition to the functional blocks already mentioned, the decoder 210 can be conceptually subdivided into a number of functional units as described below. In practical implementations operating under commercial constraints, many of these units interact closely with each other and can be integrated with each other to different extents. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the functional units below is appropriate.

[0053] The first unit can be a scaler / inverse transform unit (351). The scaler / inverse transform unit (351) receives quantized transform coefficients as symbols (321) and control information, including which transform to use, block size, quantization factor, quantization scaling matrices, etc., from the parser (320). The scaler / inverse transform unit can output a block comprising sample values, which can be input into the aggregator (355).

[0054] In some cases, the output samples of the scaler / inverse transform unit (351) can belong to an intra coded block; i.e., a block that is not using predictive information from previously reconstructed pictures, but can use predictive information from previously reconstructed parts of the current picture. Such predictive information can be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) generates a block of the same size and shape as the block being reconstructed, using surrounding already reconstructed information taken from the current (partially reconstructed) picture (356). In some cases, the aggregator (355) adds, on a per sample basis, the predictive information generated by the intra-prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351).

[0055] In other cases, the output samples of the scaler / inverse transform unit (351) can belong to an inter coded and potentially motion compensated block. In this case, a motion compensated prediction unit (353) can access a reference picture memory (357) to retrieve samples for prediction. After motion compensation of the retrieved samples according to the symbols (321), these samples can be added by the aggregator (355) to the output of the scaler / inverse transform unit (in this case referred to as residual samples or residual signal), generating the output sample information. The fetching of predictive samples by the motion compensated prediction unit from addresses within the reference picture memory can be controlled by motion vectors, and the motion vectors are made available to the motion compensated prediction unit in the form of the symbols (321), e.g., including X, Y and reference picture component. Motion compensation can also include interpolation of sample values taken from the reference picture memory when sub-sample precise motion vectors are used, motion vector prediction mechanisms, etc.

[0056] The output samples of the aggregator (355) can be subject to various loop filtering techniques in the loop filter unit (356). Video compression technologies can include in-loop filter technologies that are controlled by parameters included in the coded video bitstream as symbols (321) from the parser (320) that are made available to the loop filter unit (356). However, in other embodiments, video compression technologies can also be responsive to meta-information obtained during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, as well as responsive to previously reconstructed and in-loop filtered sample values.

[0057] The output of the loop filter unit (356) can be a sample stream that can be output to the display device (212) and stored in the reference picture memory (356) for use in the inter prediction of future pictures.

[0058] Once fully reconstructed, certain coded pictures can be used as reference pictures for future prediction. Once a coded picture has been fully reconstructed and the coded picture is identified (by, for example, parser (320)) as a reference picture, the current reference picture (356) can become a part of the reference picture buffer (357) and a fresh current picture memory can be reallocated prior to commencing the reconstruction of the following coded picture.

[0059] Video decoder 320 can perform decoding operations according to a predetermined video compression technology that can be archived as a standard, such as ITU-T H.265. In the sense that an encoded video sequence conforms to the syntax of the video compression technology or standard, and the profiles documented in the video compression technology or standard, the encoded video sequence can be said to be compliant. Also for compliance, the complexity of the encoded video sequence needs to be within limits set by the levels of the video compression technology or standard. In some cases, the limits set by the levels include maximum picture size, maximum frame rate, maximum reconstruction sample rate (measured in, for example megasamples per second), maximum reference picture size, and so on. Limits set by the levels can be further restricted by Hypothetical Reference Decoder (HRD) specifications and metadata signaled in the encoded video sequence that manages HRD buffers.

[0060] In an embodiment, receiver (310) can receive additional (redundant) data with the encoded video. The additional data can be part of the encoded video sequence. The additional data can be used by video decoder (320) to properly decode the data and / or to more accurately reconstruct the original video data. Additional data can be in the form of, for example, a temporal, spatial, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, and so on.

[0061] Figure 4 A functional block diagram of a video encoder (203) that can be in accordance with an embodiment of the present disclosure.

[0062] The encoder (203) can receive video samples from a video source (201) (that is not part of the encoder) that can capture video image(s) to be coded by the encoder (203).

[0063] The video source (201) can provide the source video sequence in the form of a digital video sample stream that is to be encoded by the encoder (203), which can have any suitable depth (for example: 8-bit, 10-bit, 12-bit,...), any suitable color space (for example BT.601 Y CrCB, RGB,...), and any suitable sampling structure (for example Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (201) can be a storage device storing previously prepared video. In a videoconferencing system, the video source (201) can comprise a camera that captures local image information as a video sequence. The video data can be provided as a plurality of individual pictures that impart motion when viewed in sequence. The pictures themselves can be organized as a spatial array of pixels, wherein each pixel can comprise one or more samples depending on the sampling structure, color space, etc. in use. A person having ordinary skill in the art can readily understand the relationship between pixels and samples. The description below focuses on samples.

[0064] According to an embodiment, the encoder (203) can encode and compress the pictures of the source video sequence into a coded video bitstream (443), also known as coded video sequence (CVS), in real time or under any other time constraints as required by the application. Enforcing appropriate coding speed is one function of controller (450). The controller controls other functional units as described below and is functionally coupled to these units. For clarity, the couplings are not pictured. Parameters set by the controller can include rate control related parameters (picture skip, quantizer, lambda value of rate-distortion optimization techniques,...), picture size, group of pictures (GOP) layout, maximum motion vector search range, and so on. A person having ordinary skill in the art can readily identify other functions of the controller (150) that pertain to video encoder (203) optimizations for a given system design.

[0065] Some video encoders operate in what those skilled in the art will readily recognize as an "encoding loop." As a simple description, the encoding loop can include an encoding portion of the source encoder (430), e.g., a prediction unit (not shown) that is responsible for creating symbols based on an input picture to be encoded and reference pictures, and a local decoder (433) embedded in the encoder 203 that reconstructs the symbols to create the sample data in a similar manner as a (remote) decoder would create them from the symbols (since any compression between symbols and the encoded video bitstream is lossless in the video compression technologies considered by the present application). The reconstructed sample stream is input to the reference picture buffer (434). Since the decoding of symbols results in a bit-exact result whether done at the encoder or the decoder location, the content of the reference picture buffer is also bit exactly corresponding between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" for the reference pictures exactly the same sample values as the decoder would "see" when using the predictions during decoding. This fundamental principle of reference picture synchronicity (and the drift that occurs if the synchronicity cannot be maintained, e.g., due to channel errors) is well known to those skilled in the art.

[0066] The operation of the "local" decoder (433) can be identical to the "remote" decoder (210) that has been described in detail above, e.g., with respect to FIG. 2. However, additional brief reference is made to FIG. 3. Figure 3 The operation of the "local" decoder (433) can be identical to the "remote" decoder (210) that has been described in detail above, e.g., with respect to FIG. 2. However, additional brief reference is made to FIG. 3. Figure 3 When symbols are available and the entropy encoder (445) and parser (320) are able to losslessly encode / decode the symbols into the encoded video sequence, the entropy decoding portion of the decoder 210 (including the channel (312), receiver (310), buffer (315), and parser (320)) can not be fully implemented in the local decoder (433).

[0067] At this point it can be observed that any decoder technology other than the parsing / entropy decoding that is present in the decoder must also be present in the corresponding encoder in substantially the same functional form. To this end, the disclosed subject matter focuses on the decoder operation. The description of the encoder technology can be abbreviated in that it is the inverse of the fully described decoder technology. More detailed description is required only in certain areas and is provided below.

[0068] As part of its operation, the source coder (430) can perform motion compensated predictive encoding. This motion compensated predictive encoding encodes an input frame predictively with respect to one or more previously coded frames from the video sequence that are designated as "reference frames." In this manner, the encoding engine (432) codes the difference between pixel blocks of an input frame and pixel blocks of reference frames that can be selected as a predictive reference for the input frame.

[0069] The local video decoder (433) can decode encoded video data of a frame that can be designated as a reference frame, based on symbols created by the source encoder (430). The operation of the encoding engine (432) can be a lossy process. When the encoded video data can be decoded by the video decoder (430), Figure 4 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (433) replicates the decoding process, which can be performed by the video decoder on the reference frame, and allows the reconstructed reference image to be stored in the reference frame cache (434). In this way, the source encoder (203) can locally store a copy of the reconstructed reference frame that shares the same content (no transmission errors) as the reconstructed reference frame that will be obtained by the remote video decoder.

[0070] The predictor (435) can perform a prediction search against the encoding engine (432). That is, for a new frame to be encoded, the predictor (435) can search in the reference image memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (435) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (435), it can be determined that the input image can have prediction references obtained from multiple reference images stored in the reference image memory (434).

[0071] The controller (450) can manage the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters for encoding video data.

[0072] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (445). The entropy encoder performs lossless compression on the symbols generated by the various functional units according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0073] The transmitter (440) can buffer the encoded video sequence created by the entropy encoder (445) in preparation for transmission via a communication channel (460), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (440) can combine the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0074] The controller (450) can manage operation of the encoder (203). During coding, the controller (450) can assign to each coded picture a certain coded picture type, which can affect the coding techniques that can be applied to the respective picture. For example, pictures often can be assigned as one of the following frame types.

[0075] An Intra Picture (I picture) can be one that can be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow for different types of Intra pictures, including, for example Independent Decoder Refresh pictures. A person of ordinary skill in the art understands the variants of I pictures and their respective applications and features.

[0076] A predictive picture (P picture) can be one that can be coded and decoded using either intra prediction or inter prediction that uses at most one motion vector and reference index to predict sample values of each block.

[0077] A bi-predictive picture (B picture) can be one that can be coded and decoded using either intra prediction or inter prediction that uses at most two motion vectors and reference indices to predict sample values of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0078] Source pictures often can be spatially subdivided into a plurality of blocks of samples (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples) and coded block-by-block. The blocks can be predictively coded with reference to other (coded) blocks, determined according to the coding assignment applied to the respective picture of the block. For example, blocks of an I picture can be non-predictively coded, or the blocks can be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Blocks of a P picture can be non-predictively coded with either spatial prediction or temporal prediction with reference to one previously coded reference picture. Blocks of a B picture can be non-predictively coded with either spatial prediction or temporal prediction with reference to one or two previously coded reference pictures.

[0079] The video encoder (203) can perform coding operations in accordance with a predetermined video coding technology or standard, such as ITU-T H.265. In its operation, the video encoder (203) can perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The coded video data, therefore, can conform to a syntax specified by the video coding technology or standard being used.

[0080] In an embodiment, the transmitter (440) can transmit additional data with the encoded video. The video encoder (530) can encode such data as part of the coded video sequence. Additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Video Usability Information (VUI) parameter set fragments, and so on.

[0081] Before certain aspects of the disclosed subject matter are described in more detail, it is instructive to introduce some terminology that will be used throughout the remainder of this document.

[0082] In certain contexts, a subpicture refers to a rectangular arrangement of samples, blocks, macroblocks, coding units, or similar entities, which are semantically grouped and can be independently coded at changed resolution. A picture can have one or more subpictures. One or more coded subpictures can form a coded picture. One or more subpictures can be assembled into a picture and one or more subpictures can be extracted from a picture. In certain environments, one or more coded subpictures can be assembled into a coded picture in the compressed domain without transcoding to the sample level and, in the same or certain other contexts, one or more coded subpictures can be extracted from a coded picture in the compressed domain.

[0083] Adaptive resolution change refers to a mechanism that enables changing the resolution of pictures or subpictures in a coded video sequence by, for example, a method of reference picture resampling. In the following, ARC parameters refer to control information required to perform adaptive resolution change, which can include, for example, filter parameters, scaling factors, resolutions of output pictures and / or reference pictures, and various control flags, etc.

[0084] The above description focuses on encoding and decoding of single, semantically independent coded video pictures. Before describing the implications of encoding / decoding multiple subpictures with independent ARC parameters and the additional complexity they imply, options for signaling ARC parameters will be described.

[0085] Referring to Figure 5 , several new options for signaling ARC parameters are shown. As described for each option, they have certain advantages and disadvantages from the perspective of coding efficiency, complexity, and architecture. A video coding standard or technology can select one or more of these options or known options from known technologies for signaling ARC parameters. The options can not be mutually exclusive and it can be understood that they can be interchanged depending on the application requirements, the standard technology involved, or the encoder's choice.

[0086] The categories of ARC parameters can include:

[0087] - upsampling / downsampling factors, separate or combined in X and Y dimensions.

[0088] - upsampling / downsampling factors, with an additional time dimension, indicating a constant speed up / down of a given number of pictures.

[0089] - either of the above can involve one or more encodings, possibly short syntax elements, that can point to a table containing the factors.

[0090] - resolution of input pictures, output pictures, reference pictures, coded pictures, in X or Y dimension, in units of samples, blocks, macroblocks, CUs or any other suitable granularity. If there is more than one resolution (e.g. one for input pictures, one for reference pictures), in some cases a set of values can be inferred from another set of values. This can be gated, for example, using a flag. See below for more detailed examples.

[0091] - "warping" coordinates similar to those used in H.263 Annex P, again in suitable granularity as described above. H.263 Annex P defines an efficient way of encoding such warping coordinates, but other potentially more efficient methods can be envisaged. For example, the variable-length invertible, "Huffman-like" encoding of the warping coordinates in Annex P can be replaced by a fixed-length binary encoding, where the length of the binary codewords can be derived, for example, from the maximum picture size possibly multiplied by a factor and offset by a value, in order to allow "warping" outside the maximum picture size boundaries.

[0092] - upsampling or downsampling filter parameters. In the simplest case, there can be a single filter for upsampling and / or downsampling. However, in some cases it can be beneficial to allow more flexibility in the filter design, which can require filter parameters to be signaled. Such parameters can be selected by an index into a list of possible filter designs, the filter can be fully specified, for example, by a list of filter coefficients, using suitable entropy coding techniques, the filter can be implicitly selected by an upsampling / downsampling ratio, which in turn is signaled according to any of the above mechanisms, and so on.

[0093] In the following, the description assumes coding of a limited set of upsampling / downsampling factors (the same factor to be used in X and Y dimension) indicated by a codeword. The codeword is advantageously variable length coded, e.g. using an Ext-Golomb code, which is commonly used for certain syntax elements in video coding specifications such as H.264 and H.265. One suitable mapping between values and upsampling / downsampling factors can for example be as shown in the following table:

[0094] Code word Ext-Golomb code Original resolution / target resolution 0 1 1 / 1 1 010 1 / 1.5 (up 50%) 2 011 1.5 / 1 (down 50%) 3 00100 1 / 2 (up 100%) 4 00101 2 / 1 (down 100%)

[0095] Many similar mappings can be designed depending on the needs of the application and the capabilities of the up- or downscaling mechanisms available in the video compression technology or standard. The table can be extended to more values. Values can also be represented using entropy coding mechanisms other than Ext-Golomb codes, e.g. using binary coding. This can have certain advantages when the resampling factors are concerned outside the video processing engine (most importantly the encoder and decoder) itself, e.g. by MANEs (media aware network elements). It should be noted that for the (assumed) most common case of no resolution change, a short Ext-Golomb code can be chosen; in the table above, only a single bit. In the most common case, this can be more coding efficient than using binary codes.

[0096] The number of table entries and their semantics can be fully or partially configurable. For example, the basic outline of the table can be conveyed in a "high" parameter set, e.g. a sequence parameter set or a decoder parameter set. Alternatively or additionally, one or more such tables can be defined in the video coding technology or standard, and can be selected by e.g. a decoder parameter set or a sequence parameter set.

[0097] In the following, we describe how to include the upsampling / downsampling factors (ARC information) coded as described above in the video coding technology or standard syntax. Similar considerations can apply to one or several codewords controlling upsampling / downsampling filters. When filters or other data structures require a relatively large amount of data, see the discussion below.

[0098] H.263 Annex P includes the ARC information 502 in the picture header 501 in the form of four warping coordinates, specifically in the H.263 PLUSP TYPE (503) header extension. This can be a sensible design choice when a) there is a picture header available, and b) the ARC information is expected to change frequently. However, when using H.263 style signaling, the overhead can be quite high, and since the picture header can be of transient nature, the scaling factors can not be applicable across picture boundaries.

[0099] The above cited JVCET-M135-vl includes ARC reference information (505) (index) in a picture parameter set (504) that indexes a list (506) of target resolutions that in turn are located within a sequence parameter set (507). According to an oral statement by the author, the placement of the possible resolutions in table (506) in the sequence parameter set (507) can prove to be correct by using the SPS as the interoperability negotiation point during the capability exchange. The resolution of different pictures can change within the limits set by the values in table (506) by referring to the appropriate picture parameter set (504).

[0100] Still referring to Figure 5 The following additional options can exist to deliver ARC information in a video bitstream. Each of these options has certain advantages over the prior art as described above. These options can exist simultaneously in the same video coding technology or standard.

[0101] In an embodiment, ARC information (509) such as resampling (scaling) factors can exist in a slice header, group of block (GOB) header, tile header, or tile group header (508) (hereinafter tile group header). This can be sufficient if the ARC information is small, like a single variable length ue (v) or a few bits of fixed length codeword as shown above. An additional advantage of having the ARC information directly in the tile group header is that the ARC information can apply to a sub-picture instead of the entire picture, for example, represented by the tile group header. See also below. Furthermore, even if the video compression technology or standard only contemplates adaptive resolution changes of the entire picture (as opposed to, for example, tile group based adaptive resolution changes), placing the ARC information in the tile group header has certain advantages from a fault tolerance point of view over placing it in a H.263 style picture header.

[0102] In the same or another embodiment, ARC information (512) can exist in a suitable parameter set (511) itself, for example, a picture parameter set, a header parameter set, a tile parameter set, an adaptation parameter set, etc. (adaptation parameter set depicted). The scope of this parameter set advantageously is not larger than a picture, for example, a tile group. The use of the ARC information is implicitly indicated by activating the relevant parameter set. For example, when the video coding technology or standard only contemplates picture based ARC, then a picture parameter set or equivalent can be suitable.

[0103] In the same or another embodiment, ARC reference information (513) can be present in a tile group header (514) or similar data structure. This reference information (513) can refer to a subset of the ARC information (515) available in a parameter set (516) with a range exceeding a single picture, e.g. a sequence parameter set or a decoder parameter set.

[0104] As used in JVET-M0135-vl, the additional level of indirect implicit activation of PPS according to tile group header, PPS, SPS, seems unnecessary, as a picture parameter set can be used (and has been in certain standards such as RFC3984) for capability negotiation or announcement just like a sequence parameter set. However, if ARC information should apply to sub-pictures represented by tile groups as well, it can be a better choice to activate the range limited to a parameter set of the tile group, e.g. an adaptive parameter set or a header parameter set. Furthermore, if the ARC information is of non-negligible size, e.g. contains filter control information such as a large number of filter coefficients, a parameter can be a better choice from an encoding efficiency point of view compared to directly using a header (508), as these settings can be reused for future pictures or sub-pictures by referring to the same parameter set.

[0105] When using a sequence parameter set or another higher parameter set with a range spanning multiple pictures, the following issues can need to be taken into account:

[0106] 1. The parameter set used to store the ARC information table (516), which in some cases can be a sequence parameter set, but in other cases advantageously is a decoder parameter set. A decoder parameter set can have an activation range of multiple CVS, i.e. all encoded video bits from the beginning of a session to the end of a session. Such a range can be more appropriate, as the possible ARC factors can be a decoder characteristic, possibly implemented in hardware, and hardware characteristics tend to not change with arbitrary CVS, at least picture groups in certain entertainment systems, one second or shorter in length. That said, placing the table into a sequence parameter set is explicitly included in the placement options described herein, in particular in connection with the following point 2.

[0107] 2. ARC reference information (513) is advantageously placed directly in the picture / slice / tile / GOB / tile group header (hereafter tile group header) (514) instead of being placed in the picture parameter set as in JVET-M0135-vl for the following reasons: When the encoder wants to change a single value in the picture parameter set, e.g. the ARC reference information, then it has to create a new PPS and refer to that new PPS. Assume that only the ARC reference information is changed while other information remains the same, e.g. quantization matrix information in the PPS. This information can be large in size and needs to be retransmitted to make the new PPS complete. Since the ARC reference information can be a single codeword, like the index in table (513), and this will be the only value that is changed, it would be cumbersome and wasteful to retransmit all the information (e.g. quantization matrix information). So far, it can be better from the coding efficiency point of view to avoid the indirection through the PPS as proposed in JVET-M0135-vl. Similarly, putting the ARC reference information into the PPS has the additional drawback that the ARC information (513) referred to by the ARC reference information has to necessarily be applied to the whole picture instead of a sub-picture, since the picture parameter set activation is picture-wise.

[0108] In the same or another embodiment, the signaling of the ARC parameters can follow the detailed examples outlined in Figure 8 Figure 6 A syntax diagram is described that represents the notation used in video coding standards since at least 1993. The notation of this syntax diagram roughly follows C-style programming. Lines shown in bold indicate syntax elements that are present in the bitstream, lines not shown in bold usually indicate control flow or variable settings.

[0109] The tile group header (601), which is an exemplary syntax structure of a header applicable to a (possibly rectangular) portion of a picture, can conditionally contain the variable-length Exp-Golomb coded syntax element dec_pic_size_idx (602) (depicted in bold). The presence of this syntax element in the tile group header can be gated by the use of adaptive resolution (603) - here, the value of the flag is not depicted in bold, which means that the flag is present in the bitstream when the flag appears in the syntax diagram. Whether adaptive resolution is used for this picture or portion of a picture can be signaled in any high-level syntax structure inside or outside the bitstream. In the example shown, it is signaled in the sequence parameter set, which is outlined as follows.

[0110] Still referring to Figure 6 ​An excerpt of the sequence parameter set (610) is also shown. The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, this flag can indicate the use of adaptive resolution, which in turn can require certain control information. In the example, this control information is conditionally present based on the value of a flag based on an if() statement in the parameter set (612) and the tile group header (601).

[0111] When adaptive resolution is used, in this example, the encoding is in terms of the output resolution in samples (613). The number 613 refers to output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can define the resolution of the output picture. In other places in the video coding technology or standard, certain limits can be defined on either of these values. For example, a horizontal limit can limit the total number of output samples, which can be the product of the two syntax element values. In addition, certain video coding technologies or standards, or external technologies or standards, e.g., system standards, can limit the number range (e.g., one or both dimensions must be divisible by a power of the number 2), or the aspect ratio (e.g., the relationship of width and height must be 4:3 or 16:9). Such limits can be introduced to facilitate hardware implementations or for other reasons, and are well known in the art.

[0112] In certain applications, it can be desirable for the encoder to indicate to the decoder to use a certain reference picture size, rather than implicitly assuming that size to be the size of the output picture. In this example, the syntax element reference_pic_size_present_flag (614) gates the conditional presence of the reference picture dimensions (615) (again, the numbers refer to width and height).

[0113] Finally, a table of possible decoded picture width and height is shown. Such a table can be represented, for example, by a table indication (num_dec_pic_size_in_luma_samples_minus1) (616). The "minus1" can refer to the interpretation of the value of this syntax element. For example, if the coded value is zero, then there is one table entry. If the value is 5, then there are six table entries. For each "row" in the table, the decoded picture width and height are included in the syntax (617).

[0114] The presented table entry (617) can be indexed using the syntax element dec_pic_size_idx (602) in the tile group header, allowing different decoding sizes - effectively scaling factors - per tile group.

[0115] Certain video coding techniques or standards, such as VP9, support spatial scalability by incorporating temporal scalability to enable certain forms of reference picture resampling (signaled in a manner quite different from the disclosed subject matter) to achieve spatial scalability. In particular, certain reference pictures can be upsampled to a higher resolution using ARC-style techniques to form the basis of a spatial enhancement layer. These upsampled pictures can be refined at the higher resolution using normal prediction mechanisms, adding detail.

[0116] The disclosed subject matter can be used in such an environment. In certain cases, the value in the NAL unit header, such as the temporal ID field, can be used to indicate not only temporal layers, but also spatial layers, in the same or another embodiment. Doing so has certain advantages for certain system designs; for example, for scalable environments, the existing Selected Forwarding Unit (SFU) creation and optimization of forwarding based on the temporal layer selection of the NAL unit header temporal ID value can be used without modification. To enable this, a mapping between the coded picture size and the temporal layer, indicated by the temporal ID field in the NAL unit header, can be needed.

[0117] In certain video coding techniques, an Access Unit (AU) can refer to a coded picture, slice, tile, NAL unit, etc. that is captured and composed of corresponding pictures / slices / tiles / NAL units code stream in a given temporal instance. The time of this instance can be the composition time.

[0118] In HEVC and certain other video coding techniques, a picture order count (POC) value can be used to indicate a selected reference picture among a plurality of reference pictures stored in a decoded picture buffer (DPB). When an Access Unit (AU) includes one or more pictures, slices, or tiles, each picture, slice, or tile belonging to the same AU can carry the same POC value, from which it can be derived that they are created from content according to the same composition time. In other words, in the scenario where two pictures / slices / tiles carry the same given POC value, this can indicate that the two pictures / slices / tiles belong to the same AU and have the same composition time. Conversely, two pictures / tiles / slices with different POC values can indicate that those belong to different AUs and have different composition times.

[0119] In embodiments of the disclosed subject matter, the rigid relationship described above can be relaxed in that an access unit can include pictures, slices, or tiles with different POC values. By allowing different POC values within an AU, the POC values can be used to identify potentially independently decodable pictures / slices / tiles with the same presentation time. Thus, multiple scalable layers can be supported without changing the reference picture selection signaling (e.g., reference picture set signaling or reference picture list signaling), as will be described in more detail below.

[0120] However, for other pictures / slices / tiles with different POC values, it is still desirable to be able to identify from the POC values alone the AU to which the pictures / slices / tiles belong. This can be achieved, as described below.

[0121] In the same or other embodiments, an access unit count (AUC) can be signaled in a high-level syntax structure, such as a NAL unit header, a slice header, a tile group header, an SEI message, a parameter set, or an AU delimiter. The value of the AUC can be used to identify which NAL units, pictures, slices, or tiles belong to a given AU. The value of the AUC can correspond to different composition time instances. The AUC value can be equal to a multiple of the POC value. The AUC value can be calculated by dividing the POC value by an integer value. In some cases, the division operation can impose a certain burden on decoder implementations. In such cases, a smaller number space for the AUC value can allow a shift operation to replace the division operation. For example, the AUC value can be equal to the Most Significant Bit (MSB) value of the POC value range.

[0122] In the same embodiments, the value of the POC cycle per AU (poc_cycle_au) can be signaled in a high-level syntax structure, such as a NAL unit header, a slice header, a tile group header, an SEI message, a parameter set, or an AU delimiter. The poc_cycle_au can indicate how many different and consecutive POC values can be associated with the same AU. For example, if the value of the poc_cycle_au is equal to 4, then pictures, slices, or tiles with POC values equal to 0 to 3 (inclusive) are associated with the AU with AUC value equal to 0, and pictures, slices, or tiles with POC values equal to 4 to 7 (inclusive) are associated with the AU with AUC value equal to 1. Thus, the value of the AUC can be inferred by dividing the POC value by the value of the poc_cycle_au.

[0123] In the same or another embodiment, the value of poc_cycle_au can be derived from information located, for example, in a video parameter set (VPS) that identifies the number of spatial layers or SNR layers in the coded video sequence. Such a possible relationship is briefly described below. The derivation as described above can save several bits in the VPS, thus improving coding efficiency, advantageously, the poc_cycle_au is explicitly coded in a suitable high-level syntax structure hierarchically below the video parameter set to enable minimization of the poc_cycle_au for a given small portion of the bitstream, such as a picture. This optimization can save more bits than by the derivation process above, as the POC values (and / or values of syntax elements that indirectly refer to POCs) can be coded in low-level syntax structures.

[0124] The techniques described above for signaling adaptive resolution parameters can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 7 A computer system 700 suitable for implementing certain embodiments of the disclosed subject matter is shown in FIG. 7.

[0125] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or like mechanisms to create code that can be executed by a computer CPU, GPU, or the like.

[0126] The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, internet of things devices, and the like.

[0127] Figure 7 The components shown in FIG. 7 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of a computer system 700.

[0128] Computer system 700 can include certain human interface input devices. Such a human interface input device can be responsive to one or more human users' inputs. The human interface input device can involve obvious devices such as a keyboard 701, mouse 702, trackpad 703, touchscreen 710, data g love 704, joystick 705, microphone 706, scanner 707, camera 708, etc.

[0129] The input human interface devices can include one or more of a keyboard 701, mouse 702, trackpad 703, touchscreen 710, data glove 704, joystick 705, microphone 706, scanner 707, camera 708 (only one of each type depicted).

[0130] Computer system 700 can also include certain human interface output devices. Such human interface output devices can be stimulating one or more of the human senses to convey information to a human user.

[0131] Computer system 700 can also include human-accessible storage and their associated media such as optical media 721, including CD / DVD ROM / RW 720, thumb-drive 722, removable hard drive or solid state drive 723, etc., as well as specialized computational devices designed to be accessed like a storage device, including ROM / ASIC / PLD 724, including security dongles (not depicted), etc.

[0132] Those skilled in the art will realize that terms such as "computer readable media" or "storage media" as used in connection with the presently disclosed subject matter are to be taken extremely broadly, including, but not limited to, solid-state memories of all types (including without limitation random access memory (RAM), non-volatile memory (e.g., flash memory, solid state drives, etc.), and other memory as is well known to those of ordinary skill in the art), magnetic storage media of all types (including, but not limited to, disks of all sizes and shapes, tapes, cassettes, etc.), optical storage media of all types (including, but not limited to, compact disc CD ROMs, DVDs, Blu-ray discs, etc.), other storage media (including, but not limited to, quantum storage, holographic storage, etc.), and / or any other tangible computer storage medium.

[0133] Computer system 700 can also include an interface to one or more communication networks. Networks can for example be wireless, wireline, optical. Networks can further be local, wide-area, metropolitan, vehicular and industrial, real-time, delay-tolerant, and so on. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G, LTE and the like, TV wireline or wireless wide area networks such as cable TV, satellite TV, and terrestrial broadcast TV, vehicular networks such as CANbus, industrial local or wide area networks, and so on. Certain networks commonly require external network interface adapters that attached to certain general purpose data ports or peripheral buses (749) (such as USB ports on computer system 700; others typically integrate required network

[0134] The above human interface devices, human-accessible storage devices, and network interfaces can be attached to the core 740 of the computer system 700.

[0135] The core 740 can include one or more Central Processing Units (CPU) 741, Graphics Processing Units (GPU) 742, specialized programmable processing units in the form of Field Programmable Gate Arrays (FPGA) 743, hardware accelerators 744 for certain tasks, and so on. These devices, along with Read-only memory (ROM) 745, Random-access memory (RAM) 746, internal mass storage such as internal non-user accessible hard drives, SSDs, and the like, can be connected through a system bus 748. In some computer systems, the system bus 748 can be accessible as a

[0136] CPUs 741, GPUs 742, FPGAs 743, and accelerators 744 can execute certain instructions that, taken either alone or in combination, can make up the aforementioned computer code. This computer code can be stored in ROM 745 or RAM 746. Transitional data can be also stored in RAM 746, whereas permanent data can be stored for example, in the internal mass storage 747. Fast storage and retrieval speeds of the memory devices can be achieved through the use of cache memory, which can be closely associated with one or more CPU 741, GPU 742, mass storage 747, ROM 745, RAM 746, etc.

[0137] The computer software can be implemented as computer program products. The software described herein can be implemented in software and / or firmware. As such, program instructions can be used to implement the processes described herein. Program instructions can be stored in computer-readable media that can be accessed by a computer, such as a computer's internal or external hard drives, memory etc. In some embodiments, hard drives can be used to store software modules and data. In some embodiments, RAM can be used to store software modules and data during the execution of the software. In some embodiments, ROM can be used to store software modules and data that are read during the execution of the software. In some embodiments, a removable media can be used to store software modules and data. In some embodiments, a removable media can be used to store software modules and data that are read during the execution of the software. In some embodiments, a computer can be used to read the instructions from the computer-readable media and execute the instructions in real-time. In some embodiments, a computer can be used to read the instructions from the computer-readable media and execute the instructions not in real-time. In some embodiments, a computer can be used to read the instructions from the computer-readable media and execute the instructions in real-time and not in real-time.

[0138] By way of example, and not limitation, the computer system having architecture 700, specifically the core 740 can provide functionality as a result of processor(s) (including CPUs, GPUs, FPGA, accelerators, and the like) executing software embodied in one or more tangible, computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as introduced above, as well as certain storage of the core 740 that are of non-transitory nature, such as core-internal mass storage 747 or ROM 745. Software can be stored, for example, in RAM 746, ROM 745, or a mass storage device 747, which can receive media from an installation device 750. As one example, a computer program can be downloaded over the network to the computer system from another computer. The software implementing the embodiments of the present disclosure can be stored in RAM 746, mass storage 747, or other types of computer- readable media, before the implementation. A computer program can also be stored and / or executed using one or more computers, computers' processors, computers' cores, or other processors or cores of the core 740. The software implementing the embodiments of the present disclosure can be executed by one or more of the computer's processors 741, the computer's cores 740, or other processors or cores of the core 740. A computer program can be loaded into the computer 700 from a computer-readable medium, which can be a floppy disk, a CD-ROM, a DVD, a Blu-ray Disc, a memory stick, a magnetic tape, or a semiconductor memory, among others.

[0139] Figure 8An example of a video sequence structure with adaptive resolution change combined with temporal id values, layer id values, POC values, and AUC values is shown. In this example, pictures, slices, or tiles in a first AU with AUC = 0 can have temporal id = 0 and layer id = 0 or 1, while pictures, slices, or tiles in a second AU with AUC = 1 can have temporal id = 1 and layer id = 0 or 1. Regardless of the values of temporal id and layer id, the value of POC increases by 1 per picture. In this example, the value of poc cycle au can be equal to 2. Preferably, the value of poc cycle au can be set equal to the number of (spatially scalable) layers. Thus, in this example, the POC value increases by 2, while the AUC value increases by 1.

[0140] In the above embodiment, the entire or a subset of the inter-picture or inter-layer prediction structure and reference picture indication can be supported by using the existing reference picture set (RPS) signaling or reference picture list (RPL) signaling in HEVC. In RPS or RPL, the selected reference picture is indicated by signaling the POC value or delta value of POC between the current picture and the selected reference picture. For the disclosed subject matter, RPS and RPL can be used to indicate the inter-picture or inter-layer prediction structure without changing the signaling, but with the following restriction. If the temporal id value of the reference picture is greater than the temporal id value of the current picture, the current picture must not use the reference picture for motion compensation or other prediction. If the layer id value of the reference picture is greater than the layer id value of the current picture, the current picture must not use the reference picture for motion compensation or other prediction.

[0141] In the same and other embodiments, the POC difference based motion vector scaling for temporal motion vector prediction can be disabled across multiple pictures within an access unit. Thus, although each picture can have different POC values within an access unit, the motion vectors are not scaled and used for temporal motion vector prediction within the access unit. This is because the reference pictures with different POCs in the same AU are considered as reference pictures with the same temporal point. Thus, in an embodiment, the motion vector scaling function can return 1 when the reference picture belongs to the AU associated with the current picture.

[0142] In the same or other embodiments, when the spatial resolution of the reference picture is different from the spatial resolution of the current picture, the POC difference based motion vector scaling for temporal motion vector prediction can be optionally prohibited across multiple pictures within an access unit. When motion vector scaling is allowed, the motion vector is scaled based on the POC difference and the spatial resolution ratio between the current picture and the reference picture.

[0143] In the same or another embodiment, especially when poc_cycle_au has a non-uniform value (when vps_contant_poc_cycle_per_au == 0), the motion vector can be scaled based on the AUC difference instead of the POC difference for temporal motion vector prediction. Otherwise (when vps_contant_poc_cycle_per_au == 1), the AUC difference based motion vector scaling can be the same as the POC difference based motion vector scaling.

[0144] In the same or another embodiment, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU (with the same AUC value) as the current picture is not scaled based on the AUC difference and is not used for motion vector prediction with or without scaling based on the spatial resolution ratio between the current picture and the reference picture.

[0145] In the same or other embodiments, the AUC value is used to identify the boundary of an AU and is used for hypothetical reference decoder (HRD) operations that require input and output timing with AU granularity. In most cases, the decoded picture with the highest layer in an AU can be output for display. The AUC value and the layer_id value can be used to identify the output picture.

[0146] In an embodiment, a picture can be composed of one or more sub-pictures. Each sub-picture can cover a partial or the entire area of the picture. The area supported by one sub-picture can or can not overlap with the area supported by another sub-picture. The area composed of one or more sub-pictures can or can not cover the entire area of the picture. If a picture is composed of one sub-picture, the area supported by the sub-picture is the same as the area supported by the picture.

[0147] In the same embodiment, a sub-picture can be encoded by a similar encoding method as the encoding method used for the encoded picture. A sub-picture can be independently encoded or can be encoded in dependence on another sub-picture or the encoded picture. A sub-picture can or can not have any parsing dependency from another sub-picture or the encoded picture.

[0148] In the same embodiment, the coded sub-pictures can be contained in one or more layers. The coded sub-pictures in a layer can have different spatial resolutions. The original sub-pictures can be spatially resampled (up-sampled or down-sampled), coded with different spatial resolution parameters, and contained in the bitstream corresponding to the layer.

[0149] In the same or another embodiment, a sub-picture with (W, H) where W indicates the width of the sub-picture and H indicates the height of the sub-picture can be coded and contained in the coded bitstream corresponding to layer 0, while an up-sampled (or down-sampled) sub-picture with (W*S w,k , H*S h,k ) from the sub-picture with the original spatial resolution can be coded and contained in the coded bitstream corresponding to layer k, where S w,k and S h,k indicate the resampling ratio in the horizontal direction and the vertical direction, respectively. If the values of S w,k and S h,k are greater than 1, the resampling is equivalent to up-sampling. If the values of S w,k and S h,k are less than 1, the resampling is equivalent to down-sampling.

[0150] In the same or another embodiment, the coded sub-pictures in a layer can have different visual quality from the coded sub-pictures in another layer in the same sub-picture or in a different sub-picture. For example, sub-picture i in layer n is coded with quantization parameter Q i,n , while sub-picture j in layer m is coded with quantization parameter Q j,m .

[0151] In the same or another embodiment, the coded sub-pictures in a layer can be independently decodable without any parsing or decoding dependency on the coded sub-pictures in another layer in the same local region. A sub-picture layer that can be decoded independently without referring to another sub-picture layer in the same local region is an independent sub-picture layer. The coded sub-pictures in an independent sub-picture layer can or can not have decoding or parsing dependency on the previously coded sub-pictures in the same sub-picture layer, but the coded sub-pictures can not have any dependency on the coded pictures in another sub-picture layer.

[0152] In the same or another embodiment, the coded sub-pictures in a layer can be dependently decodable with a resolving or decoding dependency on coded sub-pictures in another layer of the same local region. A sub-picture layer that can be dependently decoded with reference to another sub-picture layer of the same local region is a dependent sub-picture layer. A coded sub-picture in a dependent sub-picture can refer to coded sub-pictures belonging to the same sub-picture, can refer to a previous coded sub-picture in the same sub-picture layer, or to both of the aforementioned reference sub-pictures.

[0153] In the same or another embodiment, a coded sub-picture consists of one or more independent sub-picture layers and one or more dependent sub-picture layers. However, for a coded sub-picture, there can be at least one independent sub-picture layer. An independent sub-picture layer can have a value of a layer identifier (layer_id) that can be present in a NAL unit header or another high-level syntax structure equal to 0. A sub-picture layer with layer_id equal to 0 is a base sub-picture layer.

[0154] In the same or another embodiment, a picture can include one or more foreground sub-pictures and one background sub-picture. The region supported by the background sub-picture can be equal to the region of the picture. The region supported by the foreground sub-pictures can overlap the region supported by the background sub-picture. The background sub-picture can be a base sub-picture layer, while the foreground sub-pictures can be non-base (enhanced) sub-picture layers. One or more non-base sub-picture layers can refer to the same base layer for decoding. Each non-base sub-picture layer with layer_id equal to a can refer to a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0155] In the same or another embodiment, a picture can include one or more foreground sub-pictures with or without a background sub-picture. Each sub-picture can have its own base sub-picture layer and one or more non-base (enhanced) layers. Each base sub-picture layer can be referred to by one or more non-base sub-picture layers. Each non-base sub-picture layer with layer_id equal to a can refer to a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0156] In the same or another embodiment, a picture can include one or more foreground sub-pictures with or without a background sub-picture. Each coded sub-picture in a (base or non-base) sub-picture layer can be referred to by one or more non-base layer sub-pictures belonging to the same sub-picture and one or more non-base layer sub-pictures not belonging to the same sub-picture.

[0157] In the same or another embodiment, the image may include one or more foreground sub-images, with or without a background sub-image. The sub-images in layer a may be further divided into multiple sub-images within the same layer. One or more coded sub-images in layer b may reference the already divided sub-images in layer a.

[0158] In the same or another embodiment, the encoded video sequence (CVS) can be a set of coded pictures. A CVS can consist of one or more coded sub-picture sequences (CSPS), where a CSPS can be a set of coded sub-pictures covering the same local region of a picture. CSPS can have the same or different temporal resolution as the encoded video sequence.

[0159] In the same or another embodiment, the CSPS can be encoded and contained in one or more layers. The CSPS can consist of one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS can reconstruct a sequence of sub-images corresponding to the same local region.

[0160] In the same embodiment or another embodiment, the number of CSPS layers corresponding to one CSPS may be the same as or different from the number of CSPS layers corresponding to another CSPS.

[0161] In the same or another embodiment, the CSPS layer may have a different temporal resolution (e.g., frame rate) than another CSPS layer. The original (uncompressed) sub-image sequence may be temporally resampled (upsampled or downsampled), encoded with different temporal resolution parameters, and included in the bitstream corresponding to the layer.

[0162] In the same or another embodiment, a sub-picture sequence with a frame rate F can be encoded and included in the encoded bitstream corresponding to layer 0, while the sub-picture sequence with a frame rate F*S... t,k A temporally upsampled (or downsampled) sub-image sequence can be encoded and included in the encoded bitstream corresponding to layer k, where S t,k The indicator layer k's temporal sampling ratio. If S t,k If the value of S is greater than 1, then the time resampling process is equivalent to frame rate upconversion. And when S... t,k When the value is less than 1, the time resampling process is equal to the frame rate downconversion.

[0163] In the same or another embodiment, when a sub-picture with CSPS layer a is referenced by a sub-picture with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b, the decoded pixels in CSPS layer a are resampled and used for the reference. The resampling process can require up-sampling filtering or down-sampling filtering.

[0164] In the same or another embodiment, Figure 9 An example of a syntax table is shown to indicate the syntax element vps_poc_cycle_au (indicating the poc_cycle_au for all pictures / slices in the coded video sequence) in VPS (or SPS); and the syntax element slice_poc_cycle_au (indicating the poc_cycle_au of the current slice in the slice header). If the POC value is uniformly increased per AU, vps_contant_poc_cycle_per_au in VPS is set to 1, and vps_poc_cycle_au is signaled in VPS. In this case, slice_poc_cycle_au is not explicitly signaled, and the AUC value per AU is calculated by dividing the POC value by vps_poc_cycle_au. If the POC value is not uniformly increased per AU, vps_contant_poc_cycle_per_au in VPS is set to 0. In this case, vps_access_unit_cnt is not signaled, and slice_access_unit_cnt is signaled in the slice header for each slice or picture. Each slice or picture can have a different slice_access_unit_cnt value. The AUC value per AU is calculated by dividing the POC value by slice_poc_cycle_au. Figure 10 A block diagram illustrating the relevant workflow is shown.

[0165] In the same or other embodiments, even if the POC value of pictures, slices or tiles can be different, the pictures, slices or tiles corresponding to the AUs with the same AUC value can be associated with the same decoding or output time instance. Therefore, without any parsing / decoding dependency across pictures, slices or tiles in the same AU, all or a subset of the pictures, slices or tiles associated with the same AU can be decoded in parallel and can be output at the same time instance.

[0166] In the same or other embodiments, even if the POC values of pictures, slices or tiles can be different, the pictures, slices or tiles corresponding to the AUs with the same AUC value can be associated with the same composition / display time point. When the composition time is included in the container format, even if the pictures correspond to different AUs, the pictures can be displayed at the same time point when the pictures have the same composition time.

[0167] In the same or other embodiments, each picture, slice or tile can have the same temporal identifier (temporal id) in the same AU. All or a subset of the pictures, slices or tiles corresponding to a time point can be associated with the same temporal sub-layer. In the same or other embodiments, each picture, slice or tile can have the same or different spatial layer id (layer id) in the same AU. All or a subset of the pictures, slices or tiles corresponding to a time point can be associated with the same or different spatial layer.

[0168] Figure 11 An example video stream is shown that includes a background CSPS with layer id equal to 0 and multiple foreground CSPS layers. An encoded sub-picture can be composed of one or more CSPS layers, while the background region not belonging to any foreground CSPS layer can be composed of a base layer. The base layer can contain both the background region and the foreground region, while the enhancement CSPS layers contain the foreground region. In the same region, the enhancement CSPS layers can have better visual quality than the base layer. The enhancement CSPS layers can refer to the reconstructed pixels and motion vectors of the base layer corresponding to the same region.

[0169] In the same or another embodiment, the video bitstream corresponding to the base layer is contained in a track of the video file, while the CSPS layers corresponding to each sub-picture are contained in separate tracks of the video file.

[0170] In the same or another embodiment, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers with the same layer id are contained in separate tracks. In this example, the track corresponding to layer k only includes the CSPS layers corresponding to layer k.

[0171] In the same or another embodiment, each CSPS layer of each sub-picture is stored in a separate track. Each track can or can not have any parsing dependency or decoding dependency on one or more other tracks.

[0172] In the same or another embodiment, each track can contain the bitstream of layer i to layer j of the CSPS layers corresponding to all or a subset of the sub-pictures, where 0 < i =< j =< k, k is the highest layer of the CSPS.

[0173] In the same or another embodiment, a picture is composed of one or more associated media data, including depth map, alpha map, 3D geometry data, occupancy map, etc. Such associated timed media data can be partitioned into one or more data sub-streams, each data sub-stream corresponding to a sub-picture.

[0174] In the same or another embodiment, Figure 12 An example of video conferencing based on multi-layer sub-picture approach is shown. In the video stream, there is one base layer video bitstream corresponding to the background picture and one or more enhancement layer video bitstreams corresponding to foreground sub-pictures. Each enhancement layer video bitstream corresponds to a CSPS layer. In the display, the picture corresponding to the base layer is displayed by default. It contains one or more users in a picture in a picture (PIP). When a specific user is selected by the control of the client, the enhancement CSPS layer corresponding to the selected user is decoded and displayed with enhanced quality or spatial resolution. Figure 13 A diagram for this operation is shown.

[0175] In the same or another embodiment, a network middle box such as a router can send a subset of layers to a user according to its bandwidth. The picture / sub-picture organization can be used for bandwidth adaptation. For example, if a user has no bandwidth, the router can strip layers or select some sub-pictures according to their importance or based on the settings used, which can be done dynamically to adapt to the bandwidth.

[0176] Figure 14 A use case of 360 video is shown. When a spherical 360 picture is projected onto a planar picture, the projected 360 picture can be partitioned into multiple sub-pictures as the base layer. Enhancement layers for specific sub-pictures can be encoded and transmitted to the client. The decoder can decode the base layer including all sub-pictures and the enhancement layer for the selected sub-picture. If the current viewport is the same as the selected sub-picture, the displayed picture adopts the decoded sub-picture with the enhancement layer, which can have higher quality. Otherwise, the decoded picture with the base layer can be displayed with low quality.

[0177] In the same or another embodiment, any layout information for display can be present in the file as supplemental information (e.g. SEI message or metadata). One or more decoded sub-pictures can be repositioned and displayed according to the signaled layout information. The layout information can be signaled by the streaming server or broadcaster, or can be regenerated by a network entity or cloud server, or can be determined by the user's customized settings.

[0178] In an embodiment, when an input picture is divided into one or more (rectangular) sub-regions, each sub-region can be coded as an independent layer. Each independent layer corresponding to a local region can have a unique layer id value. For each independent layer, the sub-picture size and position information can be signaled. For example, the picture size (width, height) and the offset information of the top-left corner (x offset, y offset) (x offset, y offset). Figure 15 An example of the layout of the divided sub-pictures, their sub-picture size and position information, and the corresponding picture prediction structure is shown. The layout information including the sub-picture size and the sub-picture position can be signaled in a high-level syntax structure such as the parameter set, the header of a slice or tile group, or a SEI message.

[0179] In the same embodiment, each sub-picture corresponding to an independent layer can have its unique POC value within an AU. When the reference pictures among the pictures stored in the DPB are indicated by using the syntax elements in the RPS or RPL structure, the POC value of each sub-picture corresponding to a layer can be used.

[0180] In the same or another embodiment, in order to indicate the (inter-layer) prediction structure, layer id can not be used, but POC (variable) value can be used.

[0181] In the same embodiment, a sub-picture corresponding to a layer (or local region) with a POC value equal to N can or can not be used as a reference picture for a sub-picture with a POC value equal to N+K, which corresponds to the same layer (or the same local region) used for motion-compensated prediction. In most cases, the value of the number K can be equal to the maximum number of (independent) layers, which can be the same as the number of sub-regions.

[0182] In the same or another embodiment, Figure 16 An extension of Figure 15 is shown. When an input picture is divided into multiple (e.g., four) sub-regions, each local region can be coded with one or more layers. In this case, the number of independent layers can be equal to the number of sub-regions, and one or more layers can correspond to a sub-region. Thus, each sub-region can be coded with one or more independent layers and zero or more non-independent layers.

[0183] In the same embodiment, in Figure 16 , an input picture can be divided into four sub-regions. The top-right sub-region can be coded as two layers, i.e., layer 1 and layer 4, while the bottom-right sub-region can be coded as two layers, i.e., layer 3 and layer 5. In this case, layer 4 can refer to layer 1 for motion-compensated prediction, while layer 5 can refer to layer 3 for motion-compensated prediction. In the same embodiment, in Figure 16 , an input picture can be divided into four sub-regions. The top-right sub-region can be coded as two layers, i.e., layer 1 and layer 4, while the bottom-right sub-region can be coded as two layers, i.e., layer 3 and layer 5. In this case, layer 4 can refer to layer 1 for motion-compensated prediction, while layer 5 can refer to layer 3 for motion-compensated prediction.

[0184] In the same or another embodiment, in-loop filtering (such as deblocking filter, adaptive in-loop filter, reconstructor, bilateral filter, or any deep learning based filter) across layer boundaries can be (optionally) disabled.

[0185] In the same or another embodiment, motion compensated prediction or in-loop filtering across layer boundaries can be (optionally) disabled.

[0186] In the same or another embodiment, the boundary padding for motion compensated prediction or in-loop filtering at subpicture boundaries can be optionally processed. A flag indicating whether the boundary padding is processed can be signaled in a high-level syntax structure such as parameter set (VPS, SPS, PPS, or APS), slice or tile group header, or SEI message.

[0187] In the same or another embodiment, the layout information of sub-regions (or sub-pictures) can be signaled in VPS or SPS. Figure 17 An example of syntax elements in VPS and SPS is shown. In this example, vps_sub_picture_dividing_flag is signaled in VPS. This flag can indicate whether the input picture is divided into multiple sub-regions. When the value of vps_sub_picture_dividing_flag is equal to 0, the input picture in the coded video sequence corresponding to the current VPS can not be divided into multiple sub-regions. In this case, the input picture size can be equal to the coded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples) which is signaled in SPS. When the value of vps_sub_picture_dividing_flag is equal to 1, the input picture can be divided into multiple sub-regions. In this case, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled in VPS. The values of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples can be equal to the width and height of the input picture, respectively.

[0188] In the same embodiment, the values of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples can not be used for decoding, but can be used for synthesis and display.

[0189] In the same embodiment, when the value of vps_sub_picture_dividing_flag is equal to 1, syntax elements pic_offset_x and pic_offset_y can be signaled in the SPS corresponding to the particular layer. In this case, the coded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS can be equal to the width and height of the sub-region corresponding to the particular layer. In addition, the position of the top-left corner of the sub-region (pic_offset_x, pic_offset_y) can be signaled in the SPS.

[0190] In the same embodiment, the position information of the top-left corner of the sub-region (pic_offset_x, pic_offset_y) can not be used for decoding, but can be used for composition and display.

[0191] In the same or another embodiment, the layout information (size and position) of all or a subset of sub-regions of an input picture, the dependency information among layers, and the relationship between a sub-region and one or more layers can be signaled in a parameter set or a SEI message. Figure 18 An example of syntax elements is shown to indicate the layout information of sub-regions, the dependency among layers, and the relationship between a sub-region and one or more layers. In this example, syntax element num_sub_region indicates the number of (rectangular) sub-regions in the current coded video sequence. Syntax element num_layers indicates the number of layers in the current coded video sequence. The value of num_layers can be greater than or equal to the value of num_sub_region. When any sub-region is coded as a single layer, the value of num_layers can be equal to the value of num_sub_region. When one or more sub-regions are coded as multiple layers, the value of num_layers can be greater than the value of num_sub_region. Syntax element direct_dependency_flag[i][j] indicates the dependency from the j-th layer to the i-th layer. num_layers_for_region[i] indicates the number of layers associated with the i-th sub-region. sub_region_layer_id[i][j] indicates the layer_id of the j-th layer associated with the i-th sub-region. sub_region_offset_x[i] and sub_region_offset_y[i] indicate the horizontal and vertical positions of the top-left corner of the i-th sub-region, respectively. sub_region_width[i] and sub_region_height[i] indicate the width and height of the i-th sub-region, respectively.

[0192] In one embodiment, one or more syntax elements can be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message, that specifies output layer sets to indicate one of a plurality of layers to be outputted with or without profile tier level information. Referring to Figure 19 A syntax element num_output_layer_sets can be signaled in a VPS that indicates the number of output layer sets (OLSs) in a coded video sequence referring to the VPS. For each output layer set, an output_layer_flag can be signaled as many as the number of output layers.

[0193] In the same embodiment, an output_layer_flag[i] equal to 1 specifies that the i-th layer is outputted. An output_layer_flag[i] equal to 0 specifies that the i-th layer is not outputted.

[0194] In the same or another embodiment, one or more syntax elements can be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message, that specifies profile tier level information for each output layer set. Still referring to Figure 19 A syntax element num_profile_tile_level can be signaled in a VPS that indicates the number of profile tier level information for each OLS in a coded video sequence referring to the VPS. For each output layer set, a set of syntax elements for profile tier level information or an index indicating a particular profile tier level information among entries of profile tier level information can be signaled as many as the number of output layers.

[0195] In the same embodiment, a profile_tier_level_idx[i][j] specifies an index of a profile_tier_level() syntax structure that is applied to the j-th layer of the i-th OLS in a list of profile_tier_level() syntax structures in a VPS.

[0196] In the same or another embodiment, referring to Figure 20 When the number of maximum layers is greater than 1 (vps_max_layers_minus1 > 0), a syntax element num_profile_tile_level and / or num_output_layer_sets can be signaled.

[0197] In the same or another embodiment, referring to Figure 20 A syntax element vps_output_layers_mode[i] indicating the mode of output layer signaling for the i-th output layer set can be present in the VPS.

[0198] In the same embodiment, vps_output_layers_mode[i] equal to 0 specifies that only the highest layer is output in the case of the i-th output layer set. vps_output_layer_mode[i] equal to 1 specifies that all layers are output in the case of the i-th output layer set. vps_output_layer_mode[i] equal to 2 specifies that the layers output in the case of the i-th output layer set are the layers for which vps_output_layer_flag[i][j] is equal to 1. More values can be reserved.

[0199] In the same embodiment, output_layer_flag[i][j] can or can not be signaled depending on the value of vps_output_layers_mode[i] for the i-th output layer set.

[0200] In the same or another embodiment, referring to Figure 20 For the i-th output layer set, a flag vps_ptl_signal_flag[i] can be present. The profile tier level information for the i-th output layer set can or can not be signaled depending on the value of vps_ptl_signal_flag[i].

[0201] In the same or another embodiment, referring to Figure 21 The number of sub-pictures in the current CVS, max_subpics_minus1, can be signaled in a high-level syntax structure (e.g., VPS, DPS, SPS, PPS, APS, or SEI message).

[0202] In the same embodiment, referring to Figure 21 When the number of sub-pictures is greater than 1 (max_subpics_minus1 > 0), a sub-picture identifier sub_pic_id[i] for the i-th sub-picture can be signaled.

[0203] In the same or another embodiment, one or more syntax elements indicating the sub-picture identifier of each layer belonging to each output layer set can be signaled in the VPS. Referring to Figure 22sub_pic_id_layer[ i ][ j ][ k ] is present in the j-th layer of the i-th output layer set. With this information, for each layer of a particular output layer set, the decoder can identify which sub-pictures can be decoded and output.

[0204] In one embodiment, a picture header (PH) is a syntax structure containing syntax elements that apply to all slices of a coded picture. A picture unit (PU) is a set of NAL units that are associated with each other according to a specified grouping rule, are consecutive in decoding order, and contain exactly one coded picture. A PU can contain a picture header (PH) as well as one or more VCL NAL units that make up a coded picture.

[0205] In one embodiment, an SPS (RBSP) is available to the decoding process prior to being referenced, the SPS (RBSP) is included in at least one AU with Temporalld equal to 0 or provided through external means.

[0206] In one embodiment, an SPS (RBSP) is available to the decoding process prior to being referenced, the SPS (RBSP) is included in at least one AU with Temporalld equal to 0 in a CVS containing one or more PPS referring to the SPS or provided through external means.

[0207] In one embodiment, an SPS (RBSP) is available to the decoding process prior to being referenced by one or more PPS, the SPS (RBSP) is included in at least one PU with nuh layer id equal to the smallest nuh layer id value of the PPS NAL units referring to the SPS NAL unit in a CVS containing one or more PPS referring to the SPS or provided through external means.

[0208] In one embodiment, an SPS (RBSP) is available to the decoding process prior to being referenced by one or more PPS, the SPS (RBSP) is included in at least one PU with Temporalld equal to 0 and nuh layer id equal to the smallest nuh layer id value of the PPS NAL units referring to the SPS NAL unit or provided through external means.

[0209] In one embodiment, an SPS (RBSP) that is available to the decoding process prior to being referenced by one or more PPSs, the SPS (RBSP) is included in at least one PU with Temporalld equal to 0 and nuh layer id equal to the smallest nuh layer id value of the PPS NAL units that reference the SPS NAL unit in the CVS containing the one or more PPSs that reference the SPS; or the SPS (RBSP) is provided through external means.

[0210] In the same or another embodiment, pps_seq_parameter_set_id specifies the value of sps_seq_parameter_set_id for the referenced SPS. The value of pps_seq_parameter_set_id can be the same in all PPSs that are referenced by coded pictures in a CLVS.

[0211] In the same or another embodiment, all SPS NAL units with a particular value of sps_seq_parameter_set_id in a CVS can have the same content.

[0212] In the same or another embodiment, SPS NAL units can share the same value space of sps_seq_parameter_set_id regardless of the value of nuh layer id.

[0213] In the same or another embodiment, the nuh layer id value of an SPS NAL unit can be equal to the smallest nuh layer id value of the PPS NAL units that reference the SPS NAL unit.

[0214] In an embodiment, when an SPS with nuh layer id equal to m is referenced by one or more PPSs with nuh layer id equal to n, the layer with nuh layer id equal to m can be the same as the layer with nuh layer id equal to n, or the same as a (direct or indirect) reference layer of the layer with nuh layer id equal to m.

[0215] In an embodiment, a PPS (RBSP) that is available to the decoding process prior to being referenced, the PPS (RBSP) is included in at least one AU with Temporalld equal to the Temporalld of the PPS NAL unit; or the SPS (RBSP) is provided through external means.

[0216] In an embodiment, a PPS (RBSP) that is available to the decoding process prior to being referenced is included in at least one AU whose Temporalld is equal to the Temporalld of the PPS NAL unit in the CVS containing the one or more PHs (or coded slice NAL units) that reference the PPS; or the SPS (RBSP) is provided through external means.

[0217] In an embodiment, a PPS (RBSP) that is available to the decoding process prior to being referenced by one or more PHs (or coded slice NAL units) is included in at least one PU whose nuh layer id is equal to the smallest nuh layer id value of the coded slice NAL units that reference the PPS NAL in the CVS containing the one or more PHs (or coded slice NAL units) that reference the PPS; or the SPS (RBSP) is provided through external means.

[0218] In an embodiment, a PPS (RBSP) that is available to the decoding process prior to being referenced by one or more PHs (or coded slice NAL units) is included in at least one PU whose Temporalld is equal to the Temporalld of the PPS NAL unit and whose nuh layer id is equal to the smallest nuh layer id value of the coded slice NAL units that reference the PPS NAL unit in the CVS containing the one or more PHs (or coded slice NAL units) that reference the PPS; or the SPS (RBSP) is provided through external means.

[0219] In the same or another embodiment, the ph_pic_parameter_set_id in a PH specifies the value of pps_pic_parameter_set_id for the referenced PPS in use. The value of pps_seq_parameter_set_id can be the same in all PPSs referenced by coded pictures in a CLVS.

[0220] In the same or another embodiment, all PPS NAL units with a particular value of pps_pic_parameter_set_id within a PU will have the same content.

[0221] In the same or another embodiment, PPS NAL units can share the same value space of pps_pic_parameter_set_id regardless of the nuh layer id value.

[0222] In the same or another embodiment, the nuh layer id value of the PPS NAL unit can be equal to the smallest nuh layer id value of the coded slice NAL units that refer to the NAL unit that refers to the PPS NAL unit.

[0223] In an embodiment, when a PPS with nuh layer id equal to m is referred to by one or more coded slice NAL units with nuh layer id equal to n, the layer with nuh layer id equal to m can be the same as the layer with nuh layer id equal to n, or the same as a (directly or indirectly) referring layer of the layer with nuh layer id equal to m.

[0224] In an embodiment, a PPS (RBSP) that is referred to before being used in the decoding process is included in at least one AU with Temporalld equal to the Temporalld of the PPS NAL unit; or the SPS (RBSP) is provided through external means.

[0225] In an embodiment, a PPS (RBSP) that is referred to before being used in the decoding process is included in at least one AU with Temporalld equal to the Temporalld of the PPS NAL unit in the CVS containing the one or more PHs (or coded slice NAL units) referring to the PPS; or the SPS (RBSP) is provided through external means.

[0226] In an embodiment, a PPS (RBSP) that is referred to by one or more PHs (or coded slice NAL units) before being used in the decoding process is included in at least one PU with nuh layer id equal to the smallest nuh layer id value of the coded slice NAL units that refer to the PPS NAL unit in the CVS containing the one or more PHs (or coded slice NAL units) referring to the PPS; or the SPS (RBSP) is provided through external means.

[0227] In an embodiment, a PPS (RBSP) that is available for the decoding process before being referenced by one or more PHs (or coded slice NAL units) is included in at least one PU with Temporalld equal to the Temporalld of the PPS NAL unit and nuh layer id equal to the smallest nuh layer id value of the coded slice NAL units that reference the PPS NAL unit in the CVS containing the one or more PHs (or coded slice NAL units) that reference the PPS; or the SPS (RBSP) is provided through external means.

[0228] In the same or another embodiment, the value of ph_pic_parameter_set_id in a PH specifies the value of pps_pic_parameter_set_id for the PPS that is being referred to in use. The value of pps_seq_parameter_set_id can be the same in all PPSs referred to by coded pictures in a CLVS.

[0229] In the same or another embodiment, all PPS NAL units with a particular value of pps_pic_parameter_set_id within a PU will have the same content.

[0230] In the same or another embodiment, PPS NAL units can share the same value space of pps_pic_parameter_set_id regardless of the nuh layer id values.

[0231] In the same or another embodiment, the nuh layer id value of a PPS NAL unit can be equal to the smallest nuh layer id value of the coded slice NAL units that reference the NAL unit that references the PPS NAL unit.

[0232] In an embodiment, when a PPS with nuh layer id equal to m is referenced by one or more coded slice NAL units with nuh layer id equal to n, the layer with nuh layer id equal to m can be the same as the layer with nuh layer id equal to n, or the same as a (direct or indirect) reference layer of the layer with nuh layer id equal to m.

[0233] In an embodiment, as Figure 22pps_subpic_id[i] specifies the subpicture ID of the i-th subpicture. The length of the pps_subpic_id[i] syntax element is pps_subpic_id_len_minus1 plus 1 bits.

[0234] For each i value in the range of 0 to sps num subpics minusl, inclusive, the variable SubpicIdVal[ i ] is derived as follows:

[0235]

[0236] In the same or another embodiment, for any two different values i and j in the range of 0 to sps num subpics minusl, inclusive, SubpicIdVal[ i ] can not be equal to SubpicIdVal[ j ].

[0237] In the same or another embodiment, when the current picture is not the first picture of a CLVS, for each i value in the range of 0 to sps num subpics minusl, inclusive, if the value of SubpicIdVal[ i ] is not equal to the value of SubpicIdVal[ i ] of the previous picture in decoding order in the same layer, all coded slice NAL units of subpictures in the current picture with subpicture index i can have nal unit type equal to a particular value in the range of IDR W RADL to CRA NUT, inclusive.

[0238] In the same or another embodiment, when the current picture is not the first picture of a CLVS, for each i value in the range of 0 to sps num subpics minusl, inclusive, if the value of SubpicIdVal[ i ] is not equal to the value of SubpicIdVal[ i ] of the previous picture in decoding order in the same layer, sps independent subpics flag can be equal to 1.

[0239] In the same or another embodiment, if the current picture is not the first picture of a CLVS, for each i value in the range of 0 to sps num subpics minusl, inclusive, sps independent subpics flag is equal to 1 or subpic treated as pic flag[ i ] and loop filter across subpic enabled flag[ i ] are equal to 1 if the value of SubpicldVal[ i ] is not equal to the value of SubpicldVal[ i ] of the previous picture in decoding order in the same layer.

[0240] In the same or another embodiment, if the current picture is not the first picture of a CLVS, for each i value in the range of 0 to sps num subpics minusl, inclusive, sps independent subpics flag is equal to 1 or subpic treated as pic flag[ i ] and loop filter across subpic enabled flag[ i ] are equal to 1 if the value of SubpicldVal[ i ] is not equal to the value of SubpicldVal[ i ] of the previous picture in decoding order in the same layer.

[0241] In the same or another embodiment, the value of the subpicture identifier of a region can be changed within a coded video sequence when a subpicture is coded independently without any reference to another subpicture.

[0242] Samples are processed in CTBs. Each luma CTB has an array size of CtbSizeY in width and height, in samples. Each chroma CTB has an array width and height of CtbWidthC and CtbHeightC, in samples, respectively. Each CTB is assigned a partition signaling to identify the size of the block used for intra or inter prediction and for transform coding. The partitioning is a recursive quadtree partitioning. The root of the quadtree is associated with the CTB. The quadtree is split until leafs are reached, which are referred to as quadtree leafs. When the component width is not an integer multiple of the CTB size, the CTB at the right component boundary is incomplete. When the component height is not an integer multiple of the CTB size, the CTB at the bottom component boundary is incomplete.

[0243] The width and height information of each subpicture can be signaled in the SPS in CtbSizeY units. In Figure 23In particular, subpic_width_minus1[ i ] plus 1 specifies the width of the i-th subpicture in units of CtbSizeY. The length of the syntax element is Ceil( Log2( ( pic_width_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) ) bits. When not present, the value of subpic_width_minus1[ i ] is inferred to be equal to ( ( pic_width_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) - subpic_ctu_top_left_x[ i ] - 1. subpic_height_minus1[ i ] plus 1 specifies the height of the i-th subpicture in units of CtbSizeY. The length of the syntax element is Ceil( Log2( ( pic_height_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) ) bits. When not present, the value of subpic_height_minus1[ i ] is inferred to be equal to ( ( pic_height_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) - subpic_ctu_top_left_y[ i ] - 1.

[0244] When the picture width is greater than or equal to CtbSizeY, the width of each subpicture can be greater than or equal to CtbSizeY. When the picture height is greater than or equal to CtbSizeY, the height of each subpicture can be greater than or equal to CtbSizeY.

[0245] If the picture width is not greater than CtbSizeY and the picture height is not greater than CtbSizeY, the picture shall not be partitioned into more than one subpicture. In this case, the number of subpictures can be equal to 1.

[0246] When pic_width_max_in_luma_samples is not greater than CtbSizeY and pic_height_max_in_luma_samples is not greater than CtbSizeY, the value of subpic_info_present_flag shall be equal to 0. When subpic_info_present_flag is equal to 0, there is no explicit signaling of subpicture partitioning information and the number of subpictures in the picture is equal to 1.

[0247] In the same or another embodiment, sps_subpic_id_len_minus1 plus 1 specifies the number of bits used to represent the syntax elements sps_subpic_id[ i ], the syntax elements pps_subpic_id[ i ] (when present), and the syntax elements slice_subpic_id (when present). The value of sps_subpic_id_len_minus1 can be in the range of 0 to 15, inclusive. The value of 1 « (sps_subpic_id_len_minus1) can be greater than or equal to sps_num_subpics_minus1 + 1.

[0248] In the same or another embodiment, when the number of sub-pictures is equal to 1, subpic_info_present_flag can be equal to 0 and sub-picture partition information can not be explicitly signaled because in this case, the sub-picture width and height information is equal to the picture width and height information and the position of the top-left corner of the sub-picture is equal to the position of the top-left corner of the picture.

[0249] For example, subpic_ctu_top_left_x[ i ] specifies the horizontal position of the CTU of the top-left corner of the i-th subpicture in units of CtbSizeY. The length of the syntax element is Ceil( Log2( ( pic_width_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) ) bits. When not present, the value of subpic_ctu_top_left_x[ i ] is inferred to be equal to 0. subpic_ctu_top_left_y[ i ] specifies the vertical position of the CTU of the top-left corner of the i-th subpicture in units of CtbSizeY. The length of the syntax element is Ceil( Log2( ( pic_height_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) ) bits. When not present, the value of subpic_ctu_top_left_y[ i ] is inferred to be equal to 0. subpic_width_minus1[ i ] plus 1 specifies the width of the i-th subpicture in units of CtbSizeY. The length of the syntax element is Ceil( Log2( ( pic_width_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) ) bits. When not present, the value of subpic_width_minus1[ i ] is inferred to be equal to ( ( pic_width_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) - subpic_ctu_top_left_x[ i ] - 1. subpic_height_minus1[ i ] plus 1 specifies the height of the i-th subpicture in units of CtbSizeY. The length of the syntax element is Ceil( Log2( ( pic_height_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) ) bits. When not present, the value of subpic_height_minus1[ i ] is inferred to be equal to ( ( pic_height_max_in_luma_samples + CtbSizeY - 1 ) » CtbLog2SizeY ) - subpic_ctu_top_left_y[ i ] - 1.

[0250] In the same or another embodiment, when the number of subpictures is greater than 1, subpic_info_present_flag can be equal to 1 and subpicture partitioning information can be explicitly signaled in the parameter set as shown in Figure 23

[0251] ​For example, in Figure 23 sps num subpics minus2 plus 2 specifies the number of subpictures per picture in a CLVS. The value of sps num subpics minus2 can be in the range of 0 to Ceil( pic width max in luma samples ÷ CtbSizeY ) * Ceil( pic height max in luma samples ÷ CtbSizeY ) - 2, inclusive. When not present, the value of sps num subpics minus2 is inferred to be equal to 0.

[0252] In the same embodiment, for i in the range of 0 to sps num subpics minus1, inclusive, the lists SubpicWidthInTiles[ i ] and SubpicHeightInTiles[ i ] specify the width and height, in tiles, of the i-th subpicture, respectively; and, for i in the range of 0 to sps num subpics minus1, inclusive, the list subpicHeightLessThanOneTileFlag[ i ] specifies whether the height of the i-th subpicture is less than one tile row, derived as follows:

[0253]

[0254] When rect slice flag is equal to 1, for i in the range of 0 to num slices in pic minus1, inclusive, the list NumCtusInSlice[ i ] specifies the number of CTUs in the i-th slice; for i in the range of 0 to num slices in pic minus1, inclusive, the list SliceTopLeftTileIdx[ i ] specifies the tile index of the tile containing the first CTU in the slice; for i in the range of 0 to num slices in pic minus1, inclusive, and j in the range of 0 to NumCtusInSlice[ i ] - 1, inclusive, the matrix CtbAddrInSlice[ i ][ j ] specifies the picture raster scan address of the j-th CTB in the i-th slice; and, the variable NumSlicesInTile[ i ] specifies the number of slices in the tile containing the i-th slice, derived as follows:

[0255]

[0256]

[0257] Two or more independently coded sub-pictures can be merged into a coded picture, such that the coded picture can be decoded and output as a single picture.

[0258] When two or more independently coded sub-pictures are merged into a coded picture, the coded picture can consist of VCL NAL units with two or more different NAL unit types.

[0259] In Figure 23 sps_independent_subpics_flag equal to 1 specifies that all sub-picture boundaries in the CLVS are treated as picture boundaries and no in-loop filtering across sub-picture boundaries. sps_independent_subpics_flag equal to 0 does not impose such a constraint. When not present, the value of sps_independent_subpics_flag is inferred to be equal to 0.

[0260] In Figure 23 subpic_treated_as_pic_flag[ i ] equal to 1 specifies that the i-th sub-picture of each coded picture in the CLVS is treated as a picture in the decoding process (excluding in-loop filtering operations). subpic_treated_as_pic_flag[ i ] equal to 0 specifies that the i-th sub-picture of each coded picture in the CLVS is not treated as a picture in the decoding process (excluding in-loop filtering operations). When not present, the value of subpic_treated_as_pic_flag[ i ] is inferred to be equal to sps_independent_subpics_flag. When subpic_treated_as_pic_flag[ i ] is equal to 1, the requirement of bitstream conformance for each output layer in the OLS that includes the layer containing the i-th sub-picture as an output layer and its reference layers is that all of the following conditions are true:

[0261] - All pictures in the output layer and its reference layers shall have the same pic_width_in_luma_samples value and the same pic_height_in_luma_samples value.

[0262] - All SPSs of an output layer and its reference layers shall have the same sps num subpics minusl value, and for each j value in the range of 0 to sps num subpics minusl, inclusive, shall have the same subpic ctu top left x[j] value, subpic ctu top left y[j] value, subpic width minusl[j] value, subpic height minusl[j] value, and loop filter across subpic enabled flag[j] value, respectively.

[0263] - All pictures in each access unit in an output layer and its reference layers shall have the same SubpicldVal[j] value for each j value in the range of 0 to sps num subpics minusl, inclusive.

[0264] In Figure 23 , loop filter across subpic enabled flag[i] equal to 1 specifies that in-loop filtering operations can be performed across the boundary of the i-th subpicture in each coded picture in a CLVS. loop filter across subpic enabled flag[i] equal to 0 specifies that in-loop filtering operations are not performed across the boundary of the i-th subpicture in each coded picture in a CLVS. When not present, the value of loop filter across subpic enabled pic flag[i] is inferred to be equal to 1 - sps independent subpics flag.

[0265] Virtual boundary information can be signaled in a parameter set (e.g., SPS, PPS), or in a picture header or slice header. When virtual boundaries exist, any in-loop filtering process across the virtual boundaries can be disabled.

[0266] In an embodiment, in Figure 24 , virtual boundary information is signaled in SPS. sps virtual boundaries enabled flag equal to 1 specifies that the disabling of in-loop filtering across virtual boundaries can apply to coded pictures in a CLVS. sps virtual boundaries enabled flag equal to 0 specifies that the disabling of in-loop filtering across virtual boundaries does not apply to coded pictures in a CLVS. In-loop filtering operations include deblocking filter, sample adaptive offset filter, and adaptive loop filter operations.

[0267] sps_virtual_boundaries_present_flag equal to 1 specifies that virtual boundary information is signaled in the SPS. sps_virtual_boundaries_present_flag equal to 0 specifies that virtual boundary information is not signaled in the SPS. When there is one or more virtual boundaries signaled in the SPS, in-loop filtering operations across the virtual boundaries in pictures referring to the SPS are prohibited. In-loop filtering operations include the deblocking filter, the sample adaptive offset filter, and the adaptive loop filter operations. The requirement for bitstream conformance is that when the value of res_change_in_clvs_allowed_flag is equal to 1, the value of sps_virtual_boundaries_present_flag shall be equal to 0. sps_num_ver_virtual_boundaries specifies the number of sps_num_ver_virtual_boundaries_pos_x[ i ] syntax elements present in the SPS. When sps_num_ver_virtual_bounders is not present, the number is inferred to be equal to 0. sps_virtual_boundaries_pos_x[ i ] specifies the position of the i-th vertical virtual boundary in units of luma samples divided by 8. The value of sps_virtual_boundaries_pos_x[ i ] shall be in the range of 1 to Ceil( pic_width_max_in_luma_samples ÷ 8 ) - 1, inclusive. sps_num_hor_virtual_boundaries specifies the number of sps_num_hor_virtual_boundaries_pos_y[ i ] syntax elements present in the SPS. When sps_num_hor_virtual_virtual_bounders is not present, the number is inferred to be equal to 0.

[0268] When sps_virtual_boundaries_enabled_flag is equal to 1 and sps_virtual_boundaries_present_flag is equal to 1, the sum of sps_num_ver_virtual_borders and sps_num_hor_virtual_borders shall be greater than 0.

[0269] sps_virtual_boundaries_pos_y[ i ] specifies the position of the i-th horizontal virtual boundary in units of luma samples divided by 8. The value of sps_virtual_boundaries_pos_y[ i ] shall be in the range of 1 to Ceil( pic_height_max_in_luma_samples ÷ 8 ) - 1, inclusive.

[0270] In an embodiment, in Figure 25 ph_virtual_boundaries_present_flag equal to 1 specifies that information of virtual boundaries is signaled in the PH. ph_virtual_boundaries_present_flag equal to 0 specifies that information of virtual boundaries is not signaled in the PH. When there is one or more virtual boundaries signaled in the PH, in-loop filtering operations across virtual boundaries in the picture are prohibited. In-loop filtering operations include deblocking filter, sample adaptive offset filter and adaptive loop filter operations. When not present, the value of ph_virtual_boundaries_present_flag is inferred to be equal to 0. The requirement of bitstream conformance is that when subpic_info_present_flag is equal to 1, the value of ph_virtual_boundaries_present_flag shall be equal to 0.

[0271] The variable VirtualBoundariesPresentFlag is derived as follows:

[0272]

[0273] ph_num_ver_virtual_boundaries specifies the number of ph_num_ver_virtual_boundaries_pos_x[ i ] syntax elements present in the PH. When ph_num_ver_virtual_boundaries is not present, the number is inferred to be equal to 0.

[0274] The variable NumverVirtualBounders is derived as follows:

[0275]

[0276] ph_virtual_boundaries_pos_x[ i ] specifies the position of the i-th vertical virtual boundary in units of luma samples divided by 8. The value of ph_virtual_boundaries_pos_x[ i ] shall be in the range of 1 to Ceil( pic_width_in_luma_samples ÷ 8 ) - 1, inclusive.

[0277] For i in the range of 0 to NumVerVirtualBoundaries - 1, inclusive, the list VirtualBoundariesPosX[ i ] specifies the position of a vertical virtual boundary in units of luma samples, which is derived as follows: for( i = 0; i < NumVerVirtualBoundaries; i++ )

[0278] VirtualBoundariesPosX[ i ] = ( sps_virtual_boundaries_present_flag?

[0279] sps_virtual_boundaries_pos_x[ i ] : ph_virtual_boundaries_pos_x[ i ] ) * 8

[0280] The distance between any two vertical virtual boundaries shall be greater than or equal to CtbSizeY luma samples.

[0281] ph_num_hor_virtual_boundaries specifies the number of ph_num_hor_virtual_boundaries_pos_y[ i ] syntax elements present in the PH. When ph_num_hor_virtual_bounders is not present, the number is inferred to be equal to 0.

[0282] The parameter NumHorVirtualBoundaries is derived as follows:

[0283]

[0284] When sps_virtual_boundaries_enabled_flag is equal to 1 and ph_virtual_boundaries_present_flag is equal to 1, the sum of ph_num_ver_virtual_borders and ph_num_hor_virtual_borders shall be greater than 0.

[0285] When a picture can be composed of VCL NAL units with different NAL unit types, the virtual boundary information can be signaled in the SPS. In this case, the virtual boundary information can not be signaled in the picture header.

[0286] When two or more coded sub-pictures are merged into a coded picture, the virtual boundary information can be overridden. If the virtual boundary information is signaled in the picture header or slice header, it complicates the merging process.

[0287] In an embodiment, when mixed_nalu_types_in_pic_flag is equal to 1, the value of ph_virtual_boundaries_present_flag shall be equal to 0.

[0288] In an embodiment, when the virtual boundary information is not signaled in the SPS, the virtual boundary information is signaled in the PPS.

[0289] In the same embodiment, for example in Figure 26 pps_virtual_boundaries_present_flag equal to 1 specifies that the virtual boundary information is signaled in the PPS. pps_virtual_boundaries_present_flag equal to 0 specifies that the virtual boundary information is not signaled in the PPS. When there is one or more virtual boundaries signaled in the PPS, in-picture cross- virtual-boundary in-loop filtering operations are prohibited. The in-loop filtering operations include the deblocking filter, the sample adaptive offset filter, and the adaptive loop filter operations. When not present, the value of pps_virtual_boundaries_present_flag is inferred to be equal to 0.

[0290] When sps_virtual_boundaries_present_flag is equal to 1, pps_virtual_boundaries_present_flag shall be equal to 0.

[0291] The requirement of bitstream conformance is that when subpic_info_present_flag is equal to 1, the value of pps_virtual_boundaries_present_flag shall be equal to 0.

[0292] The variable VirtualBoundariesPresentFlag is derived as follows:

[0293]

[0294] pps num ver virtual boundaries specifies the number of pps num ver virtual boundaries pos x [ i ] syntax elements present in the PH. When pps num ver virtual bounders is not present, the number is inferred to be equal to 0.

[0295] The variable NumverVirtualBounders is derived as follows:

[0296]

[0297] pps virtual boundaries pos x [ i ] specifies the position of the i-th vertical virtual boundary in units of luma samples divided by 8. The value of pps virtual boundaries pos x [ i ] shall be in the range of 1 to Ceil ( pic width in luma samples ÷ 8 ) - 1, inclusive.

[0298] For i in the range of 0 to NumVerVirtualBoundaries - 1, inclusive, the list VirtualBoundariesPosx [ i ] specifies the position of a vertical virtual boundary in units of luma samples, which is derived as follows:

[0299] for ( i = 0; i < NumVerVirtualBoundaries; i++ )

[0300] VirtualBoundariesPosX [ i ] = ( sps virtual boundaries present flag?

[0301] sps virtual boundaries pos x [ i ] : pps virtual boundaries pos x [ i ] ) * 8

[0302] The distance between any two vertical virtual boundaries shall be greater than or equal to CtbSizeY luma samples.

[0303] While several exemplary embodiments have been described and illustrated, it is understood that many alternatives, modifications, permutations, and equivalents exist. Therefore, the above disclosure is intended to cover all such alternatives, modifications, permutations, and equivalents as fall within the scope of this disclosure. It should be appreciated that changes could be made in the above construction without departing from the scope of the disclosure.

Claims

1. A method for encoding video data, executable by a processor, the method comprising: receiving video data, the video data comprising one or more sub-pictures having one or more virtual boundaries; signaling, in a picture parameter set, a picture header or a slice header associated with the received video data, information corresponding to the virtual boundaries; and decoding the video data based on the signaled information corresponding to the virtual boundaries; wherein, based on a detected virtual boundary, in-loop filtering across the virtual boundary is prohibited; the information of the virtual boundary is signaled in a sequence parameter set when the picture is composed of video coding layer (VCL) NAL units having different network abstraction layer (NAL) unit types, and is not signaled in the picture header. The in-loop filtering comprises one or more of a deblocking filter, a sample adaptive offset filter and an adaptive loop filter.

2. The method of claim 1, wherein, 3. The method of claim 1, further comprising: reorganizing the one or more sub-pictures, and updating the information corresponding to the virtual boundaries at a picture level. A distance between two vertical boundaries is specified to be greater than or equal to a size corresponding to luma samples associated with the received video data.

4. The method of claim 1, wherein, The information corresponding to the virtual boundaries is a picture parameter set when not signaled in a sequence parameter set.

5. The method of claim 1, wherein, 6. A computer system for encoding video data, the computer system comprising: one or more computer-readable non-transitory storage media configured to store computer program code; and one or more computer processors configured to access the computer program code and operate as instructed by the computer program code to perform the method for encoding video data according to any one of claims 1 to 5. The in-loop filtering comprises one or more of a deblocking filter, a sample adaptive offset filter and an adaptive loop filter.

7. The computer system of claim 6, wherein, 8. The computer system of claim 6, further comprising: reorganizing code configured to cause the one or more computer processors to reorganize the one or more sub-pictures, and updating code configured to cause the one or more computer processors to update the information corresponding to the virtual boundaries at a picture level. A distance between two vertical boundaries is specified to be greater than or equal to a size corresponding to luma samples associated with the received video data.

9. The computer system of claim 6, wherein, The information corresponding to the virtual boundaries is a picture parameter set when not signaled in a sequence parameter set.

10. The computer system of claim 6, wherein, 11. A computer system for encoding video data, the computer system comprising: a receiving unit configured to receive video data, the video data comprising one or more sub-pictures having one or more virtual boundaries; a signaling unit configured to signal, in a picture parameter set, a picture header or a slice header associated with the received video data, information corresponding to the virtual boundaries; and a decoding unit configured to decode the video data based on the signaled information corresponding to the virtual boundaries. ​ ​ Among them, based on the detected virtual boundary, the in-loop filtering across the virtual boundary is prohibited; when the picture is composed of video coding layer (VCL) network abstraction layer (NAL) units with different NAL unit types, information of the virtual boundary is signaled in a sequence parameter set (SPS) but not in a picture header.

12. A non-transitory computer-readable medium storing a computer program for encoding video data, the computer program configured to cause one or more computer processors to perform the method for encoding video data according to any one of claims 1-5.

13. A method of processing a video bitstream, the method comprising: The video code stream is generated according to the method for encoding video data of any one of claims 1-5. The video code stream is generated according to the method for encoding video data of any one of claims 1-5.

Images