Method for parameter set reference constraints for coded video streams - Patents.com

Abstract

To provide a technique for notification of an adaptive image size in a video bit stream.SOLUTION: There is included a method and apparatus comprising computer code configured to cause a processor or processors to perform obtaining video data comprising data of a plurality of semantically independent source pictures, determining, among the video data, whether references are associated with any of a first access unit (AU) and a second AU according to at least one picture order count (POC) signal value included with the video data, and outputting a first quantity of the references set to the first AU and a second quantity of the references set to the second AU based on the at least one POC signal value.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 954,883, filed December 30, 2019, and U.S. Patent Application No. 17 / 063,085, filed October 5, 2020, both of which are incorporated herein in their entireties.

[0002] The disclosed subject matter relates to video coding and decoding, and more particularly, according to example embodiments, to parameter set referencing and scoping in coded video streams. [Background technology]

[0003] Video coding and decoding using inter-image prediction with motion compensation has been known for decades. Uncompressed digital video can consist of a sequence of images, each with spatial dimensions of, for example, 1920 x 1080 luma samples and associated chroma samples. The sequence of images can have a fixed or variable image rate (also informally known as the frame rate), for example, 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920 x 1080 luma sample resolution at a 60 Hz frame rate) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires over 600 GBytes of storage space.

[0004] One of the goals of video coding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, sometimes by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to techniques that allow an exact copy of the original signal to be reconstructed from a compressed version of the original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signal is small enough to make the reconstructed signal useful for its intended application. For video, lossy compression is widely adopted. The amount of acceptable distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television streaming applications. The achievable compression ratio can reflect that higher tolerable / acceptable distortion can result in a higher compression ratio.

[0005] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transforms, quantization, and entropy coding, some of which are introduced below.

[0006] Historically, video encoders and decoders have tended to operate with a given picture size, most often defined and held constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture timeframe. For example, in MPEG-2, system designs have been known to change horizontal resolution (and thereby picture size) depending on factors such as scene activity, but only for I-pictures, and thus typically for GOPs. Reference picture resampling to use different resolutions within a CVS is known, for example, from ITU-T Rec. H.263 Annex P. However, here the picture size is not changed; only the reference picture is resampled, potentially resulting in only a portion of the image canvas being used (in the case of downsampling) or only a portion of the scene being captured (in the case of upsampling). Furthermore, H.263 Annex Q allows individual macroblocks to be resampled by a factor of two (upward or downward) (in each dimension). Again, the picture size remains the same. Because the macroblock size is fixed in H.263, it does not need to be signaled.

[0007] Resizing a predicted image has become more mainstream in modern video coding. For example, VP9 allows for resampling of reference images and changing the resolution of the entire image. Similarly, certain proposals made for VVC (e.g., Hendry, et. al., "On adaptive resolution change (ARC) for VVC," Joint Video Team document JVET-M0135-v1, January 9-19, 2019, incorporated herein in its entirety) allow for resampling of the entire reference image to different resolutions (higher or lower). That document proposes that different candidate resolutions be coded in the sequence parameter set and referenced by per-image syntax elements in the picture parameter set.

[0008] Compressed domain aggregation, or the extraction of multiple semantically independent image portions into a single video image, has been of interest. Particularly in the context of, for example, 360 encoding or certain surveillance applications, multiple semantically independent source images (e.g., six cubic surfaces of a 360 scene projected onto a cube, or individual camera inputs in a multi-camera surveillance setup) may require separate adaptive resolution settings to address the varying scene-specific activity at a given time. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Hendry, et. al,"On adaptive resolution change(ARC) for VVC",Joint Video Team document JVET-M0135-v1,Jan9-19, 2019 Summary of the Invention [Means for solving the problem]

[0010] Techniques for adaptive picture size signaling in a video bitstream are disclosed.

[0011] A method and apparatus are provided, comprising: a memory configured to store computer program code; and a processor or processors configured to access the computer program code and operate according to instructions in the computer program code. The computer program includes: acquisition code configured to cause at least one processor to acquire video data including data of a plurality of semantically independent source images; determination code configured to cause the at least one processor to determine, according to at least one Picture Order Count (POC) signal value included with the video data, whether a reference is associated with a first Access Unit (AU) or a second Access Unit (AU) in the video data; and output code configured to cause the at least one processor to output, based on the at least one POC signal value, a first amount of the reference set in the first AU and a second amount of the reference set in the second AU.

[0012] According to an example embodiment, the reference includes at least one of an image, a slice, and a tile of the video data.

[0013] According to an exemplary embodiment, determining whether the reference is associated with any of the first AU and the second AU includes comparing a respective POC value of each of the references to at least one POC signal value.

[0014] According to an exemplary embodiment, determining whether the reference is associated with either the first AU or the second AU further includes setting the first amount of the reference to the first AU in response to a determination that each of the first amounts of the reference includes one of a plurality of POC values that is less than the at least one POC signal value, and setting the second amount of the reference to the second AU in response to a determination that each of the second amounts of the reference includes one of a second plurality of POC values that is greater than or equal to the at least one POC signal value.

[0015] According to an exemplary embodiment, the reference comprises a slice, and the at least one POC signal value is included in a slice header of the video data.

[0016] According to an exemplary embodiment, the video data includes video parameter set (VPS) data that identifies multiple spatial layers of the video data.

[0017] According to an exemplary embodiment, at least one POC signal value is included in a video parameter set (VPS) of the video data.

[0018] According to an exemplary embodiment, the determining code is further configured to cause the at least one processor to determine whether the VPS data includes at least one flag indicating whether one or more of the references are divided into multiple sub-regions, and if the at least one flag indicates that one or more of the references are divided into multiple sub-regions, determine at least one of a total image width and a total image height, in luminance samples, of one or more images of the references.

[0019] According to an exemplary embodiment, the determination code is further configured to cause the at least one processor to determine, if the at least one flag indicates that one or more of the references are divided into multiple sub-regions, a notification value that specifies an offset of a portion of at least one of the sub-regions included in a sequence parameter set of the video data.

[0020] According to an exemplary embodiment, the multiple semantically independent source images represent a spherical 360° image.

[0021] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 2] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 3] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 5A] As shown, a schematic diagram of options for signaling ARC parameters according to prior art or embodiments. [Figure 5B] As shown, a schematic diagram of options for signaling ARC parameters according to prior art or embodiments. [Figure 6] 1 is an example of a syntax table according to one embodiment. [Figure 7] FIG. 1 is a schematic diagram of a computer system according to one embodiment. [Figure 8] 1 is an example of a prediction structure for scalability with adaptive resolution change. [Figure 9] 1 is an example of a syntax table according to one embodiment. [Figure 10] 1 is a simplified block diagram schematic of a Picture Order Count (POC) cycle per access unit and parsing and decoding of access unit count values; [Figure 11] 1 is a schematic diagram of a video bitstream structure including multi-layer sub-images. [Figure 12] FIG. 10 is a schematic diagram of a display of a selected sub-image with increased resolution. [Figure 13] FIG. 1 is a block diagram of a process for decoding and displaying a video bitstream containing multi-layer sub-images. [Figure 14] FIG. 1 is a schematic diagram of a 360° video display with a sub-image enhancement layer. [Figure 15]1 is an example of sub-picture layout information and corresponding layer and picture prediction structures. [Figure 16] 1 is an example of sub-picture layout information and corresponding layer and picture prediction structure, with local region spatial scalability modality. [Figure 17] 10 is an example of a syntax table of sub-image layout information. [Figure 18] 10 is an example of a syntax table of a sub-image layout information SEI message. [Figure 19] 10 is an example of a syntax table showing output layers and profile / tier / level information for each output layer set. [Figure 20] 10 is an example of a syntax table showing the output layer mode of each output layer set. [Figure 21] 10 is an example syntax table showing the current sub-image of each layer of each output layer set. [Figure 22] 10 is an example of parameter set reference in a non-reference layer. DETAILED DESCRIPTION OF THE INVENTION

[0023] Because compressed domain aggregation, or the extraction of multiple semantically independent image portions into a single video image, is of interest, for example, in the context of 360 coding or certain surveillance applications, multiple semantically independent source images (e.g., six cubic surfaces of a 360 scene projected onto a cube, or individual camera inputs in a multi-camera surveillance setup) may require separate adaptive resolution settings to address the various scene-specific activity at a given time. Thus, disclosed herein, among other things, is an encoder that can choose to use different resampling factors for the different semantically independent images that make up a 360° or entire surveillance scene at a given time. Once combined into a single image, resampling of the reference image is then performed, and adaptive resolution coding notification must be available for the portions of the coded image.

[0024] 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110 and 120 interconnected via a network 150. In the case of one-way data transmission, the first terminal 110 may encode video data at a local location for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the other terminal's coded video data from the network 150, decode the coded data, and display the recovered video data. One-way data transmission may be common in media serving applications, etc.

[0025] 1 shows a second pair of terminals 130, 140 provided to support two-way transmission of coded video, such as might occur during a video conference. For two-way transmission of data, each terminal 130, 140 can code video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 can also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0026] In the example of FIG. 1 , terminals 110, 120, 130, and 140 may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure need not be so limited. Embodiments of the present disclosure apply to laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network 150 represents any number of networks that convey coded video data between terminals 110, 120, 130, and 140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 150 may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0027] 2 illustrates the placement of a video encoder and decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0028] The streaming system may include a capture subsystem 213, which may include a video source 201, such as a digital camera, that generates an uncompressed video sample stream 202. The sample stream 202, shown as a thick line to emphasize its large amount of data compared to an encoded video bitstream, may be processed by an encoder 203 connected to the camera 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream 204, shown as a thin line to emphasize its lower amount of data compared to the sample stream, may be stored on a streaming server 205 for future use. One or more streaming clients 206, 208 may access the streaming server 205 to obtain copies 207, 209 of the encoded video bitstream 204. The client 206 may include a video decoder 210 that decodes the input copy of the encoded video bitstream 207 and creates an output video sample stream 211 that can be rendered on a display 212 or other rendering device (not shown). In some streaming systems, the video bitstreams 204, 207, 209 may be encoded according to a particular video coding / compression standard. Examples of these standards include ITU-T Recommendation H.265. The developing video coding standard is informally known as Versatile Video Coding, or VVC. The disclosed subject matter may be used in the context of VVC.

[0029] FIG. 3 may be a functional block diagram of a video decoder 210 according to one embodiment of the present invention.

[0030] The receiver 310 may receive one or more coded video sequences to be decoded by the decoder 210. In the same or another embodiment, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel 312, which may be a hardware / software link to a storage device that stores the encoded video data. The receiver 310 may receive encoded video data with other data, such as coded audio data and / or auxiliary data streams, which may be transferred to each other using entities (not shown). The receiver 310 may separate the coded video sequences from the other data. To combat network jitter, a buffer memory 315 may be coupled between the receiver 310 and the entropy decoder / parser 320 (hereinafter "parser"). When the receiver 310 is receiving data from a store-and-forward device of sufficient bandwidth and controllability or from a synchronous network, the buffer 315 may not be needed or may be small. For use in a best effort packet network such as the Internet, buffer 315 may be required and can be relatively large, and advantageously can be adaptively sized.

[0031] The video decoder 210 may include a parser 320 to reconstruct symbols 321 from the entropy-coded video sequence. These symbol categories include information used to manage the operation of the decoder 210 and, potentially, information for controlling a rendering device, such as a display 212, which is not an integral part of the decoder but may be coupled to the decoder, as shown in FIG. 2. The rendering device control information may be in the form of a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not shown). The parser 320 can parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 320 can extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include Group of Pictures (GOP), images, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The entropy decoder / parser may also extract coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0032] Parser 320 may perform entropy decoding / parsing operations on the video sequence received from buffer 315 to create symbols 321 .

[0033] The reconstruction of symbols 321 may involve multiple different units, depending on the type of coded video picture or portion thereof (e.g., inter- and intra-pictures, inter- and intra-blocks, etc.), and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by parser 320. The flow of such subgroup control information between parser 320 and the following units is not shown for clarity.

[0034] Beyond the functional blocks already mentioned, decoder 210 may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate:

[0035] The first unit is a scalar / inverse transform unit 351. The scalar / inverse transform unit 351 receives quantized transform coefficients as well as control information including which transform to use, block size, quantization coefficients, quantization scaling matrix, etc. as symbols 321 from the parser 320. It may output blocks comprising sample values that may be input to an aggregator 355.

[0036] In some cases, the output samples of the scaler / inverse transform unit 351 may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image but can use prediction information from a previously reconstructed portion of the current image. Such prediction information may be provided by an intra-image prediction unit (352). In some cases, the intra-image prediction unit 352 generates blocks of the same size and shape as the block being reconstructed using surrounding already reconstructed information taken from the current (partially reconstructed) image 356. The aggregator 355 may optionally add the prediction information generated by the intra-prediction unit 352 to the output sample information provided by the scaler / inverse transform unit 351 on a sample-by-sample basis.

[0037] In other cases, the output samples of the scalar / inverse transform unit 351 may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion-compensated prediction unit 353 may access the reference picture memory 357 to retrieve samples used for prediction. After motion-compensating the retrieved samples according to the symbols 321 associated with the block, these samples may be added by the aggregator 355 to the output of the scalar / inverse transform unit (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory from which the motion compensation unit retrieves prediction samples may be controlled by motion vectors available to the motion compensation unit in the form of symbols 321, which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values retrieved from the reference picture memory when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0038] The output samples of aggregator 355 may have various loop filtering techniques applied in loop filter unit 356. Video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video bitstream and made available to loop filter unit 356 as symbols 321 from parser 320, but may also respond to meta-information obtained during decoding of a previous (in decode order) part of the coded image or coded video sequence, or may respond to previously reconstructed, loop-filtered sample values.

[0039] The output of the loop filter unit 356 may be a sample stream that may be output to the render device 212 and may also be stored in the reference image memory 356 for use in future inter-image prediction.

[0040] Once fully reconstructed, a particular coded picture can be used as a reference picture for future predictions. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 320), the current reference picture 356 can become part of reference picture buffer 357, and the new current picture memory can be relocated before beginning reconstruction of a subsequent coded picture.

[0041] The video decoder 320 may perform decoding operations according to a predetermined video compression technology, which may be documented in a standard such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technology or standard being used, in the sense of conforming to the syntax of the video compression technology or standard as specified in the video compression technology document or standard and, in particular, the profile documents therein. Compliance may also require that the complexity of the coded video sequence be within the boundaries defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further constrained by a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0042] In one embodiment, the receiver 310 may receive additional (redundant) data with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder 320 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0043] FIG. 4 may be a functional block diagram of the video encoder 203 according to one embodiment of the present disclosure.

[0044] The encoder 203 may receive video samples from a video source 201 (not part of the encoder) that may capture video images that are coded by the encoder 203 .

[0045] The video source 201 may provide a source video sequence to be coded by the encoder 203 in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source 201 may be a storage device that stores pre-prepared video. In a video conferencing system, the video source 203 may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed sequentially, impart motion. The image itself may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0046] According to one embodiment, the encoder 203 can encode and compress images of a source video sequence into a coded video sequence 443 in real time or under any other time constraint required by the application. Enforcing an appropriate coding rate is one function of the controller 450. The controller 450 controls and is operatively coupled to other functional units, as described below. Couplings are not shown for clarity. Parameters set by the controller may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, ...), picture size, Group of Pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can readily identify other functions of the controller 450 as they may pertain to optimizing the video encoder 203 for a particular system design.

[0047] Some video encoders operate in a manner that those skilled in the art can easily recognize as a "coding loop." As an overly simplified explanation, the coding loop may consist of an encoding portion of the encoder 430 (hereinafter, the "source coder") (responsible for creating symbols based on the input image to be coded and the reference image), and a (local) decoder 433 embedded in the encoder 203 that reconstructs the symbols to create sample data that the (remote) decoder also creates (because in the video compression techniques considered in the disclosed subject matter, the compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream is input to a reference image memory 434. Because decoding the symbol stream yields bit-exact results regardless of the decoder location (local or remote), the reference image buffer contents are also bit-exact between the local and remote encoders. In other words, the predictor that the encoder "sees" as the reference image samples the exact same sample values that the decoder "sees" when using prediction during decoding. This basic principle of reference image synchrony (and the resulting drift when synchrony cannot be maintained, eg, due to channel errors) is well known to those skilled in the art.

[0048] The operation of the "local" decoder 433 may be the same as the operation of the "remote" decoder 210, which has already been described in detail in connection with Figure 3. However, with brief reference also to Figure 3, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder 445 and parser 320 may be lossless, the entropy decoding portion of the decoder 210, including the channel 312, the receiver 310, the buffer 315, and the parser 320, may not be fully implemented in the local decoder 433.

[0049] An observation that can be made at this point is that any decoder technology, with the exception of parsing / entropy decoding, that is present in the decoder must also be present in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on decoder operation. Descriptions of encoder technology can be omitted since they are the inverse of the decoder technology that has been comprehensively described. Only in certain areas are more detailed descriptions required and are provided below.

[0050] As part of its operation, the source coder 430 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence designated as “reference frames.” In this manner, the coding engine 432 codes differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as predictive references for the input frame.

[0051] The local video decoder 433 may decode coded video data of frames that may be designated as reference frames based on symbols generated by the source coder 430. The operation of the coding engine 432 may advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 4), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder 433 may replicate the decoding process that may be performed on the reference frames by the video decoder and store the reconstructed reference frames in the reference picture cache 434. In this way, the encoder 203 can locally store copies of reconstructed reference frames that have common content as reconstructed reference frames that will be retrieved by the far-end video decoder (without transmission errors).

[0052] The predictor 435 may perform the prediction search of the coding engine 432. That is, for a new frame to be coded, the predictor 435 may search the reference picture memory 434 for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may serve as suitable prediction references for the new image. The predictor 435 may operate on a pixel block by sample block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor 435, the input image may have prediction references drawn from multiple reference pictures stored in the reference picture memory 434.

[0053] Controller 450 may manage the coding operations of video coder 430, including, for example, setting parameters and subgroup parameters used to encode video data.

[0054] The output of all the aforementioned functional units may undergo entropy coding in entropy coder 445. The entropy coder converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques well known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0055] The transmitter 440 may buffer the coded video sequence produced by the entropy coder 445 to prepare it for transmission over a communication channel 460, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter 440 may merge the coded video data from the video coder 430 with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0056] A controller 450 may manage the operation of the encoder 203. During encoding, the controller 450 may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures are often assigned as one of the following frame types:

[0057] An intra-picture (I-picture) may be one that can be coded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0058] A predicted image (P-image) may be an image that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values of each block.

[0059] A bidirectionally predicted image (B-image) may be an image that can be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values of each block. Similarly, a multi-predicted image may use more than two reference images and associated metadata for the reconstruction of a single block.

[0060] A source image is generally spatially subdivided into multiple sample blocks (e.g., 4x4, 8x8, 4x8, or 16x16 blocks each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by a coding assignment applied to each image of the block. For example, blocks of an I image may be nonpredictively coded or predictively coded with reference to already coded blocks of the same image (spatial prediction or intra-prediction). Pixel blocks of a P image may be nonpredictively coded via spatial prediction with reference to one previously coded reference image, or via temporal prediction. Blocks of a B image may be nonpredictively coded via spatial prediction with reference to one or two previously coded reference images, or via temporal prediction.

[0061] Video coder 203 may perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In its operation, video coder 203 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0062] In one embodiment, transmitter 440 can transmit additional data along with the encoded video. Video coder 4 can include such data as part of the coded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0063] Before describing certain aspects of the disclosed subject matter in more detail, it is necessary to introduce certain terms that will be referenced in the remainder of this description.

[0064] Hereinafter, a sub-image refers to a rectangular arrangement of samples, blocks, macroblocks, coding units, or similar entities that may be semantically grouped and individually coded at varying resolutions. One or more sub-images may be used for an image. One or more coded sub-images may form a coded image. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from an image. In certain circumstances, one or more coded sub-images may be assembled into an image coded in the compressed domain without transcoding to the sample level, and in the same or certain other cases, one or more coded sub-images may be extracted from a coded image in the compressed domain.

[0065] Hereinafter, adaptive resolution change (ARC) refers to a mechanism that can change the resolution of an image or sub-image in a coded video sequence, for example, by resampling a reference image. Hereinafter, ARC parameters refer to the control information needed to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, output and / or reference image resolutions, various control flags, etc.

[0066] The above discussion has focused on the coding and decoding of a single, semantically independent coded video picture. Before discussing the implications of coding / decoding multiple sub-pictures with independent ARC parameters and the additional complexity that this implies, it is necessary to discuss options for signaling ARC parameters.

[0067] Referring to Figure 5, several new options for signaling ARC parameters are shown. As described for each option, these have specific advantages and disadvantages in terms of coding efficiency, complexity, and architecture. A video coding standard or technology may choose one or more of these options, or options known from previous technologies, to signal ARC parameters. The options may not be mutually exclusive and may possibly be interchanged based on application needs, relevant standard technologies, or encoder choice.

[0068] Classes of ARC parameters can include: -Up / downsample factor, separate or combine in X and Y dimensions -Up / downsampling factor with added temporal dimension, indicating constant speed zoom in / out for a given number of images - Either of the above two may involve the coding of one or more possibly short syntax elements that may point to a table containing the factors. Resolution in the X or Y dimension of the input image, output image, reference image, coded image (combined or separately), sample, block, macroblock, CU, or any other suitable unit of granularity. If there are multiple resolutions (e.g., one for the input image and one for the reference image), in some cases one set of values may be inferred from another set of values. This can be gated, for example, using flags. See below for more detailed examples. - "Warping" coordinates are similar to those used in H.263 Annex P, and are denoted at the preferred granularity as described above. H.263 Annex P defines one efficient way to code such warping coordinates, but other, potentially more efficient ways may be envisioned. For example, Annex P's variable-length reversible "Huffman"-style coding of warping coordinates could be replaced by a preferred-length binary coding. The length of the binary codeword could, for example, be derived from the maximum image size, possibly multiplied by a specific factor and offset by a specific value to allow "warping" outside the bounds of the maximum image size. - Up- or down-sampling filter parameters. In the simplest case, there may be only one filter for up-sampling and / or down-sampling. However, in some cases, it may be advantageous to have more flexibility in the filter design, and signaling of filter parameters may be necessary. Such parameters may be selected from an index into a list of possible filter designs, the filter may be fully specified (e.g., via a list of filter coefficients using an appropriate entropy coding technique), or the filter may be selected implicitly by the up- / down-sampling ratio, which in turn is signaled according to one of the aforementioned mechanisms.

[0069] Hereafter, this description assumes the coding of a finite set of up / downsample factors (the same factors used in both the X and Y dimensions) denoted by a codeword. The codeword may be advantageously coded with a variable length, for example, using Exponential-Golomb codes common to certain syntax elements of video coding specifications such as H.264 and H.265. One suitable mapping of values to up / downsample coefficients may, for example, follow the table below:

[0070] [Table 1]

[0071] Many similar mappings can be devised, depending on the needs of the application and the capabilities of the upscaling and downscaling mechanisms available in the video compression technology or standard. The table can be extended to even more values. The values can also be represented using entropy coding mechanisms other than Exponential-Golomb coding, for example, binary coding. This may have particular advantages when the resampling factor is of interest outside the video processing engine itself (mainly the encoder and decoder), for example, by a MANE. Note that in the (probably) most common case where no resolution change is required, a short, 1-bit Exponential-Golomb coding can be chosen in the table above. This may be more coding efficient than using binary coding in the most common case.

[0072] The number of entries in the table and their semantics may be fully or partially configurable. For example, the basic outline of the table may be conveyed in a "high" parameter set, such as a sequence or decoder parameter set. Alternatively, or in addition, one or more such tables may be defined in a video coding technology or standard and selected, for example, via a decoder or sequence parameter set.

[0073] Below we discuss how to include the above coded upsample / downsample coefficients (ARC information) in a video coding technique or standard syntax. Similar considerations may apply to one or several codewords that control the up / downsample filters. See below for a discussion of when a filter or other data structure requires a relatively large amount of data.

[0074] H.263 Annex P, shown at 500A in the figure, includes ARC information 502 in the form of four warping coordinates in a picture header 501, specifically the H.263 PLUSPTYPE 503 header extension. This can be a wise design choice when a) there is a picture header available and b) frequent changes to the ARC information are expected. However, the overhead of using H.263-style signaling can be very high, and because picture headers can be transient in nature, scaling factors may not be related to picture boundaries.

[0075] The JVCET-M135-v1 standard, mentioned above, includes ARC reference information 505 (index) located in a picture parameter set 504, which indexes into a table 506 containing target resolutions located in sequence parameter set 507. The placement of possible resolutions in table 506 within sequence parameter set 507 can be justified, according to verbal statements by the author, by using SPS as an interoperability negotiation point during capability exchange. Resolution can be changed per picture within the limits set by the values in table 506 by referencing the appropriate picture parameter set 504.

[0076] Continuing with reference to Figure 5, the following additional options may exist for conveying ARC information in a video bitstream: Each of these options has certain advantages over existing techniques, as discussed above. Options may coexist within the same video coding technology or standard.

[0077] In one embodiment, as shown in FIG. 500B, ARC information 509, such as a resampling (zoom) factor, may be present in a slice header, GOB header, tile header, or tile group header (hereafter, tile group header) 508. This may be appropriate when the ARC information is small, such as a single variable-length ue(v) or a fixed-length codeword of a few bits, as described above. Having the ARC information directly in the tile group header has the added advantage that the ARC information can apply to, for example, a sub-image represented by that tile group, rather than the entire image. See also below. Furthermore, even if a video compression technology or standard only contemplates whole-image adaptive resolution change (e.g., as opposed to tile-group-based adaptive resolution change), placing the ARC information in the tile group header rather than in an H.263-style image header has certain advantages in terms of error resilience.

[0078] In the same or another embodiment, the ARC information 512 itself may reside in an appropriate parameter set 511, such as a picture parameter set, a header parameter set, a tile parameter set, an adaptive parameter set, etc. (An adaptive parameter set is depicted). The scope of that parameter set advantageously cannot be larger than an image, e.g., a tile group. Use of the ARC information is implicit by activation of the associated parameter set. For example, if a video coding technology or standard only envisions picture-based ARC, a picture parameter set or equivalent may be appropriate.

[0079] In the same or another embodiment, the ARC reference information 513 may reside in a tile group header 514 or similar data structure, which may reference a subset of the ARC information 515 available in a parameter set 516 having a scope beyond a single image, such as a sequence parameter set or a decoder parameter set.

[0080] As used in JVET-M0135-v1, the additional level of indirection implicitly activating the PPS from the tile group header, PPS, and SPS seems unnecessary because picture parameter sets, like sequence parameter sets, are used for feature negotiation or announcement (with specific standards such as RFC3984). However, if the ARC information needs to be applicable to sub-pictures represented by a tile group as well, a parameter set with activation scope limited to the tile group, such as an adaptive parameter set or a header parameter set, may be a better choice. Also, if the ARC information exceeds a negligible size (e.g., if it contains filter control information such as a large number of filter coefficients), parameters may be a better choice than using the header 508 directly from the perspective of coding efficiency, since these settings can be reused in future pictures or sub-pictures by referencing the same parameter set.

[0081] When using a sequence parameter set or another higher parameter set with a scope spanning multiple images, certain considerations may apply. 1. The parameter set that stores the ARC information table 516 is sometimes a sequence parameter set, but in other cases, it is advantageously a decoder parameter set. A decoder parameter set can have an activation scope of multiple CVSs, i.e., the coded video stream, i.e., all coded video bits from the start of the session to the end of the session. Possible ARC factors could be decoder features that may be implemented in hardware, and such a scope is more appropriate because hardware features tend not to change with CVSs (at least in some entertainment systems, Group of Pictures, which are one second or less in length). That said, placing the table in a sequence parameter set is explicitly included in the deployment options described here, especially in connection with point 2 below. 2. The ARC reference information 513 can be advantageously placed directly in the picture / slice / tile / GOB / tile group header (hereafter referred to as the tile group header) 514, rather than in the picture parameter set as in JVCET-M0135-v1. The reason is as follows: If an encoder wants to change a single value in the picture parameter set, such as the ARC reference information, it needs to create a new PPS and reference the new PPS. Only the ARC reference information changes; other information, such as quantization matrix information in the PPS, remains. Such information can be significant in size and must be retransmitted to complete the new PPS. Since the ARC reference information can be a single codeword, such as an index into table 513, and is the only value that changes, it would be cumbersome and wasteful to retransmit all of the quantization matrix information, for example. This can be significantly more efficient in terms of coding, as it avoids indirect referencing via the PPS, as proposed in JVET-M0135-v1. Similarly, putting the ARC reference information in the PPS has the further disadvantage that the ARC information referenced by the ARC reference information 513 necessarily needs to be applied to the entire image, not a sub-image, since the scope of the image parameter set activation is the image.

[0082] In the same or another embodiment, signaling of ARC parameters may follow the detailed example outlined in Figure 6. Figure 6 shows a syntax diagram 600 in a representation used in video coding standards since at least 1993. The notation of such syntax diagrams loosely follows C-style programming. Bolded lines indicate syntax elements present in the bitstream, while non-bolded lines often indicate control flow or variable setting.

[0083] The tile group header 601, as an example syntax structure for a header applicable to a (possibly rectangular) portion of an image, can include the conditionally variable-length Exponential-Golomb coding syntax element dec_pic_size_idx 602 (shown in bold). The presence of this syntax element in the tile group header can gate the use of adaptive resolution 603. Here, a flag value not shown in bold means that the flag is present in the bitstream at the point where it occurs in the syntax diagram. Whether adaptive resolution is used for this image or part of it can be signaled in any higher-level syntax structure, inside or outside the bitstream. In the example shown, it is signaled in the sequence parameter set, as outlined below.

[0084] 6, an excerpt of a sequence parameter set 610 is also shown. The first syntax element shown is adaptive_pic_resolution_change_flag 611. If true, that flag can indicate the use of adaptive resolution, which may require specific control information. In this example, such control information is present conditionally based on the value of the flag based on an if() statement in the parameter set 612 and the tile group header 601.

[0085] If adaptive resolution is used, in this example, it is the output resolution in samples 613 that is coded. The number 613 refers to both output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can define the resolution of the output image. Elsewhere in the video coding technology or standard, specific limits on either value can be defined. For example, a level definition might limit the total number of output samples, which could be the product of the values of these two syntax elements. Also, a particular video coding technology or standard, or an external technology or standard such as a system standard, might limit the numbering range (e.g., one or both dimensions must be divisible by a power of two) or the aspect ratio (e.g., width and height must have a relationship such as 4:3 or 16:9). Such limits may be introduced to facilitate hardware implementation or for other reasons and are well known in the art.

[0086] In certain applications, it may be advisable for an encoder to instruct a decoder to use a particular reference picture size rather than implicitly assuming that size as the output picture size. In this example, the syntax element reference_pic_size_present_flag 614 gates the conditional presence of reference picture dimensions 615 (again, the numbers refer to both width and height).

[0087] Finally, a table of possible decoded image widths and heights is shown. Such a table can be represented, for example, by the table directive (num_dec_pic_size_in_luma_samples_minus1) 616. "minus1" can refer to the interpretation of the value of that syntax element. For example, if the coded value is zero, there is one table entry. If the value is 5, there are six table entries. For each "row" in the table, the width and height of the decoded image are included in the syntax 617.

[0088] The table entries presented in 617 can be indexed using the syntax element dec_pic_size_idx 602 in the tile group header, thereby allowing for different decoded sizes (actually zoom factors) per tile group.

[0089] Certain video coding technologies or standards, e.g., VP9, support spatial scalability by implementing a specific form of reference picture resampling (signaled in a completely different manner than the disclosed subject matter) in combination with temporal scalability to enable spatial scalability. In particular, certain reference pictures may be upsampled to higher resolutions using ARC-style techniques to form the base of spatial enhancement layers. These upsampled pictures can then be refined using regular prediction mechanisms at higher resolutions to add additional detail.

[0090] The disclosed subject matter can be used in such environments. In some cases, in the same or another embodiment, values in the NAL unit header, such as the Temporal ID field, can be used to indicate not only temporal layers but also spatial layers. Doing so has certain advantages for certain system designs. For example, existing Selected Forwarding Units (SFUs) are created and optimized for selective forwarding of temporal layers based on the NAL unit header. The Temporal ID value can be used without modification for scalable environments. To enable this, a mapping between coded picture sizes and temporal layers may need to be indicated by the Temporal ID field in the NAL unit header.

[0091] In some video coding techniques, an access unit (AU) can refer to a coded picture, slice, tile, or NAL unit that is captured and organized into a respective picture / slice / tile / NAL unit bitstream at a given point in time. That instance of time may be the organization time.

[0092] In HEVC and certain other video coding technologies, a picture order count (POC) value can be used to indicate a reference picture selected from multiple reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) includes one or more pictures, slices, or tiles, each picture, slice, or tile belonging to the same AU can have the same POC value, indicating that they were created from content at the same composition time. In other words, in a scenario where two pictures / slices / tiles have the same given POC value, this may indicate two pictures / slices / tiles that belong to the same AU and have the same composition time. Conversely, two pictures / tiles / slices with different POC values may indicate pictures / slices / tiles that belong to different AUs and have different composition times.

[0093] In one embodiment of the disclosed subject matter, the aforementioned strict relationship can be relaxed in that an access unit can contain pictures, slices, or tiles with different POC values. By allowing different POC values within an AU, it becomes possible to use POC values to identify potentially independently decodable pictures / slices / tiles with the same display time. This, in turn, can enable support for multiple scalable layers without modifying reference picture selection signals (e.g., reference picture set signals or reference picture list signals), as described in more detail below.

[0094] However, it is still desirable to be able to identify, from the POC value alone, the AU to which an image / slice / tile belongs with respect to other images / slices / tiles with different POC values. This can be achieved as described below.

[0095] In the same or other embodiments, the access unit count (AUC) may be signaled in a high-level syntax structure such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or AU delimiter. The AUC value can be used to identify which NAL unit, picture, slice, or tile belongs to a given AU. The AUC value may correspond to a distinct configuration time instance. The AUC value may be equal to a multiple of the POC value. The AUC value can be calculated by dividing the POC value by an integer value. In some cases, the division operation may impose a certain burden on the decoder implementation. In such cases, due to slight limitations on the numbering space of AUC values, the division operation can be replaced with a shift operation. For example, the AUC value may be equal to the most significant bit (MSB) value of the POC value range.

[0096] In the same embodiment, the value of the POC cycle per AU (poc_cycle_au) may be signaled in a high-level syntax structure such as a NAL unit header, a slice header, a tile group header, an SEI message, a parameter set, or an AU delimiter. poc_cycle_au may indicate the number of different and consecutive POC values that can be associated with the same AU. For example, if the value of poc_cycle_au is 4, images, slices, or tiles with POC values of 0 to 3 are associated with AUs with an AUC value of 0, and images, slices, or tiles with POC values of 4 to 7 are associated with AUs with an AUC value of 1. Therefore, the value of AUC can be inferred by dividing the POC value by the value of poc_cycle_au.

[0097] In the same or another embodiment, the value of poc_cycle_au may be derived, for example, from information located in a video parameter set (VPS) that identifies the number of spatial or SNR layers in the coded video sequence. Such possible relationships are briefly described below. While the above derivation may save a few bits in the VPS and improve coding efficiency, it may be advantageous to explicitly code poc_cycle_au in an appropriate higher-level syntax structure hierarchically below the video parameter set so that poc_cycle_au can be minimized for specific, small portions of the bitstream, such as images. This optimization may save more bits than the above derivation process allows, since the POC value (and / or the values of syntax elements that indirectly reference POC) may be coded in a lower-level syntax structure.

[0098] In the same or another embodiment, Figure 9 shows an example of a syntax table 900 signaling the syntax element vps_poc_cycle_au in a VPS (or SPS), which indicates the poc_cycle_au used for all pictures / slices in the coded video sequence, and the syntax element slice_poc_cycle_au indicating the poc_cycle_au of the current slice in the slice header. If the POC value increases uniformly per AU, vps_contant_poc_cycle_per_au in the VPS is set to 1, and vps_poc_cycle_au is signaled in the VPS. In this case, slice_poc_cycle_au is not explicitly signaled, and the AUC value for each AU is calculated by dividing the POC value by vps_poc_cycle_au. If the POC value does not increase uniformly per AU, vps_contant_poc_cycle_per_au in the VPS is set to 0. In this case, vps_access_unit_cnt is not signaled, but slice_access_unit_cnt is signaled in the slice header of each slice or image. Each slice or image may have a different value of slice_access_unit_cnt. The AUC value for each AU is calculated by dividing the POC value by slice_poc_cycle_au. Figure 10 shows a block diagram illustrating the related workflow 1000.

[0099] In S10, the VPS / SPS is analyzed to determine whether one or more POC cycles per AU are constant. In S11, it is determined whether the POC cycles per AU are constant within the coded video sequence. If not, in S13, the value of the access unit count is calculated from the picture-level poc_cycle_au_value and the POC value. If yes, in S12, the value of the access unit count is calculated from the sequence-level poc_cycle_au_value and the POC value. In S14, the VPS / SPS is analyzed to determine whether the POC cycles per AU are constant. This allows the above procedure to start again or proceed to alternative processing.

[0100] In the same or other embodiments, images, slices, or tiles corresponding to AUs with the same AUC value may be associated with the same decoding or output time instance, even if the images, slices, or tiles have different POC values. Thus, even if there is no mutual analysis / decoding dependency between images, slices, or tiles within the same AU, all or a subset of images, slices, or tiles associated with the same AU may be decoded in parallel and output at the same time instance.

[0101] In the same or other embodiments, images, slices, or tiles corresponding to AUs with the same AUC value may be associated with the same composition / display time instance, even if the images, slices, or tiles have different POC values. If the composition time is included in the container format, images can be displayed at the same time instance even if they correspond to different AUs, as long as the composition times of the images are the same.

[0102] In the same or other embodiments, each image, slice, or tile may have the same temporal identifier (temporal_id) within the same AU. All or a subset of images, slices, or tiles corresponding to a time instance may be associated with the same temporal sublayer. In the same or other embodiments, each image, slice, or tile may have the same or different spatial layer id (layer_id) within the same AU. All or a subset of images, slices, or tiles corresponding to a time instance may be associated with the same or different spatial layers.

[0103] FIG. 8 shows an example of a video sequence structure with a combination of temporal_id, layer_id, POC, and AUC values and adaptive resolution change. In this example, the image, slice, or tile of the first AU with AUC=0 may have temporal_id=0 and layer_id=0 or 1, and the image, slice, or tile of the second AU with AUC=1 may have temporal_id=1 and layer_id=0 or 1. The value of POC increases by 1 for each image, regardless of the values of temporal_id and layer_id. In this example, the value of poc_cycle_au may be equal to 2. Preferably, the value of poc_cycle_au may be set equal to the number of (spatial scalability) layers. Thus, in this example, the value of POC increases by 2, and the value of AUC increases by 1.

[0104] In the above embodiments, all or a subset of the inter-picture or inter-layer prediction structures and reference picture indications may be supported by using the existing Reference Picture Set (RPS) or Reference Picture List (RPL) signaling in HEVC. In RPS or RPL, the selected reference picture is indicated by signaling the value of POC or the delta value of POC between the current picture and the selected reference picture. For the disclosed subject matter, RPS and RPL can be used to indicate the inter-picture or inter-layer prediction structure without changing the signaling, with the following limitations: If the value of the temporal_id of a reference picture is greater than the value of the temporal_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction. If the value of the layer_id of a reference picture is greater than the value of the layer_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction.

[0105] In the same and other embodiments, motion vector scaling based on POC difference for temporal motion vector prediction can be disabled across multiple images within an access unit. Thus, although each image may have a different POC value within an access unit, the motion vectors are not scaled and are used for temporal motion vector prediction within the access unit. This is because reference images with different POCs within the same AU are considered to be reference images with the same time instance. Therefore, in this embodiment, if the reference image belongs to the AU associated with the current image, the motion vector scaling function can return 1.

[0106] In the same and other embodiments, motion vector scaling based on POC difference for temporal motion vector prediction can be optionally disabled across multiple images if the spatial resolution of the reference image differs from the spatial resolution of the current image. When motion vector scaling is allowed, the motion vector is scaled based on both the POC difference and the spatial resolution ratio between the current image and the reference image.

[0107] In the same or another embodiment, for temporal motion vector prediction, motion vectors may be scaled based on the AUC difference instead of the POC difference, especially when poc_cycle_au has non-uniform values (when vps_contant_poc_cycle_per_au==0). Otherwise (when vps_contant_poc_cycle_per_au==1), motion vector scaling based on the AUC difference may be the same as motion vector scaling based on the POC difference.

[0108] In the same or another embodiment, when a motion vector is scaled based on the AUC difference, a reference motion vector in the same AU (having the same AUC value) as the current image is not scaled based on the AUC difference and is used for motion vector prediction without scaling or with scaling based on the spatial resolution ratio between the current image and the reference image.

[0109] In the same and other embodiments, the AUC value is used to identify AU boundaries and is used for hypothetical reference decoder (HRD) operations that require input and output timing at the granularity of AUs. In most cases, a decoded image with the top layer of AUs is output for display. The AUC value and layer_id value can be used to identify the output image.

[0110] In one embodiment, an image may be composed of one or more sub-images. Each sub-image may cover a local region or the entire region of the image. The region supported by a sub-image may or may not overlap with the region supported by another sub-image. The region composed of one or more sub-images may or may not cover the entire region of the image. When an image is composed of sub-images, the region supported by a sub-image is the same as the region supported by the image.

[0111] In the same embodiment, a sub-image may be coded by a coding method similar to that used for the coded image. A sub-image may be coded independently or may be coded depending on another sub-image or coded image. A sub-image may or may not have an analytical dependency from another sub-image or coded image.

[0112] In the same embodiment, the coded sub-images may be included in one or more layers. The coded sub-images within a layer may have different spatial resolutions. The original sub-images may be spatially resampled (upsampled or downsampled) and coded with different spatial resolution parameters and included in the bitstream corresponding to the layer.

[0113] In the same or another embodiment, a sub-image having (W,H), where W is the width of the sub-image and H is the height of the sub-image, may be coded and included in the coded bitstream corresponding to layer 0. Meanwhile, (W*S w,k ,H*S h,k ) may be coded and included in the coded bitstream corresponding to layer k, where S w,k ,S h,k indicates the horizontal and vertical resampling ratio. S w,k ,S h,k If the value of is greater than 1, resampling is the same as upsampling. On the other hand, if S w,k ,S h,k If the value of is less than 1, resampling is the same as downsampling.

[0114] In the same or another embodiment, a coded sub-image of one layer may have a different visual quality than a coded sub-image of another layer within the same or a different sub-image. For example, sub-image i in layer n may have a different visual quality than a coded sub-image of another layer within the same or a different sub-image. i,n while subimage j in layer m is coded with quantization parameter Q j,m is coded in

[0115] In the same or another embodiment, coded sub-images in one layer may be independently decodable without relying on parsing or decoding from coded sub-images in another layer of the same local region. A sub-image layer that can be independently decodable without reference to another sub-image layer of the same local region is an independent sub-image layer. Coded sub-images in an independent sub-image layer may or may not have decoding or parsing dependencies from previously coded sub-images in the same sub-image layer, but coded sub-images may have no dependencies from coded images of another sub-image layer.

[0116] In the same or another embodiment, coded subimages in one layer may be dependently decodable, relying on analysis or decoding from coded subimages in another layer of the same local region. A subimage layer that can be dependently decodable by reference to another subimage layer of the same local region is a dependent subimage layer. Coded subimages in a dependent subimage may reference coded subimages belonging to the same subimage, previously coded subimages in the same subimage layer, or both reference subimages.

[0117] In the same or another embodiment, a coded sub-picture consists of one or more independent sub-picture layers and one or more dependent sub-picture layers. However, at least one independent sub-picture layer may exist for a coded sub-picture. An independent sub-picture layer may have a layer identifier (layer_id) value, which may be present in the NAL unit header or another high-level syntax structure, equal to 0. A sub-picture layer with layer_id equal to 0 is a base sub-picture layer.

[0118] In the same or another embodiment, an image may consist of one or more foreground sub-images and one background sub-image. The area supported by the background sub-image may be equal to the area of the image. The area supported by the foreground sub-image may overlap with the area supported by the background sub-image. The background sub-image may be a base sub-image layer, while the foreground sub-image may be a non-base (enhancement) sub-image layer. One or more non-base sub-image layers may reference the same base layer for decoding. Each non-base sub-image layer with layer_id equal to a may reference a non-base sub-image layer with layer_id equal to b, where a is greater than b.

[0119] In the same or another embodiment, an image may consist of one or more foreground sub-images, with or without one background sub-image. Each sub-image may have its own base sub-image layer and one or more non-base (enhancement) layers. Each base sub-image layer may be referenced by one or more non-base sub-image layers. Each non-base sub-image layer with layer_id equal to a may reference a non-base sub-image layer with layer_id equal to b, where a is greater than b.

[0120] In the same or another embodiment, an image may consist of one or more foreground sub-images, with or without one background sub-image. Each coded sub-image in a (base or non-base) sub-image layer may be referenced by one or more non-base layer sub-images that belong to the same sub-image, and one or more non-base layer sub-images that do not belong to the same sub-image.

[0121] In the same or another embodiment, an image may consist of one or more foreground sub-images, with or without one background sub-image. A sub-image in layer a may be further divided into multiple sub-images within the same layer. One or more coded sub-images in layer b may reference divided sub-images in layer a.

[0122] In the same or another embodiment, a coded video sequence (CVS) may be a group of coded images. A CVS may consist of one or more coded sub-image sequences (CSPS), where a CSPS may be a group of coded sub-images covering the same local region of an image. A CSPS may have the same or different temporal resolution as the coded video sequence.

[0123] In the same or another embodiment, the CSPS may be coded and included in one or more layers. A CSPS may be composed of one or more CSPS layers. By decoding one or more CSPS layers corresponding to a CSPS, a sequence of sub-images corresponding to the same local region can be reconstructed.

[0124] In the same or different embodiments, the number of CSPS layers corresponding to a CSPS may be the same as or different from the number of CSPS layers corresponding to another CSPS.

[0125] In the same or another embodiment, a CSPS layer may have a different temporal resolution (e.g., frame rate) than another CSPS layer, and the original (uncompressed) sub-image sequence may be temporally resampled (upsampled or downsampled), coded with different temporal resolution parameters, and included in the bitstream corresponding to the layer.

[0126] In the same or another embodiment, a sub-image sequence of frame rate F may be coded and included in the coded bitstream corresponding to layer 0, while F*S t,k A sub-image sequence that is temporally upsampled (or downsampled) from the original sub-image sequence having t,k denotes the temporal sampling ratio of layer k. t,k If the value of is greater than 1, the temporal resampling process is equivalent to frame rate up-conversion. t,k If the value of is less than 1, the temporal resampling process is equivalent to a frame rate down-conversion.

[0127] In the same or another embodiment, when a sub-image having CSPS layer a is referenced by a sub-image having CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b, the decoded pixels of CSPS layer a are resampled and used for reference. The resampling process may require upsampling filtering or downsampling filtering.

[0128] FIG. 11 shows an example 1100 related to a video bitstream including a background video CSPS having a layer_id equal to 0 and a plurality of foreground CSPS layers. The encoded sub-images may consist of one or more CSPS layers, but the background regions that do not belong to the foreground CSPS layers may consist of a base layer. The base layer may include background regions and foreground regions, and the enhancement CSPS layers may include foreground regions. The enhancement CSPS layers may have better visual quality than the base layer in the same region. The enhancement CSPS layers can refer to the reconstructed pixels and motion vectors of the base layer corresponding to the same region.

[0129] In the same or another embodiment, the video bitstream corresponding to the base layer is included in a track, and the CSPS layers corresponding to each sub-image are included in separate tracks within the video file.

[0130] In the same or another embodiment, the video bitstream corresponding to the base layer is included in a track, and the CSPS layers having the same layer_id are included in separate tracks. In this example, the track corresponding to layer k includes only the CSPS layer corresponding to layer k.

[0131] In the same or another embodiment, each CSPS layer of each sub-image is stored in a separate track. Each track may or may not have an analytical or decoding dependency on one or more other tracks.

[0132] In the same or another embodiment, each track may include a bitstream corresponding to layer i to layer j of the CSPS layers of all or a subset of the sub-pictures, where 0 < i <= j <= k, and k is the top layer of the CSPS.

[0133] In the same or another embodiment, an image consists of one or more associated media data, including a depth map, an alpha map, 3D geometry data, an occupancy map, etc. Such associated timed media data may be divided into one or more data substreams, each corresponding to one sub-image.

[0134] In the same or another embodiment, FIG. 12 illustrates an example 1200 of a video conference based on a multi-layer sub-image method. The video stream includes one base layer video bitstream 1201 corresponding to a background image and one or more enhancement layer video bitstreams 1202 corresponding to foreground sub-images. Each enhancement layer video bitstream 1202 corresponds to a CSPS layer. The display displays the image corresponding to the base layer by default. Picture-in-picture (PIP) views of one or more users are included. When a specific user is selected via client control, the enhancement CSPS layer 1202 corresponding to the selected user is decoded and displayed with enhanced quality or spatial resolution. FIG. 13 illustrates a diagram 1300 of the operation.

[0135] At S20 there is decoding of the video bitstream having multiple layers, and at S21 there is identification of a background region and one or more foreground sub-images. At S22 it is considered whether a particular sub-image region has been selected, and if not, at S24 the background region is decoded and displayed, and if so, at S23 the extended sub-image is decoded and displayed.

[0136] In the same or another embodiment, a network middlebox (such as a router) can select a subset of layers to send to a user depending on its bandwidth. The image / sub-image organization can be used for bandwidth adaptation. For example, if a user does not have the bandwidth, the router can remove layers or select some sub-images by importance or based on the setup being used; this can be done dynamically to adapt to bandwidth.

[0137] FIG. 14 illustrates a use case for 360° video 1400. When a spherical 360° image 1401 is projected onto a planar image, the projected 360° image is divided into multiple sub-images as a base layer 1402. Enhancement layers 1403 for specific sub-images are coded and transmitted to the client. A decoder may be able to decode both the base layer containing all sub-images and the enhancement layer for a selected sub-image. If the current viewport is identical to the selected sub-image, the displayed image may be of higher quality with the sub-image decoded using the enhancement layer. Otherwise, the image decoded with the base layer may appear lower quality.

[0138] In the same or another embodiment, any layout information for display can be present in the file as supplemental information (such as an SEI message or metadata). Depending on the notified layout information, one or more decoded sub-images can be rearranged and displayed. The layout information may be notified by a streaming server or broadcaster, regenerated by a network entity or cloud server, or determined by a user's customized settings.

[0139] In one embodiment, when an input image is divided into one or more (rectangular) sub-regions, each sub-region may be coded as an independent layer. Each independent layer corresponding to a local region may have a unique layer_id value. For each independent layer, sub-image size and position information may be signaled. For example, image size (width, height) and offset information of the upper left corner (x_offset, y_offset). Figure 15 shows an example layout 1500 of divided sub-images, their size and position information, and corresponding image prediction structure. Layout information including sub-image size and sub-image position may be signaled in a high-level syntax structure such as a parameter set, a slice or tile group header, or an SEI message.

[0140] In the same embodiment, each sub-image corresponding to an independent layer can have its own POC value in the AU. When a syntax element is used in the RPS or RPL structure to indicate a reference image between images stored in the DPB, the POC value of each sub-image corresponding to the layer can be used.

[0141] In the same or another embodiment, the layer_id may not be used and the POC (delta) value may be used to indicate the (inter-layer) prediction structure.

[0142] In the same embodiment, a sub-image with a POC value equal to N corresponding to a layer (or local region) may or may not be used as a reference image for a sub-image with a POC value equal to N+K corresponding to the same layer (or the same local region) for motion compensation prediction. In most cases, the value of the number K may be equal to the maximum number of (independent) layers and may be the same as the number of sub-regions.

[0143] In the same or another embodiment, Figure 16 shows an extended case of Figure 15. When the input image is divided into multiple (e.g., four) subregions, each local region can be coded with one or more layers. In this case, the number of independent layers may be equal to the number of subregions, and one or more layers may correspond to the subregions. Thus, each subregion can be coded with one or more independent layers and zero or more dependent layers.

[0144] In the same embodiment, Figure 16 shows an example diagram 1600 in which an input image may be divided into four sub-regions. The top right sub-region may be coded as two layers, Layer 1 and Layer 4, and the bottom right sub-region may be coded as two layers, Layer 3 and Layer 5. In this case, Layer 4 may refer to Layer 1 for motion compensation prediction, while Layer 5 may refer to Layer 3 for motion compensation.

[0145] In the same or another embodiment, in-loop filtering (such as deblocking filtering, adaptive in-loop filtering, reshapers, bilateral filtering, or any deep learning-based filtering) across layer boundaries can be (optionally) disabled.

[0146] In the same or another embodiment, motion compensated prediction or intra block copying across layer boundaries can (optionally) be disabled.

[0147] In the same or another embodiment, boundary padding for motion compensated prediction or in-loop filtering at sub-image boundaries may be optionally processed. A flag indicating whether boundary padding is processed may be signaled in a high-level syntax structure such as a parameter set (VPS, SPS, PPS, or APS), a slice or tile group header, or an SEI message.

[0148] In the same or another embodiment, layout information for sub-regions (or sub-images) may be signaled in the VPS or SPS. FIG. 17 shows example syntax elements 1700 in the VPS and SPS. In this example, vps_sub_picture_dividing_flag is signaled in the VPS. The flag may indicate whether the input image is divided into multiple sub-regions. If the value of vps_sub_picture_dividing_flag is equal to 0, the input image of the coded video sequence corresponding to the current VPS may not be divided into multiple sub-regions. In this case, the input image size may be the same as the coded image size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS. If the value of vps_sub_picture_dividing_flag is equal to 1, the input image may be divided into multiple sub-regions. In this case, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled in the VPS. The values of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may be equal to the width and height of the input image, respectively.

[0149] In the same embodiment, the values of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may not be used for decoding, but may be used for composition and display.

[0150] In the same embodiment, if the value of vps_sub_picture_dividing_flag is equal to 1, the syntax elements pic_offset_x and pic_offset_y may be signaled in the SPS corresponding to a particular layer. In this case, the coded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to a particular layer. Also, the position of the upper left corner of the sub-region (pic_offset_x, pic_offset_y) may be signaled in the SPS.

[0151] In the same embodiment, the position information of the top left corner of the sub-region (pic_offset_x, pic_offset_y) may not be used for decoding, but may be used for composition and display.

[0152] In the same or another embodiment, layout information (size and position) of all or a subset of subregions of an input image, as well as inter-layer dependency information, may be signaled in a parameter set or SEI message. Figure 18 shows example syntax elements 1800 indicating information about the layout of subregions, inter-layer dependencies, and relationships between subregions and one or more layers. In this example, the syntax element num_sub_region indicates the number of (rectangular) subregions in the current coded video sequence. The syntax element num_layers indicates the number of layers in the current coded video sequence. The value of num_layers can be equal to or greater than the value of num_sub_region. If any subregion is coded as a single layer, the value of num_layers may be equal to the value of num_sub_region. If one or more subregions are coded as multiple layers, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] indicates a dependency from the jth layer to the ith layer. num_layers_for_region[i] indicates the number of layers associated with the i-th subregion. sub_region_layer_id[i][j] indicates the layer_id of the j-th layer associated with the i-th subregion. sub_region_offset_x[i] and sub_region_offset_y[i] indicate the horizontal and vertical positions of the top-left corner of the i-th subregion, respectively. sub_region_width[i] and sub_region_height[i] indicate the width and height of the i-th subregion, respectively.

[0153] In one embodiment, one or more syntax elements specifying an output layer set to indicate one of one or more layers to be output with or without profile tier-level information may be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message. Referring to Figure 19, a syntax element num_output_layer_sets indicating the number of output layer sets (OLS) in a coded video sequence that references a VPS may be signaled in a VPS. For each output layer set, as many output_layer_flag elements as there are output layers may be signaled.

[0154] In the same embodiment, output_layer_flag[i] equal to 1 specifies that the i-th layer is output. vps_output_layer_flag[i] equal to 0 specifies that the i-th layer is not output.

[0155] In the same or another embodiment, one or more syntax elements specifying profile tier level information for each output layer set may be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message. Further referring to Figure 19, a syntax element num_profile_tile_level indicating the number of profile tier level information per OLS in a coded video sequence referencing the VPS may be signaled in the VPS. For each output layer set, a set of profile tier level information syntax elements, or an index indicating a specific profile tier level information among the profile tier level information entries, may be signaled for each output layer.

[0156] In the same embodiment, profile_tier_level_idx[i][j] specifies an index into the list of profile_tier_level() syntax structures in the VPS for the profile_tier_level() syntax structure that applies to the jth layer of the ith OLS.

[0157] In the same or another embodiment, referring to diagram 2000 of FIG. 20, if the number of maximum layers is greater than 1 (vps_max_layers_minus1>0), the syntax elements num_profile_tile_level and / or num_output_layer_sets may be signaled.

[0158] In the same or another embodiment, referring to FIG. 20, there may be a syntax element vps_output_layers_mode[i] in the VPS that indicates the mode of output layer notification for the i-th output layer set.

[0159] In the same embodiment, vps_output_layers_mode[i] equal to 0 specifies that only the highest layer in the i-th output layer set is output. vps_output_layer_mode[i] equal to 1 specifies that all layers are output in the i-th output layer set. vps_output_layer_mode[i] equal to 2 specifies that the layers output are those in the i-th output layer set with vps_output_layer_flag[i][j] equal to 1. More values may be reserved.

[0160] In the same embodiment, output_layer_flag[i][j] may or may not be signaled depending on the value of vps_output_layers_mode[i] for the ith output layer set.

[0161] In the same or another embodiment, referring to Figure 20, a flag vps_ptl_signal_flag[i] may be present for the i-th output layer set. Depending on the value of vps_ptl_signal_flag[i], profile tier level information for the i-th output layer set may or may not be signaled.

[0162] In the same or another embodiment, referring to diagram 2100 of FIG. 21, the number of sub-images in the current CVS, max_subpics_minus1, may be signaled in a high-level syntax structure, for example, a VPS, DPS, SPS, PPS, APS, or SEI message.

[0163] In the same embodiment, referring to FIG. 21, if the number of sub-pictures is greater than 1 (max_subpics_minus1>0), the sub-picture identifier sub_pic_id[i] of the ith sub-picture may be signaled.

[0164] In the same or another embodiment, one or more syntax elements indicating sub-picture identifiers belonging to each layer of each output layer set may be signaled in the VPS. Referring to diagram 2200 of Figure 22, sub_pic_id_layer[i][j][k] indicates the kth sub-picture present in the jth layer of the ith output layer set. With this information, the decoder can know which sub-pictures can be decoded and output for each layer of a particular output layer set.

[0165] In one embodiment, a Picture Header (PH) is a syntax structure containing syntax elements that apply to all slices of a coded picture. A Picture Unit (PU) is a set of NAL units that are associated with each other according to specified classification rules, are consecutive in decoding order, and contain exactly one coded picture. A PU may contain one or more VCL NAL units that contain a Picture Header (PH) and a coded picture.

[0166] In one embodiment, the SPS (RBSP) is available to the decoding process before it is referenced, and is either contained in at least one AU with TemporalId=0 or provided via external means.

[0167] In one embodiment, the SPS (RBSP) is available to the decoding process before it is referenced and is contained in at least one AU with TemporalId=0 in the CVS that contains one or more PPSs that reference the SPS, or is provided via external means.

[0168] In one embodiment, the SPS (RBSP) is available to the decoding process before it is referenced by one or more PPSs, and is included in at least one PU with a nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL units in the CVS, including the one or more PPSs that reference the SPS, or is provided via external means.

[0169] In one embodiment, the SPS (RBSP) is available to the decoding process before it is referenced by one or more PPSs, and is included in at least one PU with TemporalId of 0 and nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL unit, or is provided via external means.

[0170] In one embodiment, the SPS (RBSP) is available to the decoding process before it is referenced by one or more PPSs, and is included in at least one PU with TemporalId of 0 and a nuh_layer_id equal to the lowest nuh_layer_id value of a PPS NAL unit that references the SPS NAL unit in the CVS, or is provided via external means, or is provided via external means.

[0171] In the same or another embodiment, pps_seq_parameter_set_id specifies the value of sps_seq_parameter_set_id of the referenced SPS. The value of pps_seq_parameter_set_id may be the same for all PPSs referenced by coded pictures in the CLVS.

[0172] In the same or another embodiment, all SPS NAL units with a particular value of sps_seq_parameter_set_id in a CVS may have the same content.

[0173] In the same or another embodiment, regardless of the nuh_layer_id value, SPS NAL units may share the same value space for sps_seq_parameter_set_id.

[0174] In the same or another embodiment, the nuh_layer_id value of an SPS NAL unit may be equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL unit.

[0175] In one embodiment, if an SPS with nuh_layer_id equal to m is referenced by one or more PPSs with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) referenced layer of the layer with nuh_layer_id equal to m.

[0176] In one embodiment, the PPS (RBSP) shall be available to the decoding process before it is referenced, and shall be contained in at least one AU whose TemporalId is equal to the TemporalId of the PPS NAL unit, or provided via external means.

[0177] In one embodiment, the PPS (RBSP) is available to the decoding process before it is referenced, and is included in at least one AU containing one or more PHs (or coded slice NAL units) that reference the PPS, whose TemporalId in the CVS is equal to the TemporalId of the PPS NAL unit, or is provided via external means.

[0178] In one embodiment, the PPS (RBSP) is available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units) and is included in at least one PU with a nuh_layer_id equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL unit in the CVS, including one or more PHs (or coded slice NAL units) that reference the PPS, or is provided via external means.

[0179] In one embodiment, a PPS (RBSP) is available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), and is included in at least one PU that contains one or more PHs (or coded slice NAL units) that reference the PPS, has TemporalId equal to the TemporalId of the PPS NAL unit, and has nuh_layer_id equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL units in the CVS, or is provided via external means.

[0180] In the same or another embodiment, the ph_pic_parameter_set_id of the PH specifies the value of the pps_pic_parameter_set_id of the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same for all PPSs referenced by coded pictures in the CLVS.

[0181] In the same or another embodiment, all PPS NAL units with a particular value of pps_pic_parameter_set_id within a PU may have the same content.

[0182] In the same or another embodiment, PPS NAL units may share the same value space for pps_pic_parameter_set_id regardless of the nuh_layer_id value.

[0183] In the same or another embodiment, the nuh_layer_id value of a PPS NAL unit may be equal to the lowest nuh_layer_id value of any coded slice NAL unit that references a NAL unit that references the PPS NAL unit.

[0184] In one embodiment, if a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0185] In one embodiment, the PPS (RBSP) shall be available to the decoding process before it is referenced, and shall be contained in at least one AU whose TemporalId is equal to the TemporalId of the PPS NAL unit, or provided via external means.

[0186] In one embodiment, the PPS (RBSP) is available to the decoding process before it is referenced, and is included in at least one AU containing one or more PHs (or coded slice NAL units) that reference the PPS, whose TemporalId in the CVS is equal to the TemporalId of the PPS NAL unit, or is provided via external means.

[0187] In one embodiment, the PPS (RBSP) is available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units) and is included in at least one PU with a nuh_layer_id equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL unit in the CVS, including one or more PHs (or coded slice NAL units) that reference the PPS, or is provided via external means.

[0188] In one embodiment, a PPS (RBSP) is available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), and is included in at least one PU that contains one or more PHs (or coded slice NAL units) that reference the PPS, has TemporalId equal to the TemporalId of the PPS NAL unit, and has nuh_layer_id equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL units in the CVS, or is provided via external means.

[0189] In the same or another embodiment, the ph_pic_parameter_set_id of the PH specifies the value of the pps_pic_parameter_set_id of the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same for all PPSs referenced by coded pictures in the CLVS.

[0190] In the same or another embodiment, all PPS NAL units with a particular value of pps_pic_parameter_set_id within a PU may have the same content.

[0191] In the same or another embodiment, PPS NAL units may share the same value space for pps_pic_parameter_set_id regardless of the nuh_layer_id value.

[0192] In the same or another embodiment, the nuh_layer_id value of a PPS NAL unit may be equal to the lowest nuh_layer_id value of any coded slice NAL unit that references a NAL unit that references the PPS NAL unit.

[0193] In one embodiment, if a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0194] In one embodiment, when the flag no_temporal_sublayer_switching_flag is signaled in a DPS, VPS, or SPS, the TemporalId value of a PPS referencing a parameter set with the flag equal to 1 may be equal to 0, and the TemporalId value of a PPS referencing a parameter set with the flag equal to 1 may be greater than or equal to the TemporalId value of the parameter set.

[0195] In one embodiment, each PPS (RBSP) is available to the decoding process before it is referenced and may be included in at least one AU whose TemporalId is less than or equal to the TemporalId of the coded slice NAL unit (or PH NAL unit) or may be provided via external means. If the PPS NAL unit is included in an AU that precedes the AU that contains the coded slice NAL unit that references the PPS, VCL NAL units that enable temporal upper layer switching or VCL NAL units whose nal_unit_type is equal to STSA_NUT (which indicates that the picture in the VCL NAL unit may be a gradual temporal sub-layer access (STSA) picture) may not be present after the PPS NAL unit and before the coded slice NAL unit that references the APS.

[0196] In the same or another embodiment, the PPS NAL unit and the coded slice NAL unit (and its PH NAL unit) that references the PPS may be included in the same AU.

[0197] In the same or another embodiment, the PPS NAL unit and the STSA NAL unit may be included in the same AU before the coded slice NAL unit (and its PH NAL unit) that references the PPS.

[0198] In the same or another embodiment, the STSA NAL unit, the PPS NAL unit, and the coded slice NAL unit that references the PPS (and its PH NAL unit) may reside in the same AU.

[0199] In the same embodiment, the TemporalId value of the VCL NAL unit containing the PPS may be equal to the TemporalId value of the previous STSA NAL unit.

[0200] In the same embodiment, the picture order count (POC) value of the PPS NAL unit may be greater than or equal to the POC value of the STSA NAL unit.

[0201] In the same embodiment, the picture order count (POC) value of a coded slice or PH NAL unit that references a PPS NAL unit may be greater than or equal to the POC value of the referenced PPS NAL unit.

[0202] APS NAL units can share the same value space for adaptation_parameter_set_id and aps_params_type regardless of the nuh_layer_id value.

[0203] The value of sps_video_parameter_set_id shall be the same for all SPSs referenced by coded images in the CVS across layers.

[0204] In one embodiment, in the NAL unit header semantics of the current VVC specification draft JVET-P2001 (editorially updated by JVET-Q0041), the value of nuh_layer_id of a non-VCL NAL unit is constrained as follows: if nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, then nuh_layer_id shall be equal to the lowest nuh_layer_id value of a coded slice NAL unit that references the NAL unit. Otherwise, if nal_unit_type is equal to SPS_NUT, then the nuh_layer_id value shall be equal to the lowest nuh_layer_id value of a PPS NAL unit that references an SPS NAL unit.

[0205] The constraints are intended to allow parameter sets (SPS, PPS, APS) to be referenced across layers, so that a coded slice NAL unit can / shall only reference PPS / APS NAL units in the same or lower layer, and a PPS NAL unit can / shall only reference SPS NAL units in the same or lower layer. Note that a coded slice VCL NAL unit can reference PPS / APS NAL units in non-reference layers, with the given constraints. For example, Figure 22 shows a simple two-layer example 2200, where the nuh_layer_id value of layer B is greater than the nuh_layer_id value of layer A, and layer A is not a direct / indirect reference layer of layer B. In that case, because there is no constraint prohibiting PPS / APS references in non-reference layers, the coded slice VCL NAL unit in layer B can reference a PPS / APS with nuh_layer_id equal to the nuh_layer_id of layer A. Nevertheless, the nuh_layer_id of the PPS / APS is equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the NAL unit. In this example, layer A and layer B may belong to different (output) layer sets, and the NAL units of layer A can be discarded by the bitstream extraction process, while the NAL units of layer B are still present in the output bitstream. And the PPS / APS NAL units referenced by coded slice NAL units of layer B may not be present in the output bitstream.

[0206] In the same or another embodiment, the current constraints need to be improved to address the above issues. The proposed constraints are: coded slice NAL units shall only reference PPS / APS units in the same layer or (direct) reference layer, and PPS NAL units shall only reference SPS units in the same layer or (direct) reference layer. In Figure 22, if layer A is a reference layer for layer B, both layers shall belong to the same (output) layer set and shall always be present in the extracted bitstream. Then, SPS / PPS / APS VCL NAL units referenced by NAL units of another layer shall not be removed.

[0207] According to an embodiment, the NAL unit header semantics include the following features: If a coded slice NAL unit references a non-VCL NAL unit with nal_unit_type equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, the layer of the non-VCL NAL unit shall be equal to the layer of the coded slice NAL unit or the layer that the coded slice NAL unit directly references, and if a PPS NAL unit references an SPS NAL unit, the layer of the SPS NAL unit shall be equal to the layer of the PPS NAL unit or the layer that the PPS NAL unit directly references.

[0208] According to an embodiment, the NAL unit header semantics include the following features: if a coded slice NAL unit references a non-VCL NAL unit with nal_unit_type equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, the layer of the non-VCL NAL unit shall be equal to the layer of the coded slice NAL unit or the direct / indirect reference layer of the coded slice NAL unit, and if a PPS NAL unit references an SPS NAL unit, the layer of the SPS NAL unit shall be equal to the layer of the PPS NAL unit or the direct / indirect reference layer of the PPS NAL unit.

[0209] The adaptive resolution parameter signaling techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 7 illustrates a computer system 700 suitable for implementing certain embodiments of the disclosed subject matter.

[0210] The computer software can be coded using any suitable machine code or computer language that can undergo mechanisms such as assembly, compilation, linking, etc. to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc. directly, or via translation, microcode execution, etc.

[0211] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0212] 7 for computer system 700 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 700.

[0213] The computer system 700 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (2D video, 3D video including stereoscopic video, etc.).

[0214] The input human interface devices may include one or more (only one of each) of a keyboard 701, a mouse 702, a trackpad 703, a touchscreen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, and a camera 708.

[0215] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen 710, data gloves, or joystick 705, although tactile feedback devices that do not function as input devices may also be present), audio output devices (e.g., speakers 709, headphones (not shown)), visual output devices (e.g., screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional hypervisible output via means such as stereo output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0216] The computer system 700 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 720 with media 721 such as CDs / DVDs, thumb drives 722, removable hard drives or solid state drives 723, legacy magnetic media such as tape and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), and the like.

[0217] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0218] The computer system 700 may also include an interface to one or more communications networks 755. The network 755 may be, for example, wireless, wired, or optical. The network 755 may further be local, wide-area, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of networks 755 include local area networks such as Ethernet, WLAN, etc.; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial networks including CANBus; and the like. Certain networks generally require an external network interface adapter 754 attached to a particular general-purpose data port or peripheral bus 749 (e.g., a USB port on the computer system 700). Others are generally integrated into the core of the computer system 700 by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 700 can communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast TV), transmit only (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0219] The aforementioned human interface devices, human access storage devices, and network interfaces may be attached to the core 740 of the computer system 700 .

[0220] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 743, hardware accelerators 744 for specific tasks, etc. These devices may be connected via a system bus 748, along with read-only memory (ROM) 745, random access memory 746, internal mass storage 747 such as an internal non-user-accessible hard drive, SSD, etc. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 748 or via a peripheral bus 749. Peripheral bus architectures include PCI, USB, etc.

[0221] The CPU 741, GPU 742, FPGA 743, and accelerator 744 may execute certain instructions that, in combination, may constitute the above-mentioned computer code. That computer code may be stored in ROM 745 or RAM 746. Transient data may also be stored in RAM 746, while persistent data may be stored, for example, in internal mass storage 747. Rapid storage and retrieval from any of the memory devices may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU 741, GPU 742, mass storage 747, ROM 745, RAM 746, etc.

[0222] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0223] By way of example and not limitation, architecture 700, and specifically a computer system having core 740, can provide functionality as a result of a processor executing software embodied in one or more tangible computer-readable media (including a CPU, GPU, FPGA, accelerator, etc.). Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of core 740 that is non-transitory in nature, such as core internal mass storage 747 or ROM 745. Software implementing various embodiments of the present disclosure can be stored on such devices and executed by core 740. The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause core 740, and specifically the processor therein (including a CPU, GPU, FPGA, etc.), to perform particular processes or particular portions of particular processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to software-defined operations. Additionally, or alternatively, a computer system may provide functionality as a result of logic embodied in hardwired or otherwise circuitry (e.g., accelerator 744), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software may encompass logic, where appropriate, and vice versa. References to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0224] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0225] 4 Video Coder 100 Communication Systems 110 First Terminal 120 Second Terminal 130 Second Terminal Pair 140 Second Terminal Pair 150 Communication Network 201 Camera 201 Video Sources 202 video sample streams 203 Encoder, Video Coder, Video Encoder 203 Video Sources 204 Video Bitstream 205 Streaming Server 206 Streaming Client 207 Video Bitstream 208 Streaming Client 209 Video Bitstream 210 Video Decoder 211 output video sample stream 212 Render Device, Display 213 Capture Subsystem 310 Receiver 312 channels 315 Buffer Memory 320 parser, video decoder 321 Symbol 351 Reverse conversion unit 352 Intra-image prediction unit, Intra-prediction unit 353 Motion Compensation Prediction Unit 355 Aggregator 356 Loop Filter Unit 357 Reference Image Memory 430 Video Coders, Encoders, Source Coders 432 Coding Engine 433 Local Video Decoder 434 Reference Image Memory, Reference Image Cache 435 Predictor 440 Transmitter 443 Video Sequence 445 Entropy Coder 450 Controller 460 Communication Channels 501 Image Header 502 ARC information 504 Image Parameter Set 505 ARC Reference Information 506 Table 507 Sequence Parameter Set 508 header 509 ARC information 511 parameter sets 512 ARC information 513 ARC Reference Information 514 Tile Group Header 515 ARC information 516 ARC information table, parameter set 600 Syntax Diagrams 601 Tile Group Header 603 Adaptive Resolution 610 Sequence Parameter Set 612 parameter sets 615 Reference Image Dimensions 617 Syntax 700 Architecture, Computer Systems 701 Keyboard 702 Mouse 703 Trackpad 705 Joystick 706 Microphone 707 Scanner 708 Camera 709 Speaker 710 Touchscreen 721 Medium 722 thumb drive 723 Solid State Drive 740 cores 741 CPU 743 FPGA 744 Hardware Accelerator 745 ROM 746 RAM 747 Large Internal Storage 748 System Bus 749 Peripheral Bus 754 External Network Interface Adapter 755 Communication Networks 900 Syntax Table 1201 Base Layer Video Bitstream 1202 Enhancement Layer Video Bitstream 1402 Base Layer 1403 Enhancement Layer

Claims

1. 1. A method executed by at least one processor, comprising: obtaining video data including data for a plurality of pictures of pixels; determining whether the subgroup of pixels included in the video data is associated with a first access unit (AU) or a second AU; outputting an AUC value identifying an AU of the first AU and the second AU associated with the subgroup; Including, The subgroups are assigned picture order count values (POC values) expressed as integers; the video data includes a POC cycle value indicating the number of the POC values per AU; the POC value set for the subgroup associated with the same AU is any value within a range of consecutive values corresponding to the number of POC cycle values; The first AU and the second AU are identified by the AUC values that are represented by integer values different from each other, The AUC value set for the first AU is represented by an integer component of a value obtained by dividing the POC value set for the subgroup associated with the first AU by the POC cycle value; The AUC value set for the second AU is represented by an integer component of a value obtained by dividing the POC value set for the subgroup associated with the second AU by the POC cycle value; The step of determining whether the subgroup is associated with the first AU or the second AU calculates the AUC value by dividing the POC value set for the subgroup by the POC cycle value included in the video data. method.

2. The method of claim 1 , wherein the subgroup is at least one of a picture, a slice, and a tile of the video data.

3. The method of claim 2 , wherein the video data includes video parameter set (VPS) data that identifies multiple spatial layers of the video data.

4. the subgroup is the slice, The method of claim 3 , wherein the POC value is included in a slice header of the video data.

5. determining whether the VPS data includes at least one flag indicating whether the picture of the subgroup is divided into a plurality of sub-regions; if the at least one flag indicates that the picture is divided into the plurality of sub-regions, determining at least one of a picture width and a picture height in luma samples; The method of claim 3 further comprising:

6. if the at least one flag indicates that the picture is divided into the plurality of sub-regions, determining a signaling value included in a sequence parameter set of the video data that specifies an offset of a portion of at least one of the sub-regions. The method of claim 5 further comprising:

7. The method of claim 1 , wherein the plurality of pictures represent a spherical 360° picture.

8. at least one memory configured to store computer program code; at least one processor configured to access said computer program code and to perform the method of any one of claims 1 to 7 according to the instructions of said computer program code; An apparatus comprising:

9. A computer program causing a computer to carry out the method according to any one of claims 1 to 7.

Images