Video encoding method and apparatus - Patents.com

Abstract

To provide a method and an apparatus for video encoding / decoding.SOLUTION: An apparatus includes a processing circuit that obtains a gate flag for each of at least one constraint information group in constraint information signaled in a coded video bitstream. Each gate flag indicates whether a constraint information group of the at least one constraint information group corresponding to the respective gate flag is present in the constraint information. The constraint information is for a subset of coding blocks in the coded video bitstream. Whether the constraint information group of the gate flag is present in the constraint information is determined based on the gate flag. The constraint information group includes at least one constraint flag for the subset of coding blocks. Prediction information for the subset of coding blocks is determined based on whether the constraint information group of the gate flag is present in the constraint information.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is related to U.S. Provisional Application No. 63 / 005,345, filed April 5, 2020, entitled "Method for signaling of subpicture and slice partitioning in coded video stream," U.S. Provisional Application No. 63 / 010,272, filed April 15, 2020, entitled "Method for signaling picture header and slice header in coded video stream," U.S. Provisional Application No. 63 / 024,140, ​​filed May 13, 2020, entitled "Techniques for coding tool control with constraint flags in coded video stream," U.S. Provisional Application No. 63 / 027,848, filed May 20, 2020, entitled "Techniques for signaling field coding information in multilayered video stream," and U.S. Provisional Application No. 63 / 028,222, filed May 21, 2020, entitled "Techniques for conditional signaling of constraint flags in coded video stream." No. 17 / 220,672, filed April 1, 2021, which claims benefit of priority to U.S. Patent Application No. 17 / 220,672, entitled "METHOD AND APPARATUS FOR VIDEO CODING," which claims benefit of priority to U.S. Patent Application No. 17 / 220,672, filed April 1, 2021, ... entitled

[0002] This disclosure describes embodiments generally related to video encoding. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. The inventors' work, to the extent that it is described in this background section, and aspects of the description that may not be admitted as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a sequence of images, each having spatial dimensions of, for example, 1920x1080 luma samples and associated chrominance samples. The sequence of images can have a fixed or variable picture rate (also informally known as frame rate), for example, 60 images per second or 60 Hz. Uncompressed video has specific bitrate requirements. For example, a 1080p60 4:2:0 video (1920x1080 luma sample resolution at a 60 Hz frame rate) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005] One of the goals of video encoding and decoding may be the reduction of redundancy in the input video signal through compression. Compression may help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than one order of magnitude. Both lossless and lossy compression, as well as combinations thereof, may be used. Lossless compression refers to techniques where an exact replica of the original signal can be reconstructed from a compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. For video, lossy compression has been widely adopted. The amount of acceptable distortion depends on the application, e.g., users of a particular consumer streaming application may tolerate higher distortion than users of a television distribution application. The achievable compression ratio may reflect that a higher tolerable / acceptable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders may utilize techniques from a number of broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[0007] Video codec techniques can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to the samples or other data from a previously reconstructed reference picture. In some video codecs, an image is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the image may be an intra-image. Intra-images and derivatives of intra-images, such as independent decoder refresh images, can be used to reset the decoder state and thus can be used as the first image or as a still image in the coded video bitstream and video session. Samples of an intra-block may be subjected to a transform, and the transform coefficients may be quantized prior to entropy coding. Intra prediction may be a technique that minimizes sample values ​​in a pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are required for a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, for example as known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to do so from, for example, surrounding sample data and / or metadata obtained during the encoding / decoding of blocks of spatially adjacent and preceding data in decoding order. Such techniques are hereafter referred to as "intra-prediction" techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from a reference picture.

[0009] Intra prediction may take many different forms. When two or more of such techniques may be used in a given video coding technique, the technique in use may be coded in an intra prediction mode. In certain cases, a mode may have sub-modes and / or parameters, which may be coded separately or included in a mode codeword. Which codeword is used for a given mode / sub-mode / parameter combination may affect the coding efficiency gains via intra prediction, as may the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further refined with new coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). Predictor blocks can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are replicated in the predictor block according to the direction. The reference to the direction in use can be coded in the bitstream or can itself be predicted.

[0011] Referring to FIG. 1A, a subset of nine known predictor directions from the 33 possible predictor directions (corresponding to the 33 angle modes of the 35 intra modes) of H.265 is depicted at the bottom right. The point (101) where the arrows converge represents the sample to be predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the top right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the bottom left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] 1A, a square block (104) of 4×4 samples is shown at the top left (indicated by a dashed bold line). The square block (104) contains 16 samples, each labeled with an “S” and including its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. Since the size of the block is 4×4 samples, S44 is at the bottom right. Additionally, a reference sample is shown that follows a similar numbering scheme. The reference sample is labeled with an R and is labeled with its Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the predicted samples are adjacent to the block being reconstructed, and therefore there is no need to use negative values.

[0013] Intra-image prediction may work by copying reference sample values ​​from adjacent samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes a signal for this block indicating a prediction direction that matches the arrow (102), i.e., the sample is predicted from one or more prediction samples at an angle of 45 degrees from the horizontal to the upper right. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, values ​​of multiple reference samples may be combined, for example by interpolation, to calculate a reference sample, especially when the orientation is not evenly divisible by 45 degrees.

[0015] As video coding techniques develop, the number of possible directions is increasing. In H.264 (2003), it was possible to represent 9 different directions. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of disclosure. Experiments have been carried out to identify the most likely directions, and certain techniques in entropy coding are used to represent those likely directions with a small number of bits, accepting certain penalties for less likely directions. Furthermore, the direction itself may be predictable from the neighboring directions used in adjacent already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (180) showing 65 intra prediction directions with JEM to illustrate the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits in an encoded video bitstream to represent directions may vary from one video coding technique to another and may range, for example, from simple direct mapping of prediction directions to complex adaptation schemes involving intra-prediction modes, codewords, most likely modes, and similar techniques. In all cases, however, there may be certain directions that are statistically less likely to occur in the video content than other certain directions. Since the goal of video compression is to reduce redundancy, a well-functioning video coding technique will ensure that such less likely directions are represented by more bits than more likely directions.

[0018] Motion compensation can be a lossy compression technique, and can refer to a technique in which blocks of sample data from a previously reconstructed image or part thereof (reference image) are used to predict a newly reconstructed image or image part after being spatially shifted in a direction indicated by a motion vector (hereafter MV). In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions X and Y, or three dimensions, the third dimension being an indication of the reference image being used (the latter can indirectly be the temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from other MVs, e.g., MVs associated with another region of sample data that is spatially adjacent to the region under reconstruction and precedes that MV in decoding order. Doing so can substantially reduce the amount of data required to encode the MV, thereby eliminating redundancy and increasing compression. For example, when encoding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there is a statistical likelihood that regions larger than the region to which a single MV is applicable will move in similar directions and therefore, in some cases, can be predicted using similar motion vectors derived from MVs of neighboring regions. This makes the MV found for a given region similar or the same as the MV predicted from the surrounding MVs and, after entropy encoding, can be represented with fewer bits than would be used to directly encode the MV. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself can be lossy, e.g., due to rounding errors when computing a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, High Efficiency Video Coding, December 2016). In this specification, we will describe a technique called "spatial merging" among the many MV prediction mechanisms provided by H.265.

[0021] Referring to Figure 2, the current block (201) contains samples found by the encoder during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of directly encoding its MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., the most recent (in decoding order) reference picture, using the MV associated with any one of the five surrounding samples denoted A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, MV prediction can use predictors from the same reference picture as the neighboring blocks are using. Summary of the Invention [Means for solving the problem]

[0022] Aspects of the present disclosure provide a method and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit is configured to obtain a gating flag for each of at least one constraint information group in the constraint information signaled in the encoded video bitstream. Each gating flag may indicate whether a constraint information group of the at least one constraint information group corresponding to the respective gating flag is present in the constraint information. The constraint information may relate to a subset of coding blocks in the encoded video bitstream. The processing circuit is configured to determine whether a constraint information group of the gating flag is present in the constraint information based on a gating flag of the constraint information group of the at least one constraint information group, the constraint information group of the gating flag including at least one constraint flag for the subset of coding blocks. The processing circuit is configured to determine prediction information of the subset of coding blocks based on whether the constraint information group of the gating flag is present in the constraint information. The processing circuit is configured to reconstruct the subset of coding blocks based on the prediction information.

[0023] In one embodiment, a constraint information group of gating flags is determined to be present in the constraint information. The processing circuitry is configured to determine prediction information for a subset of the coding blocks based on at least one constraint flag in the constraint information group of the gating flags.

[0024] In one embodiment, the gate flag constraint information group is determined to be absent in the constraint information.

[0025] In one embodiment, each gate flag indicates that the constraint information group corresponding to the respective gate flag is not present in the constraint information and that no constraint flag is present in the constraint information.

[0026] In one embodiment, the constraint information is in a high level syntax for the at least one image, and the subset of coding blocks includes coding blocks in the at least one image.

[0027] In one embodiment, one or more of the at least one constraint information group are present in the encoded video bitstream. The constraint flags in the one or more of the at least one constraint information group are signaled according to a predetermined order. The processing circuit is configured to run-decode the constraint flags and determine prediction information for a subset of the coding blocks based on the constraint flags.

[0028] In one embodiment, the at least one constraint flag in the constraint information group of the gating flag includes a plurality of constraint flags that are signaled according to a predetermined order. The processing circuitry is configured to run-decode the plurality of constraint flags.

[0029] In one embodiment, the offset and length are present in the constraint information and indicate that at least one constraint flag in the constraint information group of the gate flag is present in the constraint information.

[0030] In one embodiment, one or more of the at least one constraint information group are present in the encoded video bitstream. For each of the one or more constraint information groups of the at least one constraint information group, an offset and a length are present in the constraint information, and at least one constraint flag in each of the one or more constraint information groups of the at least one constraint information group indicates the presence in the constraint information. The number of offsets is signaled in the encoded video bitstream.

[0031] In one embodiment, one or more of the at least one constraint information group are present in the encoded video bitstream. One or more syntax elements in the constraint information are byte-aligned. The one or more syntax elements may include at least a gate flag for each of the constraint information groups and a plurality of constraint flags in one or more of the constraint information groups of the at least one constraint information group.

[0032] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for video decoding.

[0033] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0034] [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 2 is a diagram of an example intra-prediction direction. [Diagram 2] FIG. 2 is a schematic diagram of a current block and its surrounding spatial merging candidates in one example. [Diagram 3] 1 is a simplified schematic block diagram of a communication system (300) according to one embodiment. [Figure 4] FIG. 4 is a simplified schematic block diagram of a communication system (400) according to one embodiment. [Diagram 5] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 7] 4 shows a block diagram of an encoder according to another embodiment; [Figure 8] 4 shows a block diagram of a decoder according to another embodiment; [Figure 9] 1 illustrates an example for signaling Adaptive Resolution Change (ARC) parameters according to an embodiment of the present disclosure. [Figure 10] FIG. 13 illustrates an example of an ARC parameter notification according to an embodiment of the present disclosure. [Figure 11] 1 illustrates an example set of Profile Tile Layer (PTL) syntax elements according to one embodiment of the present disclosure. [Figure 12]1 illustrates an example of generic constraint information according to an embodiment of the present disclosure. [Figure 13A] 1 illustrates an example set of PTL syntax elements according to one embodiment of the present disclosure. [Figure 13B] 1 illustrates an example set of PTL syntax elements according to one embodiment of the present disclosure. [Figure 14A] 1 illustrates an example of a generic constraint information syntax structure according to one embodiment of the present disclosure. [Figure 14B] 1 illustrates an example of a generic constraint information syntax structure according to one embodiment of the present disclosure. [Figure 15A] 1 illustrates an example of a generic constraint information syntax structure according to one embodiment of the present disclosure. [Figure 15B] 1 illustrates an example of a generic constraint information syntax structure according to one embodiment of the present disclosure. [Figure 15C] 1 illustrates an example of a generic constraint information syntax structure according to one embodiment of the present disclosure. [Figure 15D] 1 illustrates an example of a generic constraint information syntax structure according to one embodiment of the present disclosure. [Figure 16] 1 illustrates an example of a set of PTL syntax elements in a PTL syntax structure according to one embodiment of the present disclosure. [Figure 17] 1 illustrates an example of a syntax table for indicating field coding information in a sequence parameter set according to one embodiment of the present disclosure. [Figure 18] 1 illustrates an example of a syntax table for indicating field coding information in a video parameter set according to one embodiment of the present disclosure. [Figure 19] 1 shows a flowchart outlining a process according to one embodiment of the present disclosure. [Figure 20] FIG. 2 illustrates an example syntax table according to an embodiment of the present disclosure. [Figure 21] 1 illustrates a flowchart for parsing and decoding a Picture Order Count (POC) cycle per access unit and an Access Unit Count value according to one embodiment of the present disclosure. [Figure 22] 1 illustrates an example of a prediction structure for scalability with adaptive resolution change, according to one embodiment of the present disclosure. [Figure 23] 1 illustrates an example of a video bitstream structure including multiple sub-images according to one embodiment of the present disclosure. [Figure 24] FIG. 2 is a schematic diagram of a display of a selected sub-image with enhanced resolution according to one embodiment of the present disclosure. [Diagram 25] 1 illustrates a block diagram of a decoding and display process for a video bitstream containing multiple sub-images according to one embodiment of the present disclosure. [Figure 26] FIG. 1 is a schematic diagram of a 360 video display with an enhanced layer of sub-images according to one embodiment of the present disclosure. [Figure 27] 1 illustrates an example of sub-image layout information and corresponding layer and image prediction structures according to one embodiment of the present disclosure. [Figure 28] FIG. 1 illustrates an example of layout information for sub-images with local region spatial scalability modality and corresponding layer and image prediction structures according to one embodiment of the present disclosure. [Figure 29A] 1 illustrates an example syntax table for sub-image layout information according to one embodiment of the present disclosure. [Figure 29B] 1 illustrates an example syntax table for sub-image layout information according to one embodiment of the present disclosure. [Diagram 30] 1 illustrates an example of a syntax table of a sub-image layout information SEI message according to an embodiment of the present disclosure. [Diagram 31] FIG. 2 illustrates an example of a syntax table showing PTL information for output layers and output layer sets according to one embodiment of the present disclosure. [Diagram 32] 1 illustrates an example syntax table illustrating output layer modes for an output layer set, according to one embodiment of the present disclosure. [Diagram 33] 13 illustrates an example of a syntax table showing the current sub-image of each layer for each output layer set, according to one embodiment of the present disclosure. [Diagram 34] 1 illustrates an example of a syntax table for a video parameter set according to an embodiment of the present disclosure. [Diagram 35]1 illustrates an example of a syntax table illustrating an output tier set with an output tier set mode, according to one embodiment of the present disclosure. [Diagram 36] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices capable of communicating with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform a unidirectional transmission of data. For example, the terminal device (310) may encode video data (e.g., a stream of video images captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to reconstruct the video image, and display the video image according to the reconstructed video data. One-way data transmission may be common, such as in media serving applications.

[0036] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of encoded video data, such as may occur during a video conference. For the bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video images captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) may also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to recover the video images, and display the video images on an accessible display device according to the recovered video data.

[0037] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure need not be so limited. Embodiments of the present disclosure find use with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (350) represents any number of networks that convey encoded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. The communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless otherwise described herein below.

[0038] 4 illustrates the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter can be equally applied to other video-enabled applications including, for example, video conferencing, digital television, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0039] The streaming system may include a video source (401) and a capture subsystem (413) that may include, for example, a digital camera that creates, for example, a stream of uncompressed video images (402). In one example, the stream of video images (402) includes samples taken by a digital camera. The stream of video images (402), depicted as a thick line to emphasize the high amount of data compared to the encoded video data (404) (or encoded video bitstream), may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as a thin line to emphasize the lower amount of data compared to the stream of video images (402), may be stored on a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to obtain copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example, in an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and creates an output stream (411) of video images that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video encoding / compression standard, such as ITU-T Recommendation H.265. In one example, the video encoding standard under development is informally known as Versatile Video Coding (VVC).The disclosed subject matter may be used in the context of a VVC.

[0040] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0041] 5 shows a block diagram of a video decoder (510) according to one embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) of the example of FIG. 4.

[0042] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510), in the same or another embodiment, one encoded video sequence at a time, with the decoding of each encoded video sequence being independent of the other encoded video sequences. The encoded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (531) may receive the encoded video data together with other data, e.g., encoded audio data and / or auxiliary data streams, that may be forwarded to a respective using entity (not shown). The receiver (531) may separate the encoded video sequences from the other data. To combat network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other cases, it may be external to the video decoder (510) (not shown). In still others, there may be a buffer memory (not shown) external to the video decoder (510), for example to combat network jitter, and another buffer memory (515) internal to the video decoder (510), for example to handle playback timing. When the receiver (531) is receiving data from a store / forward device of sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be needed or may be small. For use with best-effort packet networks such as the Internet, a buffer memory (515) may be needed, may be relatively large, may be advantageously adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (510).

[0043] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but may be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device(s) may be in the form of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI) parameter set fragments (not shown). The parser (520) may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may follow a video encoding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context-dependent, etc. The parser (520) may extract from the coded video sequence at least one set of subgroup parameters for a subgroup of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) may also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0044] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to produce symbols (521).

[0045] The reconstruction of the symbols (521) can involve a number of different units, depending on the type of encoded video image or part thereof (inter-image and intra-image, inter-block and intra-block, etc.), and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the encoded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not depicted for clarity.

[0046] Beyond the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate:

[0047] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as well as control information from the parser (520) including the transform to use, block size, quantization factor, quantization scaling matrix, etc. as symbol(s) (521). The scalar / inverse transform unit (551) may output a block comprising sample values ​​that may be input to an aggregator (555).

[0048] In some cases, the output samples of the scalar / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image, but may use prediction information from a previously reconstructed portion of the current image. Such prediction information may be provided by an intra-image prediction unit (552). In some cases, the intra-image prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current image buffer (558). The current image buffer (558) may, for example, buffer a partially reconstructed current image and / or a fully reconstructed current image. The aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scalar / inverse transform unit (551).

[0049] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) associated with the block, these samples may be added to the output of the scalar / inverse transform unit (551) by the aggregator (555) to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector and are available to the motion compensated prediction unit (553) in the form of symbols (521), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0050] The output samples of the aggregator (555) may be subject to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded image or coded video sequence, and may also be responsive to previously reconstructed and loop filtered sample values.

[0051] The output of the loop filter unit (556) may be a sample stream that may be output to the render device (512) as well as stored in a reference image memory (557) for use in future inter-image prediction.

[0052] Once a particular coded image is fully reconstructed, it can be used as a reference image for future predictions. For example, once a coded image corresponding to a current image is fully reconstructed and the coded image is identified (e.g., by the parser (520)) as a reference image, the current image buffer (558) can become part of the reference image memory (557) and a new current image buffer can be reallocated before starting the reconstruction of the next coded image.

[0053] The video decoder (510) may perform decoding operations according to a standard predefined video compression technique, such as ITU-T Rec. H.265. The encoded video sequence may comply with a syntax specified by the video compression technique or standard being used, in the sense that the encoded video sequence complies with both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select a specific tool from all tools available in the video compression technique or standard as the only tool available under that profile. Also required for compliance is that the complexity of the encoded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference image size, etc. The limits set by the level may be further limited in some cases by a hypothetical reference decoder (HRD) specification and HRD buffer management metadata conveyed in the encoded video sequence.

[0054] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0055] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the example video encoder (403) of FIG.

[0056] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) that can capture the video image(s) to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0057] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual images that give motion when viewed in sequence. The image itself may be organized as a spatial array of pixels, each pixel may contain one or more samples, depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0058] According to one embodiment, the video encoder (603) may encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraint required by the application. Enforcing an appropriate encoding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units as described below. For clarity, couplings are not depicted. Parameters set by the controller (650) may include rate control related parameters (e.g., picture skip, quantizer, lambda value for rate distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions for the video encoder (603) optimized for a particular system design.

[0059] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As an oversimplified explanation, in one example, the encoding loop can include a source coder (630) (e.g., responsible for generating symbols, such as a symbol stream, based on an input image to be encoded and a reference image(s)) and a (local) decoder (633) built into the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar manner that the (remote) decoder also does (since in the video compression techniques contemplated in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). The decoding of the symbol stream results in a bit-exact result regardless of the location of the decoder (local or remote), so the contents in the reference image memory (634) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference image samples that the decoder "sees" when using the prediction during decoding. This basic principle of reference image synchrony (and the drift that occurs when synchrony cannot be maintained, e.g., due to channel errors) is also used in several related technologies.

[0060] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510), which is described in detail above in connection with Figure 5. However, with brief reference also to Figure 5, symbols may be available, and the encoding / decoding of the symbols into an encoded video sequence by the entropy coder (645) and parser (520) may be lossless, and the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (633).

[0061] An observation that can be made at this point is that any decoder techniques other than analysis / entropy decoding present in a decoder must necessarily be present in a corresponding encoder in substantially identical functional form. For this reason, the disclosed subject matter focuses on the decoder operation. A description of the encoder techniques can be omitted, since they are the inverse of the decoder techniques described generically. Only in certain areas is more detailed description necessary and is provided below.

[0062] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input image with reference to one or more previously coded images from a video sequence designated as “reference images.” In this manner, the coding engine (632) codes differences between pixel blocks of the input image and pixel blocks of reference image(s) that may be selected as predictive reference(s) to the input image.

[0063] The local video decoder (633) may decode the encoded video data of the image that may be designated as a reference image based on the symbols created by the source coder (630). The operation of the encoding engine (632) may advantageously be a lossy process. If the encoded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a replica of the source video sequence, usually with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference image and cause the reconstructed reference image to be stored in a reference image cache (634). In this way, the video encoder (603) may locally store a replica of the reconstructed reference image that has common content as the reconstructed reference image obtained by the far-end video decoder (without transmission errors).

[0064] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new image to be coded, the predictor (635) may search the reference image memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference image motion vectors, block shapes, etc. that may serve as suitable prediction references for the new image. The predictor (635) may operate on one sample block per pixel block to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input image may have prediction references drawn from multiple reference images stored in the reference image memory (634).

[0065] The controller (650) may manage the encoding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0066] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0067] The transmitter (640) may buffer the encoded video sequence(s) created by the entropy coder (645) and prepare them for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) may merge the encoded video data from the video coder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0068] A controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a particular encoded image type to each encoded image, which may affect the encoding technique that may be applied to each image. For example, images may often be assigned as any of the following image types:

[0069] An intra picture (I-picture) may be one that can be coded and decoded without using other pictures in the sequence as a source of prediction. Some video codecs can use different types of intra pictures, such as, for example, Independent Decoder Refresh ("IDR") Pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0070] A predicted image (P-image) may be one that can be encoded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0071] A bidirectionally predicted image (B-image) may be one that can be encoded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted images may use more than two reference images and associated metadata for the reconstruction of a single block.

[0072] A source image may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by a coding assignment applied to the respective image of the block. For example, blocks of an I image may be non-predictively coded or predictively coded with reference to already coded blocks of the same image (spatial or intra prediction). Pixel blocks of a P image may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference image. Blocks of a B image may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference images.

[0073] The video encoder (603) may perform encoding operations according to a given video encoding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0074] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0075] A video may be captured as multiple source images (video images) in time sequence. Intra-image prediction (often abbreviated as intra prediction) exploits spatial correlation in a given image, while inter-image prediction exploits correlation (temporal or other) between images. In one example, a particular image being encoded / decoded, called the current image, is divided into blocks. When a block in the current image is similar to a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference image and may have a third dimension that identifies the reference image if multiple reference images are used.

[0076] In some embodiments, bi-prediction techniques can be used for inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which are before the decoding order of the current picture in the video (but the display order may be past and future, respectively). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0077] Furthermore, to improve coding efficiency, merge mode techniques can be used for inter-picture prediction.

[0078] According to some embodiments of the present disclosure, predictions such as inter-image prediction and intra-image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a sequence of video images are divided into coding tree units (CTUs) for compression, and the CTUs in an image have the same size, such as 64 x 64 pixels, 32 x 32 pixels, or 16 x 16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a CTU of 64x64 pixels can be partitioned into one CU of 64x64 pixels, or four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. The CU is partitioned into one or more prediction units (PUs) depending on the temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in encoding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values) of 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0079] 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video image in a sequence of video images and to encode the processed block into an encoded image that is part of an encoded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) of the example of FIG. 4.

[0080] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8×8 samples. The video encoder (703) determines whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode, for example using rate-distortion optimization. If the processing block is coded in intra mode, the video encoder (703) may use intra prediction techniques to code the processing block into a coded image, and when the processing block is coded in inter mode or bi-predictive mode, the video encoder (703) may use inter prediction techniques or bi-predictive techniques, respectively, to code the processing block into a coded image. In certain video coding techniques, the merge mode may be an inter-picture prediction submode in which a motion vector is derived from one or more motion vector predictors without benefit of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0081] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), coupled together as shown in FIG.

[0082] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in reference images (e.g., blocks in previous and subsequent images), generate inter-prediction information (e.g., a description of redundant information through inter-coding techniques, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a prediction block) based on the inter-prediction information using any suitable technique. In some examples, the reference image is a decoded reference image that is decoded based on the encoded video information.

[0083] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block with blocks already encoded in the same image, generate quantized coefficients after transformation, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (722) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same image.

[0084] The generic controller (721) is configured to determine generic control data and control other components of the video encoder (703) based on the generic control data. In one example, the generic controller (721) determines a mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is an intra mode, the generic controller (721) controls the switch (726) to select an intra mode result to be used by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information to be included in the bitstream, and when the mode is an inter mode, the generic controller (721) controls the switch (726) to select an inter prediction result to be used by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information to be included in the bitstream.

[0085] The residual calculation unit (723) calculates the difference (residual data) between the received block and a prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate on the residual data and encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be suitably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter-encoder (730) may generate a decoded block based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) may generate a decoded block based on the decoded residual data and the intra-prediction information. In some examples, the decoded block may be appropriately processed to generate a decoded image, which may be buffered in a memory circuit (not shown) and used as a reference image.

[0086] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information in accordance with an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. It should be noted that, according to the disclosed subject matter, when encoding a block in a merged sub-mode of either the inter-mode or the bi-prediction mode, no residual information is present.

[0087] 8 shows a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive encoded images that are part of an encoded video sequence and decode the encoded images to generate reconstructed images. In one example, the video decoder (810) is used in place of the video decoder (410) of the example of FIG. 4.

[0088] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter-decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-decoder (872), coupled together as shown in FIG.

[0089] The entropy decoder (871) may be configured to reconstruct from the coded image certain symbols representing syntax elements that make up the coded image. Such symbols may include, for example, prediction information (e.g., intra-prediction information or inter-prediction information, etc.) that may identify the mode in which the block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, the latter two being merged submode or another submode), certain samples or metadata used for prediction by the intra-decoder (872) or the inter-decoder (880), respectively, residual information, e.g., in the form of quantized transform coefficients, etc. In one example, if the prediction mode is an inter-prediction mode or a bi-prediction mode, the inter-prediction information is provided to the inter-decoder (880), and if the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may undergo inverse quantization and is provided to the residual decoder (873).

[0090] The inter decoder (880) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0091] The intra decoder (872) is configured to receive the intra prediction information and to generate a prediction result based on the intra prediction information.

[0092] The residual decoder (873) is configured to perform inverse quantization to extract inverse quantized transform coefficients and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (871) (the data path not shown as this may be low-volume control information only).

[0093] The reconstruction module (874) is configured to combine, in the spatial domain, the residual as output by the residual decoder (873) and the prediction result (possibly as output by an inter- or intra-prediction module) to form a reconstructed block that may be part of the reconstructed image, which may be part of the reconstructed video. It should be noted that other suitable operations, such as deblocking operations, may be performed to improve visual quality.

[0094] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0095] Aspects of the present disclosure include control of encoding tool(s) and features using constraint flags in the encoded video stream.

[0096] Video encoders and decoders may operate with a given picture size that is defined and remains constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. In examples such as MPEG-2, system designs are known to change the horizontal resolution (and therefore picture size) depending on factors such as scene activity, but only in I-pictures, and thus typically for GOPs. Resampling of reference pictures to use different resolutions within a CVS is known, for example from ITU-T Rec. H.263 Annex P. However, the picture size does not change, only the reference picture is resampled, and it is possible that only a portion of the image canvas is used (e.g. in downsampling) or only a portion of the scene is captured (e.g. in upsampling). In some examples, such as H.263 Annex Q, it is permitted to resample individual macroblocks by a factor of two in each dimension (e.g., above or below). The picture size may remain the same. The size of the macroblocks may be fixed, for example in H.263, and does not need to be signaled.

[0097] The picture size of the predicted picture can be changed. Examples such as VP9 allow resampling of the reference picture and changing the resolution of the whole picture. One example (including, for example, Hendry et al., "On adaptive resolution change (ARC) for VVC," Joint Video Team document JVET-M0l35-vl, Jan 9-l9, 20l9, which is incorporated herein in its entirety) allows resampling of the whole reference picture to a different resolution (e.g., higher or lower). The different candidate resolutions can be coded in the sequence parameter set (SPS) and referenced by per-picture syntax elements in the picture parameter set (PPS).

[0098] SUMMARY Techniques for adaptive picture size signaling in a video bitstream are disclosed in this disclosure.

[0099] When an image is coded into a bitstream that includes one or more layers with different qualities, the bitstream may have syntax elements that specify which layer(s) (or set of layers) can be output at the decoder side. The set of layers that are output may be defined as an output layer set. In an exemplary video codec that supports multiple layers and scalability, one or more output layer sets may be signaled in a video parameter set (VPS). Syntax elements that specify the profile tier level (PTL) of the entire bitstream or one or more output layer sets may be signaled in a VPS, a DPS that is the same as the decoder capability information (DCI), an SPS, a PPS, an SEI message, etc. In the PTL information, there may be generic constraint information that can specify constraints of coding tools and capabilities. It is desirable to efficiently represent and signal constraint information of various coding tools and capabilities.

[0100] A sub-image may refer to, for example, a rectangular arrangement of samples, blocks, macroblocks, coding units, or similar entities that may be semantically grouped and coded independently at a changed resolution. One or more sub-images may form an image. One or more coded sub-images may form a coded image. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from an image. In some examples, one or more coded sub-images may be assembled in the compressed domain without transcoding to a coded image down to the sample level. In some examples, one or more coded sub-images may be extracted from a coded image in the compressed domain.

[0101] Adaptive Resolution Change (ARC) may refer to a mechanism that allows changing the resolution of an image or sub-image in a CVS, for example, by reference image resampling. ARC parameters may refer to control information used to perform adaptive resolution change. ARC parameters may include filter parameters, scaling factors, output and / or reference image resolution, various control flags, etc.

[0102] The above description can be applied to the encoding and decoding of a single, semantically independent encoded video image. Before describing the encoding / decoding of multiple sub-images with independent ARC parameters and their implied additional complexities, signaling of the ARC parameters will be described.

[0103] FIG. 9 illustrates examples (e.g., options) for signaling ARC parameters according to an embodiment of the present disclosure. Coding efficiency, complexity, and architecture may vary from example to example. A video coding standard or technology may select one or more of the examples or other variations for signaling ARC parameters. These examples may not be mutually exclusive and may be interchanged based on application needs, standard technology, encoder choice, and / or the like.

[0104] The classes of ARC parameters may include: The upsampled and / or downsampled coefficients may be separated or combined in the X and Y dimensions and one or more short syntax elements may be coded that may point to tables containing the upsampled and / or downsampled coefficients. The upsample and / or downsample coefficients with an added time dimension may code one or more short syntax elements that indicate a constant speed zoom-in and / or zoom-out of a given number of pictures and may point to a table containing the upsample and / or downsample coefficients with an added time dimension. The resolution in the X or Y dimension of the input image, output image, reference image, coded image, combined or separate, in samples, blocks, macroblocks, CUs, or any other suitable units of granularity. If there are more than one resolution (e.g. one resolution for the input image, another for the reference image), a set of values ​​may be inferred from another set of values, which may be gated, for example, by the use of flags, as described below. Warping coordinates similar to those used in H.263 Annex P, with the appropriate granularity as described above. H.263 Annex P defines an efficient way of encoding the warping coordinates. Other efficient methods can be devised. For example, the variable-length reversible Huffman-style encoding of the warping coordinates in Annex P could be replaced by a binary code of the appropriate length, and the length of the binary codeword could be derived from the maximum image size multiplied by a coefficient and offset by a value that allows warping outside the bounds of the maximum image size. Upsample and / or downsample filter parameters. In one example, there is only a single filter for upsampling and / or downsampling. In one example, it is advantageous to allow more flexibility in filter design. In one example, the filter parameters are signaled. The filter parameters may be selected using an index in a list of possible filter designs. The filter may be fully specified (e.g., by specifying a list of filter coefficients using a suitable entropy coding technique) or the filter may be selected implicitly by the upsample or downsample ratio, etc., signaled according to any of the mechanisms described above.

[0105] This description may be applicable to the coding of a finite set of upsampled or downsampled coefficients (the same coefficients used in both X and Y dimensions) indicated by a codeword. The codeword may advantageously be variable-length coded, for example, using Ext-Golomb codes for certain syntax elements in video coding specifications (e.g., H.264 and H.265). An appropriate mapping of values ​​to upsampled or downsampled coefficients may be based, for example, on Table 1.

[0106] [Table 1]

[0107] Similar mappings can be devised according to the applications and capabilities of upscaling and downscaling mechanisms available in video compression techniques or standards. Table 1 can be appropriately extended to additional values. Values ​​can be represented by entropy coding mechanisms other than Ext-Golomb codes, for example by using binary coding. In one example, entropy coding mechanisms other than Ext-Golomb codes can have certain advantages when the resampling factor is of interest outside the video processing engine (e.g., encoder and decoder), for example by a media aware network element (MANE). If no change in resolution is required (e.g., original / target resolution is 1 in Table 1), a short Ext-Golomb code (e.g., only 1 bit as shown in Table 1) can be selected, which can have coding efficiency advantages over using binary codes in the most general case, for example.

[0108] Some entries in Table 1 and corresponding semantics may be fully or partially configurable. For example, the basic outline of Table 1 is conveyed in a high-level parameter set such as SPS or DPS. Alternatively or additionally, one or more tables similar to Table 1 may be defined in a video coding technique or standard and may be selected, for example, via SPS or DPS.

[0109] How the above-coded upsample or downsample coefficients (ARC information) can be included in a video coding technique or standard syntax is described above. Similar considerations can be applied to one or a few codewords that control an upsample or downsample filter. Below, we will discuss the case where a filter or other data structure requires relatively more data.

[0110] 9, in an example such as H.263 Annex P, ARC information (902) in the form of four warping coordinates is included in the picture header (901), specifically the H.263 PLUSPTYPE (903) header extension. The picture header (901) including the ARC information (902) and the H.263 PLUSPTYPE (903) header extension can be applied when i) a picture header is available and ii) frequent changes of the ARC information are expected. However, the overhead when using H.263-style signaling can be high, and because picture headers can be transient in nature, the scaling factors may not be applicable across picture boundaries.

[0111] Referring to FIG. 9, in an example such as JVCET-M 135-v1, ARC reference information (905) (e.g., index) located in a PPS (904) can point to a table (or target resolution table) (906) containing target resolutions (e.g., resolutions 1-3) where the table (906) is located in an SPS (907). Placing the target resolutions in the table (906) in an SPS (907) can be justified by using the SPS as an interoperability negotiation point during capability exchange. Resolutions can vary from one image to another, within the limits set by the values ​​(e.g., resolutions 1-3) in the table (906), by referencing the appropriate PPS (904).

[0112] 9, additional options or examples may be used to convey ARC information in the video bitstream. Multiple options may exist simultaneously for the same video encoding technology or standard.

[0113] In one embodiment, the ARC information (909), such as the resampling factor (or zoom factor), can be present in a header, such as a slice header, a GOB header, a tile header, a tile group header, etc. A tile group header (908) is shown, for example, in Figure 9. This embodiment may be appropriate when the memory space for the ARC information (909) is small, such as a single variable-length ue(v) or a fixed-length codeword of a few bits, as described above.

[0114] Having the ARC information (909) directly in a header (e.g., the tile group header (908), slice header, or tile header of FIG. 9) may have an additional advantage in that the ARC information (909) may be applicable to, for example, a sub-image represented by a corresponding tile group (or slice, tile) rather than to the entire image. Furthermore, in examples, even if a video compression technology or standard only contemplates whole-image adaptive resolution changes (as opposed to, for example, tile-group-based adaptive resolution changes), putting the ARC information (909) in the tile group header (908) versus putting it in an H.263-style image header may have certain advantages in terms of error resiliency.

[0115] With reference to FIG. 9, the ARC information (912) may reside in a parameter set (911), such as a PPS, a header parameter set, a tile parameter set, an adaptation parameter set (APS), etc. An APS (911) is shown, for example, in FIG. 9. The scope of the parameter set (911) may advantageously be an image, e.g., a tile group, or less. The use of the ARC information (e.g., ARC information (912)) may be implicit by the activation of an associated parameter set (e.g., APS (911)). For example, if a video coding technology or standard only contemplates image-based ARC, then a PPS or equivalent may be appropriate.

[0116] In some embodiments, the ARC reference information (913) may be present in a tile group header (914) or a similar data structure (e.g., a picture header, a slice header, a tile header, or a GOP header), as described above. A tile group header (914) is shown, for example, in FIG. 9. The ARC reference information (913) may reference a subset of the ARC information (915) available in a parameter set (916) that has scope beyond a single picture, such as, for example, an SPS, a DPS, etc. An SPS (916) is shown, for example, in FIG. 9.

[0117] 10 is a diagram illustrating an example of an ARC parameter signaling according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a syntax diagram used in a video coding standard. In one example, the notation of the syntax diagram roughly follows C-style programming. Bolded lines may indicate syntax elements present in the bitstream, and non-bolded lines may indicate control flow(s) or setting of variables.

[0118] Referring to FIG. 10, the tile group header (1001) as an example syntax structure of a header applicable to a portion (e.g., a rectangular portion) of an image can conditionally include a variable-length Exp-Golomb coding syntax element dec_pic_size_idx (1002) (depicted in bold). The presence of a syntax element (e.g., dec_pic_size_idx (1002)) in the tile group header (1001) can be gated based on an adaptive resolution represented, for example, by a flag (e.g., adaptive_pic_resolution_change_flag) (1003). The value of the flag (e.g., adaptive_pic_resolution_change_flag) (1003) is not shown in bold, and thus the flag is present in the bitstream at the point where the flag occurs in the syntax diagram. Whether adaptive resolution is used for an image or a portion of an image can be signaled in a higher level syntax structure (e.g., SPS (1010) as described below) inside or outside the bitstream.

[0119] Referring to Figure 10, an excerpt of an SPS (1010) is shown. The first syntax element shown is a flag (e.g., adaptive_pic_resolution_change_flag) (1011). If the flag is true, the flag can indicate the use of adaptive resolution, which may require specific control information. In one example, the specific control information is conditionally present based on the value of the flag (1011), as indicated by the if() statement (1012) in the SPS (1010) and the tile group header (1001).

[0120] As shown in the example of FIG. 10, when adaptive resolution is used, an output resolution (or output image resolution) (1013) in samples may be coded. The output resolution (1013) may refer to both a width resolution (e.g., output_pic_width_in_luma_samples) and a height resolution (e.g., output_pic_height_in_luma_samples). The width and height resolution may define the output resolution (1013). A video coding technology or standard may define specific limitations on the value(s) of the output resolution (1013). For example, a level definition may limit the total number of output samples (e.g., the product of output_pic_width_in_luma_samples and output_pic_height_in_luma_samples). A video encoding technology or standard, or an external technology or standard (e.g., a system standard), may limit the range (e.g., the width and / or height resolution is divisible by a power of 2), the aspect ratio (e.g., the ratio of width resolution to height resolution is 4:3 or 16:9), etc. In one example, the above restrictions may be introduced to facilitate hardware implementation.

[0121] In certain applications, an encoder can instruct a decoder to use a particular reference picture size rather than implicitly assuming that size is the output picture size. For example, a syntax element (e.g., reference_pic_size_present_flag) (1014) gates the conditional presence of a reference picture dimension (1015). The reference picture dimension (1015) can refer to both a width (e.g., reference_pic_width_in_luma_samples) and a height (e.g., reference_pic_height_in_luma_samples).

[0122] Figure 10 shows a table of applicable decoded image widths and heights. The table can be represented by a table representation (e.g., syntax element num_dec_pic_size_in_luma_samples_minus1) (1016). "minus1" can refer to an interpretation of the value of the syntax element (1016). For example, if the encoded value is 0, there is one table entry. If the encoded value is 5, there are six table entries. For each row of the table, the width and height of the decoded image are included in the syntax element (1017).

[0123] The table entries represented by the syntax element (1017) can be indexed using the syntax element dec_pic_size_idx (1002) in the tile group header (1001), thus allowing different decoding sizes and zoom factors for each tile group.

[0124] Certain video coding techniques or standards (e.g., VP9) may support spatial scalability by implementing a particular form of reference image resampling in conjunction with temporal scalability, thus enabling spatial scalability. In one embodiment, the reference image is upsampled to a higher resolution using ARC-style techniques to form the basis of the spatial enhancement layer. The upsampled image may be refined using normal prediction mechanisms (e.g., motion compensated prediction for inter prediction from the reference image) at high resolution, e.g., to add detail.

[0125] The present disclosure can be used in such environments. In some examples, values ​​in the Network Abstraction Layer (NAL) unit header, e.g., the Temporal ID field, are used to indicate the temporal and spatial layers. Advantages include the ability to use existing Selected Transport Units (SFUs) without modification in scalable environments. SFUs can be created and optimized for transport selected in the temporal layer based on the NAL unit header Temporal ID value. To enable this, a mapping can be performed between the coded picture size and the temporal layer, as indicated by the Temporal ID field in the NAL unit header.

[0126] When an image is encoded into a bitstream containing one or more layers with different qualities, the bitstream may have a syntax element that specifies which layer(s) can be output at the decoder side. The set of layers that are output can be defined as an output layer set. For video codecs that support multiple layers and scalability, one or more output layer sets can be signaled in one or more VPSs. One or more syntax elements that specify PTL information such as the entire bitstream, CVS, and each output layer set can be signaled in high-level syntax structures such as VPS, DPS, DCI, SPS, PPS, APS, GOP, sequence, header, and SEI messages.

[0127] Block-level coding tools may be used to decode pixels or samples in an image to reconstruct the image. Block-level coding tools may include any suitable coding tool(s) used to reconstruct a coding block, such as coding tools for inter prediction (or inter coding tools), coding tools for intra prediction (or intra coding tools), adaptive loop filters (ALFs), deblocking filters (DBFs), entropy coding, transforms, etc.

[0128] High-level syntax (HLS) can specify information about capabilities, system interfaces, picture-level control of tools and buffer control, etc. For example, HLS may specify partitions (e.g., tiles, slices, sub-images), buffer management, random access (e.g., IDR, clean random access (CRA)), parameter set(s) (e.g., VPS, SPS, PPS, APS), reference picture resampling (RPR), scalability, etc. High-level syntax can be above the block level.

[0129] The control information may include SPS level tool control information, PPS level tool control information, sequence level control information, bitstream level control information, and the like.

[0130] The constraint flags can be part of the HLS structure.

[0131] In one example, the constraint flag indicates the control of the tool. The constraint flag may be provided in one of the sequence level control information and the bitstream level control information. In one example, if a particular tool is disabled by the constraint flag, the control information is not present in the HLS, and the tool is not used, for example, to encode blocks within a range corresponding to the HLS.

[0132] Constraint flags can be signaled in the profile syntax structure.

[0133] The constraint information may indicate whether a particular tool, feature, or syntax element is present or used in the corresponding video stream or encoded video bitstream. For example, the constraint flag may indicate whether an inter-coding tool, an intra-coding tool, a DBF, an entropy coding, a transform, a partition (e.g., tile, slice), a buffer management, a random access (e.g., IDR), a parameter set (e.g., SPS, PPS), etc. are present or used in the encoded video bitstream. The constraint information may be signaled in a parameter set (e.g., SPS, VPS, or DCI). The constraint flag may be signaled in a high-level syntax structure (e.g., SPS, VPS, DCI).

[0134] 11 illustrates an example of a set of PTL syntax elements according to one embodiment of the present disclosure. The set of PTL syntax elements may include general_profile_idc, general_tier_flag, general_level_idc, num_sub_profiles, general_sub_profile_idc, sublayer_level_present_flag, ptl_alignment_0_bit, sublayer_level_idc, and general constraint information.

[0135] The generic constraint information may include constraint information regarding video source type, encoding tool, capabilities, etc. FIG. 12 illustrates an example of generic constraint information according to one embodiment of the present disclosure. The generic constraint information may include multiple constraint flags as illustrated in FIG. 12. Referring to FIG. 12, a constraint flag (e.g., intra_only_constraint_flag) (1205) equal to 1 indicates that sh_slice_type is I (i.e., the slice is an intra slice). A constraint flag (e.g., intra_only_constraint_flag) (1205) equal to 0 imposes no constraint (e.g., sh_slice_type is I) on all coded pictures within the scope of the PTL information where other information (e.g., profile_idc) may allow non-intra slices. In one example, a constraint flag (e.g., no_alf_constraint_flag) (1206) equal to 1 may indicate that sps_alf_enabled_flag is equal to 0 for all CVSs within the scope of the PTL information, and thus adaptive loop filtering is not used even if adaptive loop filtering is allowed based on, for example, the profile_idc. A constraint flag (e.g., no_alf_constraint_flag) (1206) equal to 0 does not impose the above constraint.

[0136] As shown in Fig. 12, a constraint flag (e.g., no_lossless_coding_tool_constraint_flag) (1201) can be signaled in the generic constraint information. A constraint flag (e.g., no_lossless_coding_tool_constraint_flag) (1201) equal to 1 can indicate that no coding tools related to lossless coding can be used within the scope of the PTL information including the constraint flag (1201). A constraint flag (e.g., no_lossless_coding_tool_constraint_flag) (1201) equal to 0 does not impose the above constraint.

[0137] As shown in Fig. 12, a constraint flag (e.g., no_lossy_coding_tool_constraint_flag) (1202) can be signaled in the generic constraint information. A constraint flag (e.g., no_lossy_coding_tool_constraint_flag) (1202) equal to 1 can indicate that no coding tools related to lossy coding can be used within the scope of the PTL information including the constraint flag (1202). A constraint flag (e.g., no_lossy_coding_tool_constraint_flag) (1202) equal to 0 does not impose the above constraint.

[0138] In one embodiment, if the constraint flag (e.g., no_lossy_coding_tool_constraint_flag) (1202) is equal to 1, then the constraint flag (e.g., no_lossless_coding_tool_constraint_flag) (1201) may not be equal to 1. Alternatively, if the constraint flag (e.g., no_lossless_coding_tool_constraint_flag) (1201) is equal to 1, then the constraint flag (e.g., no_lossy_coding_tool_constraint_flag) (1202) may not be equal to 1.

[0139] The multiple constraint flags in the generic constraint information may be sorted in a certain order. The order may be set, for example, based on the likelihood of each mechanism and / or tool not being used in the scope of the PTL. This order may be referred to as a priority order. In the generic constraint information syntax structure, an order from high priority to low priority may be presented, with a high priority indicating a high probability of non-use of the tool (or mechanism) and a low priority indicating a low probability of non-use of the tool (or mechanism). Additional factors influencing the order may include tools that are likely to be used only for a particular use case (e.g., tools for sub-image, scalability, and / or interlace support), the impact of the tool on the encoder / decoder / implementation complexity, etc.

[0140] 13A-13B show an example of PTL information (e.g., including a set of PTL syntax elements) according to an embodiment of the present disclosure. In this PTL information, a syntax element indicating the number of constraint flags (e.g., num_available_constraint_flags) (1301) can be signaled. For example, the constraint flag number can be signaled outside the generic constraint information bracket as shown in FIG. 13A. Alternatively, the syntax element indicating the number of constraint flags can be signaled at the beginning of the generic constraint information syntax structure. When the syntax element (e.g., num_available_constraint_flags) is present and the value of the syntax element (e.g., num_available_constraint_flags) is equal to N, the first N constraint flags can be present in the generic constraint information syntax structure. Furthermore, other constraint flags may not be present and can be inferred to be equal to a specific value. N can be a non-negative integer.

[0141] In one embodiment, the value N (e.g., num_available_constraint_flags) ranges from 0 to the maximum number of constraint flags (e.g., MaxNumConstraintFlags). The maximum number of constraint flags may be any positive integer. The value of the maximum number of constraint flags (e.g., MaxNumConstraintFlags) may be predefined to be 16, 32, 64, 128, etc. When the value N (e.g., num_available_constraint_flags) is equal to 0, there are no constraint flags in the generic constraint information syntax structure. The encoding of the value N (e.g., num_available_constraint_flags) may be selected such that the sum of the value N and the entropy encoded representations corresponding to the constraint flags is divisible by 8 to ensure byte alignment.

[0142] According to aspects of the present disclosure, the constraint flags can be classified into one or more constraint information groups. Each constraint information group can include one or more constraint flags and can have a corresponding gating flag. The gating flag of the corresponding constraint information group can indicate whether the constraint flag(s) in the corresponding constraint information group may be present. In one example, the gating flag can be referred to as a constraint group present flag. In general, the gating flag is associated with the corresponding constraint information group and associated with the constraint flag(s) in the corresponding constraint information group. In one embodiment, the gating flag gates whether the constraint flag(s) in the corresponding constraint information group are present (or signaled) in the constraint information. For example, if the gating flag of the corresponding constraint information group is 1, the constraint flag(s) corresponding to the constraint information group may be present, for example, in the constraint information. For example, if the gating flag of the corresponding constraint information group is 0, the constraint flag(s) corresponding to the constraint information group may not be present, for example, in the constraint information. In one example, if all gating flags are equal to 0, no constraint flags are present.

[0143] Constraint flags can have different scopes. For example, the scope of a constraint flag in a DCI can be the coded video bitstream. The scope of a constraint flag in a VPS can be a CLVS with multiple layers. The scope of a constraint flag in an SPS can be a single CLVS.

[0144] The flag (e.g., gate_flag_all) may indicate whether gate flag(s) are present. In an example, the flag is false, indicating that gate flag(s) are not present. In one example, if gate flag(s) are not present, then no constraint flags are present in the constraint information. In one example, the flag is true, indicating that one or more gate flag(s) are present.

[0145] According to aspects of the present disclosure, a gating flag for each of at least one constraint information group in the constraint information signaled in the encoded video bitstream may be obtained. Each gating flag may indicate whether a constraint information group of the at least one constraint information group corresponding to the respective gating flag is present in the constraint information. The constraint information may relate to a subset of coding blocks in the encoded video bitstream. Whether a constraint information group of a gating flag is present in the constraint information may be determined based on a gating flag of the constraint information group of the at least one constraint information group. The constraint information group of the gating flag may include at least one constraint flag for the subset of coding blocks. Prediction information of the subset of coding blocks may be determined based on whether a constraint information group of the gating flag is present in the constraint information. The subset of coding blocks may be reconstructed based on the prediction information.

[0146] In one embodiment, the constraint information group of the gating flag is determined to be present in the constraint information. The prediction information of the subset of the coding blocks can be determined based on at least one constraint flag in the constraint information group of the gating flag. In one embodiment, the constraint information group of the gating flag is determined to be absent in the constraint information.

[0147] In one embodiment, each gate flag indicates that the constraint information group corresponding to the respective gate flag is not present in the constraint information and that no constraint flag is present in the constraint information.

[0148] In one embodiment, the constraint information is in a high level syntax for at least one image, and the subset of the coding blocks includes coding blocks in the at least one image. The high level syntax may be a VPS, DPS, DCI, SPS, PPS, APS, GOP, sequence, etc. In one example, the high level syntax is a DCI or DPS, and the scope of the constraint information includes the coded video bitstream and thus includes coding blocks in the coded video bitstream. In one example, the high level syntax is a VPS, and the scope of the constraint information includes a CLVS having multiple layers and thus includes coding blocks in the CLVS. In one example, the high level syntax is an SPS, and the scope of the constraint information includes a single CLVS and thus includes coding blocks in the single CLVS.

[0149] In one embodiment, each constraint flag that may be present (or signaled) in the constraint information may be classified into one or more constraint information groups and thus gated by a respective gate flag. In one example, the one or more constraint information groups include a first constraint information group and a second constraint information group. The constraint flags include at least one first constraint flag and at least one second constraint flag. The at least one first constraint flag is classified into the first constraint information group, and the at least one second constraint flag is classified into the second constraint information group. The first gate flag of the first constraint information group may indicate whether at least one first constraint flag (or first constraint information group) is present (e.g., signaled) in the constraint information. The second gate flag of the second constraint information group may indicate whether at least one second constraint flag (or second constraint information group) is present (e.g., signaled) in the constraint information. Thus, in this example, the constraint information does not include additional constraint flag(s) to the at least one first constraint flag and the at least one second constraint flag.

[0150] In one embodiment, a first subset of the constraint flags is classified into one or more constraint information groups and can therefore be gated by the respective gating flags. Furthermore, a second subset of the constraint flags is outside the one or more constraint information groups and therefore is not gated by the gating flag(s). In one example, the one or more constraint information groups include a third constraint information group and a fourth constraint information group. The first subset of the constraint flags includes at least one third constraint flag and at least one fourth constraint flag. The at least one third constraint flag is classified into the third constraint information group and the at least one fourth constraint flag is classified into the fourth constraint information group. The third gating flag of the third constraint information group can indicate whether at least one third constraint flag (or third constraint information group) is present (e.g., signaled) in the constraint information. The fourth gating flag of the fourth constraint information group can indicate whether at least one fourth constraint flag (or fourth constraint information group) is present (e.g., signaled) in the constraint information. Further, the constraint information may include a second subset of constraint flags. In one example, the third gate flag and the fourth gate flag indicate that the third constraint information group and the fourth constraint information group are not notified in the constraint information, and thus the constraint information includes the second subset of constraint flags and does not include at least one third constraint flag and at least one fourth constraint flag. In one example, the third gate flag and the fourth gate flag indicate that the third constraint information group and the fourth constraint information group are notified in the constraint information, and thus the constraint information includes the second subset of constraint flags, at least one third constraint flag and at least one fourth constraint flag.

[0151] 14A-14B illustrate an example generalized constraint information syntax structure 1400 according to one embodiment of the present disclosure. The generalized constraint information syntax structure 1400 can represent constraint information (e.g., referred to as generalized constraint information). The general constraint information (or general constraint information syntax structure (1400)) may include one or more gate flags, such as a gate flag (e.g., general_frame_structure_constraint_group_flag) (1401), a gate flag (e.g., high_level_functionality_constraint_group_flag) (1402), a gate flag (e.g., scalability_constraint_group_flag) (1403), a gate flag (e.g., partitioning_constraint_group_flag) (1404), a gate flag (e.g., intra_coding_tool_constraint_group_flag) (1405), a gate flag (e.g., inter_coding_tool_constraint_group_flag) (1406), a gate flag (e.g., transform_contrain_group_flag) (1407), or a gate flag (e.g., inloop_filtering_constraint_group_flag) (1408) of FIG. 14A. As shown in FIG. 14A, one or more gate flags (eg, gate flags (1401)-(1408)) may be present at the beginning of a general constraint information syntax structure (1400).

[0152] A gate flag (e.g., general_frame_structure_constraint_group_flag) (1401) is associated with a constraint information group (1410), and is associated with constraint flags (1411) to (1414) within the constraint information group (1410). When the gate flag (e.g., general_frame_structure_constraint_group_flag) (1401) is set to 1, it is possible to specify that the constraint flags (1411) to (1414) may exist within the constraint information group (1410).

[0153] The constraint information group (1410) (or constraint flags (1411)-(1414)) may correspond to an input source and a frame packing (e.g., packed frame or projected frame). With reference to FIG. 14A , the constraint flags (1411)-(1414) correspond to the general_non_packed_constraint_flag (1411), the general_frame_only_constraint_flag (1412), the general_non_projected_constraint_flag (1413), and the general_one_picture_only_constraint_flag (1414). Otherwise, a gate flag (e.g., the general_frame_structure_constraint_group_flag) (1401) equal to 0 may specify that the constraint flags (1411)-(1414) within the constraint information group (1410) may not be present.

[0154] 14B, a gating flag (e.g., high_level_functionality_constraint_group_flag) (1402) equal to 1 may specify that there may be constraint flags associated with high-level functionality (e.g., reference image resampling) in the constraint information group (1420). Otherwise, a gating flag (e.g., high_level_functionality_constraint_group_flag) (1402) equal to 0 may specify that there may not be constraint flags in the constraint information group (1420).

[0155] 14A, a gate flag (e.g., scalability_constraint_group_flag) (1403) equal to 1 may specify that constraint flag(s) for scalability (e.g., inter-layer prediction) may be present. Otherwise, constraint flag(s) for scalability may not be present.

[0156] A gate flag (e.g., partitioning_constraint_group_flag) (1404) equal to 1 may specify that there may be constraint flag(s) associated with a higher-level partition (e.g., a subimage or tile). Otherwise, there may be no constraint flags associated with the higher-level partition.

[0157] A gate flag (e.g., intra_coding_tool_constraint_group_flag) (1405) equal to 1 may specify that constraint flag(s) for intra coding (e.g., intra prediction) may be present. Otherwise, constraint flag(s) for intra coding may not be present.

[0158] A gate flag (e.g., inter_coding_tool_constraint_group_flag) (1406) equal to 1 may specify that constraint flag(s) for inter-coding (e.g., motion compensation for inter-picture prediction) may be present. Otherwise, constraint flags for inter-coding may not be present.

[0159] A gate flag (e.g., transform_contrain_group_flag) (1407) equal to 1 may specify that there may be constraint flag(s) associated with the transform coding (e.g., multiple transform matrices). Otherwise, there may be no constraint flags associated with the transform coding.

[0160] In one embodiment, if all gate flags (eg, gate flags (1401)-(1408) in FIG. 14A) are equal to 0, then no constraint flags are present in the constraint information syntax structure (eg, general constraint information syntax structure (1400)).

[0161] According to an aspect of the present disclosure, the syntax can be designed such that the gate flags (e.g., gate flags (1401)-(1408)), associated constraint flags (e.g., constraint flags (1411)-(1412) and constraint flags in the constraint information group (1420)), additional control information, etc., are byte-aligned, e.g., some flags are divisible by 8 to maintain byte alignment. In one example, the number of gate flags and constraint flags in the constraint information (e.g., the generic constraint information syntax structure (1400)) is divisible by 8. A byte alignment mechanism can be used to achieve byte alignment of the control information. Referring to FIG. 14B, a syntax (e.g., a while loop) (1430) can be used for byte alignment.

[0162] In one embodiment, offset information, such as an offset (e.g., syntax element constraint_info_offset[]) and length information, such as a length (e.g., syntax element constraint_info_length[]), are present in the constraint information (e.g., at the beginning of a generic constraint information syntax structure) to indicate that at least one constraint flag in the constraint information group of the gate flag is present in the constraint information. In one embodiment, one or more of the at least one constraint information group are present in the encoded video bitstream. An offset and a length may be present in the constraint information to indicate that at least one constraint flag in each of one or more constraint information groups of the at least one constraint information group is present in the constraint information. Some offsets may be signaled in the encoded video bitstream.

[0163] In one embodiment, zero or more offsets (or constraint information offsets) (e.g., syntax element constraint_info_offset[]) and corresponding zero or more lengths (or constraint information lengths) (e.g., indicated by syntax element constraint_info_length[]) may be present within the constraint information, such as at the beginning of a generic constraint information syntax structure. The number of zero or more constraint information offsets and the number of zero or more constraint information lengths may be explicitly indicated by the syntax element(s).

[0164] In one example, the number of constraint information offsets that is 0 or more and the number of constraint information lengths that is 0 or more are the same, and the number of constraint information offsets that is 0 or more (represented by M) is explicitly indicated (or signaled) by a syntax element (e.g., num_constraint_info_set). M may be an integer that is 0 or more. M being 0 may indicate that there are no constraint information offsets and constraint information lengths in the constraint information.

[0165] In one embodiment, the constraint information offset (e.g., syntax element constraint_info_offset[i]) and the constraint information length (e.g., syntax element constraint_info_length[i]) can specify the available constraint flags present in the constraint information (e.g., the generic constraint information syntax structure). In one example, if the constraint information offset (e.g., syntax element constraint_info_offset[i]) has a value of 5 and the constraint information length (e.g., syntax element constraint_info_length[i]) has a value of 3, the fifth, sixth, and seventh constraint flags are present in the constraint information (e.g., the generic constraint information syntax structure).

[0166] In one example, when M is equal to 0, no constraint flags are present in the constraint information (eg, the general constraint information syntax structure).

[0167] In one example, run-length encoding may be used to encode the constraint flags that are specified in a predefined order (or a given sequence).

[0168] In one embodiment, run encoding can be used when constraint flags are specified in a predefined order (or a given sequence). Instead of directly encoding the constraint flags, a suitably encoded list of "skip" values ​​can indicate constraint flags equal to 0, with the following constraint flags being implied to be equal to 1. The above run encoding can be particularly efficient when (i) the number of constraint flags is large and (ii) a small percentage of the constraint flags are equal to 1.

[0169] In one embodiment, one or more of the at least one constraint information group are present in the encoded video bitstream. The constraint flags in the one or more of the at least one constraint information group are signaled according to a predetermined order. This allows the constraint flags to be run-coded (e.g., run-encoded or run-decoded). Furthermore, prediction information for a subset of the coding blocks can be determined based on the constraint flags.

[0170] In one embodiment, at least one constraint flag in the constraint information group of the gate flag includes multiple constraint flags that are notified according to a predetermined order, so that the multiple constraint flags can be run-coded (e.g., run-encoded or run-decoded).

[0171] In one embodiment, the complete list of constraint flags may be specified in a video encoding standard (e.g., the VVC specification), an external table, etc. In one example, only available constraint flags of the constraint flags are indicated, for example, by one or more of the following, present in the encoded video stream: the number of available constraint flags (e.g., num_available_constraint_flags), a gate flag (or constraint group present flag), constraint information offset information, and constraint information length information.

[0172] In one example, a complete list of constraint flags is specified and available at the encoder and the decoder. The complete list of constraint flags can be stored in the decoder. The complete list of constraint flags can include 100 constraint flags. Ten of the 100 constraint flags are present in the constraint information of the CLVS and are therefore available to a subset of the coding blocks in the CLVS. The ten of the 100 constraint flags are referred to as ten available constraint flags. In one example, the number of available constraint flags (e.g., ten) is signaled. In one example, the ten available constraint flags are in two constraint information groups and are gated by a first gate flag and a second gate flag. Thus, the first gate flag and the second gate flag can be signaled to indicate the ten available constraint flags.

[0173] In one example, a first constraint information offset (e.g., syntax element constraint_info_offset[0]) and a first constraint information length (e.g., syntax element constraint_info_length[0]) are signaled. A second constraint information offset (e.g., syntax element constraint_info_offset[1]) and a second constraint information length (e.g., syntax element constraint_info_length[1]) are signaled. For example, the syntax element constraint_info_offset[0] is 15, the syntax element constraint_info_length[0] is 3, the syntax element constraint_info_offset[1] is 82, and the syntax element constraint_info_length[1] is 7, thus indicating that the 15th to 17th constraint flags and the 82nd to 88th constraint flags of the entire list (e.g., 100 constraint flags) are available or present in the constraint information.

[0174] In one embodiment, any of the various techniques (or methods, embodiments, examples) for efficiently encoding the constraint flags can be combined using appropriate control information. The combination can be any suitable combination of two or more of such techniques. Alternatively, one of the various techniques (or methods, embodiments, examples) can be used independently. The constraint flags can be grouped. Certain group(s) can use run encoding, while other group(s) can use simple binary encoding.

[0175] The value of the maximum number of constraint flags (eg, MaxNumConstraintFlags) can be predefined to be 16, 32, 64, 128, and so on.

[0176] Since the value of the maximum number of constraint flags (e.g., MaxNumConstraintFlags) can be determined by profile information such as general_profile_idc and general_sub_profile_idc, or codec version information, the range of the number of constraint flags (e.g., num_available_constraint_flags (1301)) can be limited by profile information or version information. For example, the value of the number of constraint flags (e.g., num_available_constraint_flags (1301)) in the main profile (e.g., where MaxNumConstraintFlags=64) can be in the range of 0 to 64, while the value of the number of constraint flags (e.g., num_available_constraint_flags (1301)) in the advanced profile (e.g., where MaxNumConstraintFlags=128) can be in the range of 0 to 128.

[0177] In one embodiment, the value of the number of constraint flags (e.g., num_available_constraint_flags) may be inferred to be equal to a value predefined by profile information such as general_profile_idc or general_sub_profile_idc, or codec version information, such that the value of num_available_constraint_flags can be determined without explicit notification.

[0178] In one embodiment, in FIG. 12, the reserved byte information can be present in the generic constraint information syntax structure. For example, the flags gci_num_reserved_bytes (1203) and gci_reserved_bytes[] (1204) can be present in the generic constraint information syntax structure to extend the generic constraint information syntax structure. The flag gci_num_reserved_bytes can specify the number of reserved constraint bytes. In one example, the reserved constraint bytes are for signaling additional flags (e.g., additional constraint flags). The flag gci_reserved_byte[] can have any suitable value.

[0179] In one embodiment, the value of gci_num_reserved_bytes may be limited or determined by profile information such as general_profile_idc or general_sub_profile_idc, or codec version information. In the Basic Profile (or Main Profile), the value of the flag gci_num_reserved_bytes may be 0. In the Extended Profile (or Advanced Profile), the value of gci_num_reserved_bytes may be greater than 0.

[0180] The field sequence flag may be signaled in the coded video bitstream. The field sequence flag may indicate whether the images in the output layer are coded with field coding. The field sequence flag may be signaled in the SPS as sps_field_seq_flag. In one embodiment, the flag sps_field_seq_flag may be present in the SPS. The flag sps_field_seq_flag equal to 1 may indicate that the CLVS carries images representing fields. The flag sps_field_seq_flag equal to 0 may indicate that the CLVS carries images representing frames.

[0181] In the general constraint information syntax structure of FIG. 12, a flag general_frame_only_constraint_flag may be present. The flag general_frame_only_constraint_flag being 1 may specify that the scope of the output layer set (e.g., OlsInScope) carries images that represent frames. The flag general_frame_only_constraint_flag being equal to 0 specifies that the scope of the output layer set (e.g., OlsInScope) carries images that may or may not represent frames. In one embodiment, the flag general_frame_only_constraint_flag indicates whether the images in the output layer set are coded with field coding. The output layer set may include a subset of coded blocks. The flag sps_field_seq_flag may be false based on the flag general_frame_only_constraint_flag (e.g., being 1) indicating that the subset of images is not coded with field coding. The subset of images may be within one layer of the output layer set.

[0182] When the flag general_frame_only_constraint_flag is equal to 1, the value of the flag sps_field_seq_flag may be equal to 0.

[0183] In one embodiment, the flag pps_mixed_nalu_types_in_pic_flag may be present in the PPS. The flag pps_mixed_nalu_types_in_pic_flag equal to 1 may specify that each picture that references the PPS has two or more VCL NAL units, and that the VCL NAL units do not have the same value of nal_unit_type. The flag pps_mixed_nalu_types_in_pic_flag equal to 0 may specify that each picture that references the PPS has one or more VCL NAL units, and that the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type. In the generic constraint information syntax structure of FIG. 12, the flag no_mixed_nalu_types_in_pic_constraint_flag may be present. The flag no_mixed_nalu_types_in_pic_constraint_flag equal to 1 may specify that the value of pps_mixed_nalu_types_in_pic_flag is equal to 0. If the flag no_mixed_nalu_types_in_pic_constraint_flag is equal to 0, no such constraint is imposed.

[0184] In one embodiment, the flag general_one_picture_only_constraint_flag may be present in the general constraint information syntax structure of Figure 12. When general_one_picture_only_constraint_flag is equal to 1, it may specify that there is only one picture coded in the bitstream. When the flag general_one_picture_only_constraint_flag is equal to 0, no such constraint is imposed.

[0185] In one embodiment, the flag single_layer_constraint_flag may be present in the general constraint information syntax structure of Figure 12. The flag single_layer_constraint_flag equal to 1 may specify that sps_video_parameter_set_id is equal to 0. When the flag single_layer_constraint_flag is equal to 0, no such constraint is imposed. When the flag general_one_picture_only_constraint_flag is 1, the value of the flag single_layer_constraint_flag may be 1.

[0186] In one embodiment, the flag all_layers_independent_constraint_flag may be present in the generic constraint information syntax structure of Figure 12. The flag all_layers_independent_constraint_flag equal to 1 may specify that the flag vps_all_independent_layers_flag may be equal to 1. When the flag all_layers_independent_constraint_flag is equal to 0, no such constraint is imposed. When the flag single_layer_constraint_flag is equal to 1, the value of the flag all_layers_independent_constraint_flag may be equal to 1.

[0187] In one embodiment, the flag no_res_change_in_clvs_constraint_flag may be present in the generic constraint information syntax structure of Figure 12. The flag no_res_change_in_clvs_constraint_flag equal to 1 may specify that the flag sps_res_change_in_clvs_allowed_flag is equal to 0. If the flag no_res_change_in_clvs_constraint_flag is equal to 0, no such constraint is imposed. If the flag no_ref_pic_resampling_constraint_flag is 1, the value of the flag no_res_change_in_clvs_constraint_flag may be equal to 1.

[0188] In one embodiment, the flag no_mixed_nalu_types_in_pic_constraint_flag may be present in the generic constraint information syntax structure of Figure 12. The flag no_mixed_nalu_types_in_pic_constraint_flag being 1 specifies that the value of the flag pps_mixed_nalu_types_in_pic_flag may be 0. When the flag no_mixed_nalu_types_in_pic_constraint_flag is equal to 0, no such constraint is imposed. When the flag one_subpic_per_pic_constraint_flag is 1, the value of the flag no_mixed_nalu_types_in_pic_constraint_flag may be 1.

[0189] In one embodiment, the flag no_trail_constraint_flag may be present in the general constraint information syntax structure of Figure 12. The flag no_trail_constraint_flag being 1 may specify that there may be no NAL units with nuh_unit_type equal to TRAIL_NUT present in OlsInScope. When the flag no_trail_constraint_flag is equal to 0, no such constraint is imposed. When the flag general_one_picture_only_constraint_flag is equal to 1, the flag no_trail_constraint_flag may be equal to 1.

[0190] In one embodiment, the flag no_stsa_constraint_flag may be present in the general constraint information syntax structure of Figure 12. The flag no_stsa_constraint_flag being 1 may specify that there may be no NAL units with nuh_unit_type equal to STSA_NUT present in OlsInScope. If the flag no_stsa_constraint_flag is equal to 0, no such constraint is imposed. If the flag general_one_picture_only_constraint_flag is equal to 1, the flag no_stsa_constraint_flag may be equal to 1.

[0191] In one embodiment, the flag no_trail_constraint_flag may be present in the general constraint information syntax structure of Figure 12. The flag no_trail_constraint_flag being 1 may specify that there may be no NAL units with nuh_unit_type equal to TRAIL_NUT present in OlsInScope. When the flag no_trail_constraint_flag is equal to 0, no such constraint is imposed. When the flag general_one_picture_only_constraint_flag is equal to 1, the flag no_trail_constraint_flag may be equal to 1.

[0192] In one embodiment, the flag no_stsa_constraint_flag may be present in the general constraint information syntax structure of Figure 12. The flag no_stsa_constraint_flag being 1 may specify that there may be no NAL units with nuh_unit_type equal to STSA_NUT present in OlsInScope. If the flag no_stsa_constraint_flag is equal to 0, no such constraint is imposed. If the flag general_one_picture_only_constraint_flag is equal to 1, the flag no_stsa_constraint_flag may be equal to 1.

[0193] In one embodiment, the flag no_idr_constraint_flag may be present in the generic constraint information syntax structure of Figure 12. no_idr_constraint_flag equal to 1 may specify that NAL units with nuh_unit_type equal to IDR_W_RADL or IDR_N_LP may not be present in OlsInScope. When the flag no_idr_constraint_flag is equal to 0, no such constraint is imposed.

[0194] In one embodiment, the flag no_cra_constraint_flag may be present in the generic constraint information syntax structure of Figure 12. The flag no_cra_constraint_flag may be set to 1 to specify that there are no NAL units with nuh_unit_type equal to CRA_NUT present in OlsInScope. When the flag no_cra_constraint_flag is set to 0, no such constraint is imposed.

[0195] In one embodiment, the flag no_rasl_constraint_flag may be present in the generic constraint information syntax structure of Figure 12. The flag no_rasl_constraint_flag being 1 may specify that no NAL units with nuh_unit_type equal to RASL_NUT are present in OlsInScope. When the flag no_rasl_constraint_flag is equal to 0, no such constraint is imposed. When the flag no_cra_constraint_flag is equal to 1, the value of the flag no_rasl_constraint_flag may be equal to 1.

[0196] In one embodiment, the flag no_radl_constraint_flag may be present in the generic constraint information syntax structure of Figure 12. The flag no_radl_constraint_flag being 1 may specify that there are no NAL units with nuh_unit_type of RADL_NUT present in OlsInScope. If the flag no_radl_constraint_flag is equal to 0, no such constraint is imposed. If the flag no_idr_constraint_flag is equal to 1 and the flag no_cra_constraint_flag is equal to 1, the value of the flag no_rasl_constraint_flag may be equal to 1.

[0197] 15A-15D illustrate an exemplary generic constraint information syntax structure (1500) according to one embodiment of the present disclosure. The generic constraint information syntax structure (1500) can represent constraint information (e.g., referred to as generic constraint information). The generic constraint information (or generic constraint information syntax structure (1500)) can include a flag (e.g., gci_present_flag) indicating whether a constraint flag is present in the generic constraint information syntax structure (1500). In one example, if the flag (e.g., gci_present_flag) is true, one or more of the constraint flags can be present in the generic constraint information syntax structure (1500). If the flag (e.g., gci_present_flag) is false, a constraint flag cannot be present in the generic constraint information syntax structure (1500).

[0198] As described above with reference to Figures 14A-14B, the constraint flags in Figures 15A-15D can be classified into groups (e.g., constraint information groups). Whether a constraint flag in a constraint information group can be present in the generic constraint information syntax structure (1500) can be controlled, for example, by a corresponding gate flag for the constraint information group.

[0199] The generic constraint information syntax structure (1500) may include any suitable number of constraint information groups. In one example, the generic constraint information syntax structure (1500) may further include a constraint flag that is not included in a constraint information group, and thus is outside of the constraint information group.

[0200] In the example shown in Figures 15A-15D, the generic constraint information syntax structure (1500) includes nine constraint information groups (1510)-(1518). The constraint information group (1510) is related to generic constraint information and includes three constraint flags (e.g., gci_intra_only_constraint_flag, gci_all_layers_independent_constraint_flag, and gci_one_au_only_constraint_flag). In one example, a gate flag can gate or control whether the constraint information group (1510) is present in the generic constraint information syntax structure (1500).

[0201] The constraint information group (1511) and the corresponding constraint flag relate to the image format constraint information. In one example, a gating flag can gate or control whether the constraint information group (1511) is present in the generic constraint information syntax structure (1500).

[0202] The constraint information group (1512) and the corresponding constraint flag relate to the NAL unit type constraint information. In one example, a gating flag can gate or control whether the constraint information group (1512) is present in the generic constraint information syntax structure (1500).

[0203] The constraint information group (1513) and corresponding constraint flags relate to tile, slice, or sub-image partition constraint information. In one example, a gating flag can gate or control whether the constraint information group (1513) is present in the generic constraint information syntax structure (1500).

[0204] The constraint information group (1514) and corresponding constraint flags relate to CTU and block partition constraint information. In one example, a gate flag can gate or control whether the constraint information group (1514) is present in the generic constraint information syntax structure (1500).

[0205] The constraint information group (1515) and the corresponding constraint flag relate to constraint information for intra-coding. In one example, a gating flag can gate or control whether the constraint information group (1515) is present in the generic constraint information syntax structure (1500).

[0206] The constraint information group (1516) and the corresponding constraint flag relate to the constraint information for inter-coding. In one example, a gating flag can gate or control whether the constraint information group (1516) is present in the generic constraint information syntax structure (1500).

[0207] The constraint information group (1517) and corresponding constraint flags relate to transform, quantization, and residual constraint information. In one example, a gating flag can gate or control whether the constraint information group (1517) is present in the generic constraint information syntax structure (1500).

[0208] The constraint information group (1518) and corresponding constraint flag relate to loop filter constraint information. In one example, a gating flag can gate or control whether the constraint information group (1518) is present in the generic constraint information syntax structure (1500).

[0209] In one embodiment, in FIG. 15D, the syntax elements gci_num_reserved_bits and gci_reserved_0_bit[] may be present in the generic constraint information syntax structure (1500) for the extension of the generic constraint information syntax structure. The syntax element gci_num_reserved_bits may specify the number of reserved constraint bits. The syntax element gci_reserved_zero_bit[] may have any suitable value. In one example, the presence of the syntax element gci_reserved_zero_bit[] and the value of gci_reserved_zero_bit[] do not affect the decoding process specified in a particular video encoding standard (e.g., the VVC specification). The presence of the syntax element gci_reserved_zero_bit[] and the value of gci_reserved_zero_bit[] may be used for some video encoding standards.

[0210] 16 illustrates an example of a set of PTL syntax elements in a PTL syntax structure according to one embodiment of the disclosure. The set of PTL syntax elements may include general_profile_idc, general_tier_flag, general_level_idc, ptl_frame_only_constraint_flag, ptl_multilayer_enabled_flag, plt_sublayer_level_present_flag[], ptl_reserved_zero_bit, sublayer_level_idc[], ptl_num_sub_profiles, general_sub_profile_idc, and / or generic constraint information. Whether some of the set of PTL syntax elements (e.g., general_profile_idc, general_tier_flag, generic constraint information, ptl_num_sub_profiles) may be present may be based on the value of a flag (e.g., profileTierPresentFlag).

[0211] A video sequence with one or more layers may be coded as a field coding structure in which two or more successive field images may contain a full-frame image. The field coding information may be signaled in a high-level syntactic structure such as a parameter set or SEI message.

[0212] In one embodiment, each interlaced video frame has two fields for each frame. In some examples, three-to-two pulldown (3:2 pulldown) can be used to convert 24 frames per second to 29.97 or 30 frames per second. 3:2 pulldown can convert approximately four frames to five frames, and also slow down the speed slightly.

[0213] In one embodiment, a flag (e.g., sps_field_seq_flag) may be signaled in the SPS to indicate whether field coding is used, as shown in FIG. 17. A flag (e.g., sps_field_seq_flag) equal to 1 may indicate that the CLVS carries an image representing a field. A flag (e.g., sps_field_seq_flag) equal to 0 may indicate that the CLVS carries an image representing a frame. If the flag (e.g., general_frame_only_constraint_flag) is equal to 1, the value of the flag (e.g., sps_field_seq_flag) may be equal to 0. In one example, a flag (e.g., general_frame_only_constraint_flag) equal to 1 indicates that the value of the flag (e.g., sps_field_seq_flag) is 0.

[0214] In one embodiment, a frame field information SEI message may be present for all coded pictures in the CLVS when a flag (e.g., sps_field_seq_flag) is equal to 1. A video coding standard or technology may require such presence.

[0215] In one embodiment, the decoding process may not treat images representing fields or frames differently than other images. A sequence of images representing fields may be encoded with the image dimensions of the individual fields. For example, an image representing 1080i fields may have a cropped output dimension of 1920x540, but the sequence image rate may represent the rate of the source fields (e.g., typically between 50 and 60 Hz) instead of the source frame rate (e.g., typically between 25 and 30 Hz).

[0216] In one embodiment, the value of the flag (e.g., sps_field_seq_flag) may be the same across layers. In one example, the value of the flag (e.g., sps_field_seq_flag) may be the same for all SPSs referenced by a CLVS in a CVS, for example, because all layers may have the same field structure or the same frame structure.

[0217] In one embodiment, the value of a flag (eg, sps_field_seq_flag) of layer A is equal to the value of the flag (eg, sps_field_seq_flag) of a dependent layer of layer A in the output layer set.

[0218] In one embodiment, when a flag (e.g., sps_field_seq_flag) is equal to 1, all fields that contain the same frame reference the same PPS, and as a result, all fields of the same frame can have the same image size, matching window, and tile division.

[0219] In one embodiment, two fields of a frame may have the same partitioning structure with the same tile syntax elements or the same sub-picture syntax elements.

[0220] In one embodiment, the two fields of a frame may have the same layer structure with the same number of layers and sub-layers.

[0221] In one embodiment, as shown in FIG. 18, a flag (e.g., vps_field_seq_flag) may be signaled in the VPS to indicate whether a layer referencing the VPS is coded as a field or a frame. A flag (e.g., vps_field_seq_flag) equal to 1 may indicate that a CVS referencing the VPS carries an image representing a field. A flag (e.g., vps_field_seq_flag) equal to 0 may indicate that a CVS referencing the VPS carries an image representing a frame. When a flag (e.g., general_frame_only_constraint_flag) is equal to 1, the value of the flag (e.g., sps_field_seq_flag) shall be equal to 0.

[0222] If a flag (eg, vps_field_seq_flag) is present in a VPS, the value of the flag (eg, sps_field_seq_flag) in an SPS that references the VPS may be equal to the value of the flag (eg, vps_field_seq_flag).

[0223] High-level syntax cleanup in SPS The following high level syntax (HLS) cleanups may be performed in the sequence parameter set (SPS) in accordance with embodiments of the present disclosure: (1) notification sps_picture_header_in_slice_header_flag, (2) inter-layer alignment of subpic_treated_as_pic_flag values, (3) constraints on the number of sub-images for a small image, and (4) value range correction for sps_subpic_id_len_minus1.

[0224] 1. Notify the picture_header_in_slice_header flag When a picture header is included in the slice header, in CLVS referencing sequence parameter set (SPS), sub-picture information is not used and only one slice per picture is assumed. This assumption was hypothetically agreed upon but was not explicitly stated. To clean up HTTP Live Streaming, a new flag sps_picture_header_in_slice_header_flag can be defined and used in SPS. sps_picture_header_in_slice_header_flag equal to 1 specifies that the picture header (PH) syntax structure is present in the slice header and only one slice is present per picture in CLVS. Also, when sps_picture_header_in_slice_header_flag is 1, no sub-picture information is present.

[0225] Below is an example semantic and syntactic definition of the new flag sps_picture_header_in_slice_header_flag: seq_parameter_set_rbsp(){ Descriptor … sps_picture_header_in_slice_header_flag u(1) if(!sps_picture_header_in_slice_header_flag) subpic_info_present_flag u(1) if(subpic_info_present_flag) { … } … }

[0226] sps_picture_header_in_slice_header_flag equal to 1 specifies that the PH syntax structure is present in the slice header and there is only one slice per picture in CLVS. sps_picture_header_in_slice_header_flag equal to 0 specifies that the PH syntax structure is not present in the slice header in CLVS.

[0227] subpic_info_present_flag equal to 1 specifies that subpic_info_present_flag is present in the CLVS and that there may be one or more subpictures for each image in the CLVS. subpic_info_present_flag equal to 0 specifies that subpic_info_present_flag is not present in the CLVS and that there is only one subpicture for each image in the CLVS. If not present, the value of subpic_info_present_flag is inferred to be equal to 0.

[0228] When res_change_in_clvs_allowed_flag is equal to 1, the value of subpic_info_present_flag shall be equal to 0.

[0229] In one embodiment, the picture_header_in_slice_header_flag is signaled in the image header.

[0230] picture_header_in_slice_header_flag equal to 1 specifies the presence of PH syntax structures in the slice header. picture_header_in_slice_header_flag equal to 0 specifies the absence of PH syntax structures in the slice header.

[0231] The value of picture_header_in_slice_header_flag shall be equal to sps_picture_header_in_slice_header_flag.

[0232] It is a bitstream conformance requirement that when picture_header_in_slice_header_flag is equal to 1 for a coded slice, there are no VCL NAL units in the CLVS with nal_unit_type equal to PH_NUT.

[0233] When picture_header_in_slice_header_flag is equal to 0, all coded slices in the current picture shall have picture_header_in_slice_header_flag equal to 0 and the current PU shall have a PH NAL unit.

[0234] In addition, a constraint flag pic_header_in_slice_header_constraint_flag may be added to general_constraint_info(). When pic_header_in_slice_header_constraint_flag is equal to 1, the value of picture_header_in_slice_header_flag in PH shall be equal to 1.

[0235] [Table 2]

[0236] pic_header_in_slice_header_constraint_flag equal to 1 specifies that the PH syntax structure is present in the slice header. pic_header_in_slice_header_constraint_flag equal to 0 imposes no such constraint. If pic_header_in_slice_header_constraint_flag is equal to 1, the value of picture_header_in_slice_header_flag shall be equal to 1.

[0237] When pic_header_in_slice_header_constraint_flag is equal to 1, rpl_info_in_ph_flag, dbf_info_in_ph_flag, sao_info_in_ph_flag, wp_info_in_ph_flag, and qp_delta_info_in_ph_flag shall be equal to 0.

[0238] When pic_header_in_slice_header_constraint_flag is equal to 1, rect_slice_flag is equal to 1.

[0239] When pic_header_in_slice_header_constraint_flag is equal to 1, subpic_info_present_flag shall be equal to 0.

[0240] When pic_header_in_slice_header_constraint_flag is equal to 1, separate_colour_plane_flag shall be equal to 0.

[0241] Below is an example of the syntax definition of the new constraint flags:

[0242] [Table 3A] [Table 3B]

[0243] Below is an example of the semantics definition of the new constraint flags.

[0244] separate_colour_plane_flag equal to 1 specifies that the three colour components of the 4:4:4 chroma format are coded separately. separate_colour_plane_flag equal to 0 specifies that the colour components are not coded separately. If separate_colour_plane_flag is not present, it is inferred to be equal to 0. When separate_colour_plane_flag is equal to 1, the coded image consists of three separate components, each of which consists of coded samples of one colour plane (Y, Cb or Cr), and uses the monochrome coding syntax. In this case, each colour plane is associated with a specific colour_plane_id value.

[0245] NOTE - There is no dependency in the decoding process between color planes with different colour_plane_id values, e.g. the decoding process of a monochrome image with one value of colour_plane_id does not use data from monochrome images with a different value of colour_plane_id for inter prediction.

[0246] Depending on the value of separate_colour_plane_flag, the value of the variable ChromaArrayType is assigned as follows:

[0247] -If separate_colour_plane_flag is equal to 0, ChromaArrayType is set equal to chroma_format_idc.

[0248] - Otherwise (separate_colour_plane_flag is equal to 1), ChromaArrayType is set equal to 0.

[0249] When pic_header_in_slice_header_constraint_flag is equal to 1, separate_colour_plane_flag shall be equal to 0.

[0250] subpic_info_present_flag equal to 1 specifies that subpic_info_present_flag is present in the CLVS and that there may be one or more subpictures for each image in the CLVS. subpic_info_present_flag equal to 0 specifies that subpic_info_present_flag is not present in the CLVS and that there is only one subpicture for each image in the CLVS.

[0251] When res_change_in_clvs_allowed_flag is equal to 1, the value of subpic_info_present_flag shall be equal to 0.

[0252] NOTE - If the bitstream is the result of a sub-bitstream extraction process and contains only a subset of the sub-pictures of the input bitstream to the sub-bitstream extraction process, it may be necessary to set the value of subpic_info_present_flag equal to 1 in the RBSP of the SPS.

[0253] When pic_header_in_slice_header_constraint_flag is equal to 1, subpic_info_present_flag shall be equal to 0.

[0254] rect_slice_flag equal to 0 specifies that the tiles in each slice are in raster scan order and slice information is not signaled in the PPS. rect_slice_flag equal to 1 specifies that the tiles in each slice cover a rectangular region of the image and slice information is signaled in the PPS. If not present, rect_slice_flag is inferred to be equal to 1. When subpic_info_present_flag is equal to 1, the value of rect_slice_flag shall be equal to 1.

[0255] When pic_header_in_slice_header_constraint_flag is equal to 1, rect_slice_flag is equal to 1.

[0256] qp_delta_info_in_ph_flag equal to 1 specifies that QP delta information is present in the PH syntax structure and is not present in slice headers that reference PPSs that do not contain a PH syntax structure. qp_delta_info_in_ph_flag equal to 0 specifies that QP delta information is not present in the PH syntax structure and may be present in slice headers that reference PPSs that do not contain a PH syntax structure.

[0257] When pic_header_in_slice_header_constraint_flag is equal to 1, rpl_info_in_ph_flag, dbf_info_in_ph_flag, sao_info_in_ph_flag, wp_info_in_ph_flag, and qp_delta_info_in_ph_flag shall be equal to 0.

[0258] one_subpic_per_pic_constraint_flag equal to 1 specifies that each image should contain only one subimage. one_subpic_per_pic_constraint_flag equal to 0 imposes no such constraint. When one_slice_per_pic_constraint_flag is equal to 1, the value of one_subpic_per_pic_constraint_flag shall be equal to 1.

[0259] pic_header_in_slice_header_constraint_flag equal to 1 specifies that the PH syntax structure is present in the slice header. pic_header_in_slice_header_constraint_flag equal to 0 imposes no such constraint. If pic_header_in_slice_header_constraint_flag is equal to 1, the value of picture_header_in_slice_header_flag shall be equal to 1.

[0260] 2. Alignment of subpic_treated_as_pic_flag values ​​between layers For subpicture extraction from a multi-layer bitstream, when subpic_treated_as_pic_flag[i] is equal to 1, for each output layer and its reference layers in the OLS, all images in the output layer and its reference layers shall have values ​​of subpic_treated_as_pic_flag[j] equal to 1 for each j value in the range from 0 to sps_num_subpics_minus1. Therefore, according to an embodiment of the present disclosure, the following constraints are added:

[0261] subpic_treated_as_pic_flag[i] equal to 1 specifies that the i-th subpicture of each coded image in the CLVS is treated as an image in the decoding process excluding in-loop filtering. subpic_treated_as_pic_flag[i] equal to 0 indicates that the i-th subpicture of each coded image in the CLVS is not treated as an image in the decoding process excluding in-loop filtering. If not present, the value of subpic_treated_as_pic_flag[i] is inferred to be equal to sps_independent_subpics_flag.

[0262] When subpic_treated_as_pic_flag[i] is equal to 1, it is a bitstream conformance requirement that all of the following conditions are true for each output layer in the OLS, including the layer that contains the i-th subimage as an output layer, and its reference layers:

[0263] -All images in the output layer and its reference layers shall have the same value of pic_width_in_luma_samples and the same value of pic_height_in_luma_samples.

[0264] - All SPSs referenced by the output layer and its reference layers shall have the same value of sps_num_subpics_minus1 for each value of j in the range 0 to sps_num_subpics_minus1 (inclusive), and shall have the same values ​​of subpic_ctu_top_left_x[j], subpic_ctu_top_left_y[j], subpic_width_minus1[j], subpic_height_minus1[j], and loop_filter_across_subpic_enabled_flag[j], respectively.

[0265] -All pictures in each access unit in the output layer and its reference layers shall have the same value of SubpicIdVal[j] for each value of j in the range from 0 to sps_num_subpics_minus1, inclusive.

[0266] - All images in the output layer and its reference layers shall have a value of subpic_treated_as_pic_flag[j] equal to 1 for each value of j in the range from 0 to sps_num_subpics_minus1, inclusive.

[0267] 3. Limitation on the number of sub-images for a small image In one embodiment, if the image size is less than or equal to CtbSizeY, an additional constraint can be imposed such that the number of sub-images is equal to 1.

[0268] The following are examples of syntax and / or semantic definitions of additional constraints:

[0269] If pic_width_max_in_luma_samples is less than or equal to CtbSizeY and pic_height_max_in_luma_samples is less than or equal to CtbSizeY, the value of sps_num_subpics_minus1 shall be equal to 0.

[0270] 4. Correction of the value range of sps_subpic_id_len_minus1 In one embodiment, the semantics of sps_subpic_id_len_minus1 may be modified as follows to allow the syntax element to have a more precise range of values:

[0271] sps_subpic_id_len_minus1+1 specifies the number of bits used to represent the syntax element sps_subpic_id[i], if present, the syntax element pps_subpic_id[i], and if present, the syntax element slice_subpic_id. The value of sps_subpic_id_len_minus1 shall be in the range 0 to 15, inclusive. The value of 1<<(sps_subpic_id_len_minus1) shall be greater than or equal to sps_num_subpics_minus1+1.

[0272] The above-described embodiments may be used separately or combined in any order. Furthermore, each of the embodiments, the encoder, and / or the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium.

[0273] FIG. 19 shows a flow chart outlining a process (1900) according to an embodiment of the present disclosure. The process (1900) can be used in the reconstruction of a block to generate a prediction block of the block being reconstructed. The term block in this disclosure can be interpreted as a prediction block, CB, CU, etc. In various embodiments, the process (1900) can be performed by processing circuits such as the processing circuits of the terminal devices (310), (320), (330), and (340), the processing circuits performing the functions of the video encoder (403), the processing circuits performing the functions of the video decoder (410), the processing circuits performing the functions of the video decoder (510), the processing circuits performing the functions of the video encoder (603), etc. In some embodiments, the process (1900) is implemented by software instructions, and thus the processing circuits perform the process (1900) when the processing circuits execute the software instructions. Processing starts at (S1901) and proceeds to (S1910).

[0274] At (S1910), a gating flag may be obtained for each of the at least one constraint information group in the constraint information signaled in the encoded video bitstream. Each gating flag may indicate whether a constraint information group of the at least one constraint information group corresponding to the respective gating flag is present in the constraint information. The constraint information may relate to a subset of coding blocks in the encoded video bitstream.

[0275] The constraint information may be present in a high level syntax (e.g., SPS, VPS, DCI, DPS) for at least one picture, and the subset of coding blocks may include coding blocks in the at least one picture.

[0276] In one example, each gate flag indicates that the constraint information group corresponding to the respective gate flag is not present in the constraint information and that no constraint flag is present in the constraint information.

[0277] In (S1920), whether a constraint information group of the gate flag is present in the constraint information can be determined based on the gate flag of the constraint information group of the at least one constraint information group. The constraint information group of the gate flag can include at least one constraint flag for a subset of the coding blocks.

[0278] At (S1930), prediction information for the subset of coding blocks may be determined based on whether a constraint information group of the gating flag is present in the constraint information.

[0279] In one example, it is determined that a constraint information group of a gating flag is present in the constraint information. Prediction information of the subset of coding blocks can be determined based on at least one constraint flag in the constraint information group of the gating flag.

[0280] In one example, it is determined that the constraint information group of the gating flag is not present in the constraint information. Prediction information for a subset of the coding blocks can be determined independently from the constraint information group of the gating flag.

[0281] At (S1940), a subset of the coding blocks may be reconstructed based on the prediction information.

[0282] The process (1900) may be adapted as appropriate. Steps of the process (1900) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used. For example, one or more of the at least one constraint information group are present in the encoded video bitstream. The constraint flags in the one or more of the at least one constraint information group may be signaled according to a predefined order. The constraint flags may be run-decoded, and prediction information for a subset of the coding blocks may then be determined based on the constraint flags.

[0283] In one example, at least one constraint flag in the constraint information group of the gating flag includes a plurality of constraint flags that are signaled according to a predetermined order, and the plurality of constraint flags in the constraint information group of the gating flag are run-decodable.

[0284] In one example, the offset and length are present in the constraint information and indicate that at least one constraint flag in the constraint information group of the gate flag is present in the constraint information.

[0285] In one example, one or more of the at least one constraint information group are present in the encoded video bitstream. An offset and a length are present in the constraint information to indicate that at least one constraint flag in each of the one or more constraint information groups of the at least one constraint information group is present in the constraint information. Some offsets may be signaled in the encoded video bitstream.

[0286] In one example, one or more of the at least one constraint information group are present in the encoded video bitstream. One or more syntax elements in the constraint information may be byte-aligned. The one or more syntax elements may include at least a gate flag for each of the constraint information groups and a plurality of constraint flags in one or more of the constraint information groups of the at least one constraint information group.

[0287] In one example, the at least one gate flag includes a frame-only constraint flag (e.g., general_frame_only_constraint_flag) that indicates whether the images in the output layer set are coded with field coding. The output layer set can include a subset of the coded blocks. The field sequence flag (e.g., sps_field_seq_flag) is false based on the frame-only constraint flag indicating that the subset of images is not coded with field coding and that the subset of images is in one layer of the output layer set.

[0288] In one example, a field sequence flag is signaled in the coded video bitstream, and the field sequence flag indicates whether the images in the output layer are coded with field coding.

[0289] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0290] Aspects of the present disclosure relate to notification field encoding information in an encoded video stream, for example having multiple layers.

[0291] When an image is coded into a bitstream containing multiple layers with different qualities, the bitstream may have syntax elements that specify which layers can be output at the decoder side. For certain video codecs that support multiple layers and scalability, one or more output layer sets can be signaled in the VPS. Syntax elements that specify the output layer sets and their dependencies, profile / tier / level, and hypothetical decoder reference model parameters should be efficiently signaled in a parameter set. When a multi-layer video sequence is coded as a field structure where two or more field images contain one frame image, the field coding information may be signaled in a high-level syntax structure such as an SPS, VPS, or SEI message.

[0292] In some video coding techniques, an access unit (AU) may refer to a coded picture, slice, tile, NAL unit, etc. that is captured and composed into a respective picture / slice / tile / NAL unit bitstream at a given instance in time. A given instance in time may be a composition time.

[0293] In some examples, such as HEVC and certain other video coding techniques, a picture order count (POC) value may be used to indicate a reference picture selected from among multiple reference pictures stored in a decoded picture buffer (DPB). When an AU includes one or more pictures, slices, or tiles, each picture, slice, or tile belonging to the same AU may have the same POC value, from which it may be derived that one or more pictures, slices, or tiles are created from content with the same composition time. Thus, two pictures / slices / tiles with the same given POC value may indicate that the two pictures / slices / tiles belong to the same AU and have the same composition time. Conversely, two pictures / tiles / slices with different POC values ​​may indicate that the two pictures / slices / tiles belong to different AUs and have different composition times.

[0294] In one embodiment, the rigid relationship described above, in which an AU can contain pictures, slices, or tiles with different POC values, can be relaxed. By allowing different POC values ​​within an AU, the POC values ​​can be used to identify potentially independently decodable pictures / slices / tiles that have the same presentation time. Thus, as described in more detail below, multiple scalable layers can be supported without modifying the reference picture selection signal (e.g., reference picture set signal or reference picture list signal).

[0295] In some instances, it may be desirable to identify the AU to which an image / slice / tile belongs from the POC value alone, relative to other images / slices / tiles with different POC values. This can be achieved as follows:

[0296] In one embodiment, the access unit count (AUC) may be signaled in a high-level syntax structure such as a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter. The value of the AUC (or AUC values) may be used to identify which NAL units, pictures, slices, or tiles belong to a given AU. The values ​​of the AUC may correspond to distinct composition time instances. The AUC value may be equal to a multiple of the POC value. The AUC value may be calculated by dividing the POC value by an integer value. In some examples, the division operation may impose a certain burden on the implementation of a decoder. Therefore, a small restriction on the numbering space of the AUC values ​​may allow the replacement of the division operation by a shift operation. For example, the AUC value may be equal to the most significant bit (MSB) value of the POC value range.

[0297] In one embodiment, the value of POC cycle per AU (e.g., poc_cycle_au) may be signaled in a high-level syntax structure such as a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter. The value of POC cycle per AU (e.g., poc_cycle_au) may indicate how many different consecutive POC values ​​can be associated with the same AU. For example, if the value of POC cycle per AU (e.g., poc_cycle_au) is equal to 4, then an image, slice, or tile with a POC value between 0 and 3 is associated with an AU with an AUC value equal to 0, and an image, slice, or tile with a POC value between 4 and 7 is associated with an AU with an AUC value equal to 1. Thus, the value of AUC can be inferred by dividing the POC value by the value of POC cycle per AU (e.g., poc_cycle_au).

[0298] In one embodiment, the value of the POC cycle per AU (e.g., poc_cycle_au) may be derived from information located, for example, in the VPS, that identifies the number of spatial or SNR layers in the encoded video sequence. Although the derivation as described above may save a few bits in the VPS and thus improve the coding efficiency, it may be advantageous to explicitly code the value of the POC cycle per AU (e.g., poc_cycle_au) in an appropriate high-level syntax structure hierarchical below the VPS in order to minimize the value of the POC cycle per AU (e.g., poc_cycle_au) for a given small portion (e.g., image) of the bitstream. Because the POC value (and / or the value of a syntax element that indirectly references the POC) may be coded in a low-level syntax structure, the optimization may save more bits than can be saved through the above derivation process.

[0299] In one embodiment, Figure 20 shows an example of a syntax table for signaling the vps_poc_cycle_au syntax element in the VPS (or SPS), which indicates the poc_cycle_au used for all images / slices in the coded video sequence, and the slice_poc_cycle_au syntax element, which indicates the poc_cycle_au of the current slice in the slice header.

[0300] If the POC value increases uniformly per AU, the syntax element vps_contant_poc_cycle_per_au in the VPS is set equal to 1, the syntax element vps_poc_cycle_au is signaled in the VPS, and the syntax element slice_poc_cycle_au is not explicitly signaled, and the value of the AUC per AU can be calculated by dividing the value of POC by the syntax element vps_poc_cycle_au.

[0301] If the POC value does not increase uniformly per AU, the syntax element vps_contant_poc_cycle_per_au in the VPS is set equal to 0. Therefore, the syntax element vps_access_unit_cnt is not signaled and the syntax element slice_access_unit_cnt is signaled in the slice header for each slice or picture. Each slice or picture may have a different value of slice_access_unit_cnt. The value of AUC per AU can be calculated by dividing the value of POC by the syntax element slice_poc_cycle_au.

[0302] Figure 21 shows a block diagram illustrating the related workflow described above. The workflow starts at (S2101) and proceeds to (S2110).

[0303] In (S2110), the VPS or SPS can be analyzed to identify whether the POC cycles per AU are constant.

[0304] At (S2120), it can be determined whether the POC cycle for each AU is constant within the encoded video sequence. If it is determined that the POC period for each AU is constant within the encoded video sequence, the workflow proceeds to (S2130). If not, the workflow proceeds to (S2140).

[0305] At (S2130), an access unit count value may be calculated from the sequence level poc_cycle_au value and the POC value, and the workflow proceeds to (S2150).

[0306] At (S2140), the value of the access unit count can be calculated from the picture level poc_cycle_au value and the POC value, and the workflow then proceeds to (S2150).

[0307] In (S2150), the VPS or SPS can be analyzed to identify whether the POC cycles per AU are constant.

[0308] In one embodiment, images, slices, or tiles corresponding to AUs with the same AUC value may be associated with the same decoding or output time instance, even if the images, slices, or tiles have different POC values. Thus, all or a subset of images, slices, or tiles associated with the same AU may be decoded in parallel and output simultaneously, without dependency between parsing / decoding across images, slices, or tiles within the same AU.

[0309] In one embodiment, images, slices, or tiles that correspond to AUs with the same AUC value may be associated with the same composition / display time instance, even if the images, slices, or tiles have different POC values. If the composition time is included in the container format, images may be displayed at the same time instance if they have the same composition time, even if they correspond to different AUs.

[0310] In one embodiment, each image, slice, or tile may have the same temporal identifier (e.g., temporal_id) within the same AU. All or a subset of images, slices, or tiles corresponding to a time instance may be associated with the same temporal sublayer. In one embodiment, each image, slice, or tile may have the same or different spatial layer identification (e.g., layer_id) within the same AU. All or a subset of images, slices, or tiles corresponding to a time instance may be associated with the same or different spatial layers.

[0311] FIG. 22 shows an example of a video sequence structure with a combination of temporal_id, layer_id, POC, and AUC values ​​with adaptive resolution change. In the example shown in FIG. 22, an image, slice, or tile in a first AU with AUC=0 may have temporal_id=0 and layer_id=0 or 1, and an image, slice, or tile in a second AU with AUC=1 may have temporal_id=1 and layer_id=0 or 1, respectively. Regardless of the values ​​of temporal_id and layer_id, the value of POC increases by 1 for each image. Referring to FIG. 22, the value of poc_cycle_au may be equal to 2. In one example, the value of poc_cycle_au is preferably set equal to the number of (spatial scalability) layers. In this example, the value of POC increases by 2 and the value of AUC increases by 1.

[0312] In the above embodiment, all or a subset of the inter-picture or inter-layer prediction structure and reference picture indication can be supported by using a reference picture set (RPS) signaling in a video standard such as HEVC or a reference picture list (RPL) signaling. In the RPS or RPL, the selected reference picture can be indicated by signaling a value of POC or a delta value of POC between the current picture and the selected reference picture. In various examples, the RPS and RPL can be used to indicate the inter-picture or inter-layer prediction structure without modifying the signaling and with the following restrictions: If the value of the temporal_id of the reference picture is greater than the value of the temporal_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction. If the value of the layer_id of the reference picture is greater than the value of the layer_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction.

[0313] In one embodiment, the scaling of motion vectors based on POC difference for temporal motion vector prediction can be disabled across multiple images in an AU. Thus, although each image may have different POC values ​​in an AU, reference images with different POCs in the same AU are considered as reference images with the same time instance, so that motion vectors are not scaled and are not used for temporal motion vector prediction in the AU. Thus, the motion vector scaling function can return 1 when the reference image belongs to the AU associated with the current image.

[0314] In one embodiment, if the spatial resolution of the reference image is different from that of the current image, the scaling of the motion vector based on the POC difference for temporal motion vector prediction can be optionally disabled across multiple images. If the scaling of the motion vector is allowed, the motion vector is scaled based on both the POC difference and the spatial resolution ratio between the current image and the reference image.

[0315] In one embodiment, motion vectors may be scaled based on AUC difference instead of POC difference for temporal motion vector prediction, especially when poc_cycle_au has a non-uniform value (e.g., when vps_contant_poc_cycle_per_au==0), otherwise (e.g., when vps_contant_poc_cycle_per_au==1), the scaling of motion vectors based on AUC difference may be identical to the scaling of motion vectors based on POC difference.

[0316] In one embodiment, when a motion vector is scaled based on the AUC difference, a reference motion vector within the same AU (having the same AUC value) as the current image is not scaled based on the AUC difference and is used for motion vector prediction without scaling or with scaling based on the spatial resolution ratio between the current image and the reference image.

[0317] In one embodiment, the AUC value is used to identify AU boundaries and is used for hypothetical reference decoder (HRD) operations that may require input and output timing with AU granularity. In various examples, a decoded image with the top layer in the AU may be output for display. The AUC value and layer_id value can be used to identify the output image.

[0318] In one embodiment, an image may include one or more sub-images. Each sub-image may cover a local area or the entire area of ​​the image. The area supported by a sub-image may or may not overlap with the area supported by another sub-image. The area comprised by one or more sub-images may or may not cover the entire area of ​​the image. If an image includes sub-images, the area supported by the sub-image may be the same as the area supported by the image.

[0319] In one embodiment, the sub-images may be coded by a coding method similar to that used for the coded image. The sub-images may be coded independently or may be coded dependent on another sub-image or coded image. The sub-images may or may not have analytical dependencies from another sub-image or coded image.

[0320] In one embodiment, the coded sub-images may be included in one or more layers. The coded sub-images within a layer may have different spatial resolutions. The original sub-images may be spatially resampled (e.g., upsampled or downsampled), coded with different spatial resolution parameters, and included in the bitstream corresponding to the layer.

[0321] In one embodiment, a sub-image having (W,H) may be coded and included in the coded bitstream corresponding to layer 0. W denotes the width of the sub-image and H denotes the height of the sub-image. An upsampled (or downsampled) sub-image may be upsampled (or downsampled) from a sub-image having an original spatial resolution. An upsampled (or downsampled) sub-image has a spatial resolution (W*S w,k ,H*S h,k ), which may be encoded and included in the encoded bitstream corresponding to layer k. w、k and S h、k can denote the horizontal and vertical resampling ratios, respectively. w、k , S h、k If the value of is greater than 1, the resampling is an upsampling. w、k , S h、k If the value of is less than 1, the resampling is a downsampling.

[0322] In one embodiment, a coded sub-image in a layer may have a different visual quality than a coded sub-image in another layer in the same sub-image or in a different sub-image. For example, sub-image i in layer n may have a quantization parameter (e.g., Q i,n ), and subimage j of layer m is coded with a quantization parameter (e.g., Q J,m )

[0323] In one embodiment, the coded sub-images within a layer may be independently decodable without parsing or decoding dependencies from coded sub-images in another layer of the same local region. A sub-image layer that may be independently decodable without reference to another sub-image layer of the same local region is an independent sub-image layer. Coded sub-images within an independent sub-image layer may or may not have decoding or parsing dependencies from previously coded sub-images in the same sub-image layer. Coded sub-images may have no dependencies from coded images in another sub-image layer.

[0324] In one embodiment, an encoded subimage in a layer may be dependently decodable with any parsing or decoding dependency from an encoded subimage in another layer of the same local region. A subimage layer that may be dependently decodable with reference to another subimage layer of the same local region is a dependent subimage layer. An encoded subimage in a dependent subimage may reference an encoded subimage belonging to the same subimage, a previously encoded subimage in the same subimage layer, or both reference subimages.

[0325] In one embodiment, a coded sub-picture includes one or more independent sub-picture layers and one or more dependent sub-picture layers. However, there may be at least one independent sub-picture layer for a coded sub-picture. An independent sub-picture layer may have a value of a layer identifier (layer_id) that may be present in a NAL unit header or another high-level syntax structure equal to 0. A sub-picture layer with layer_id equal to 0 is a base sub-picture layer.

[0326] In one embodiment, an image may include one or more foreground sub-images and one background sub-image. The area supported by the background sub-image may be equal to the area of ​​the image. The area supported by the foreground sub-image may overlap with the area supported by the background sub-image. The background sub-image may be a base sub-image layer, and the foreground sub-image may be a non-base sub-image layer (e.g., an extension sub-image layer). One or more non-base sub-image layers may reference the same base layer for decoding. Each non-base sub-image layer with layer_id equal to a may reference a non-base sub-image layer with layer_id equal to b, where a is greater than b.

[0327] In one embodiment, an image may contain one or more foreground sub-images, with or without a background sub-image. Each sub-image may have its own base sub-image layer and one or more non-base (enhanced) layers. Each base sub-image layer may be referenced by one or more non-base sub-image layers. Each non-base sub-image layer with layer_id equal to a may reference a non-base sub-image layer with layer_id equal to b, where a is greater than b.

[0328] In one embodiment, an image may contain one or more foreground subimages, with or without a background subimage. Each coded subimage in a (base or non-base) subimage layer may be referenced by one or more non-base layer subimages that belong to the same subimage and one or more non-base layer subimages that do not belong to the same subimage.

[0329] In one embodiment, an image may contain one or more foreground sub-images, with or without background sub-images. A sub-image in layer a may be further divided into multiple sub-images in the same layer. One or more coded sub-images in layer b may reference a divided sub-image in layer a.

[0330] In one embodiment, a coded video sequence (CVS) can be a group of coded images. A CVS can contain one or more coded sub-image sequences (CSPS), where a CSPS can be a group of coded sub-images covering the same local region of an image. A CSPS can have the same or different temporal resolution as the coded video sequence.

[0331] In one embodiment, the CSPS may be encoded and included in one or more layers. A CSPS may include one or more CSPS layers. Decoding one or more CSPS layers corresponding to a CSPS may reconstruct a sequence of sub-images corresponding to the same local region.

[0332] In one embodiment, some CSPS layers corresponding to a CSPS may be the same as or different from some CSPS layers corresponding to another CSPS.

[0333] In one embodiment, a CSPS layer may have a different temporal resolution (e.g., frame rate) than another CSPS layer, and the original (e.g., uncompressed) sub-image sequence may be temporally resampled (e.g., upsampled or downsampled), coded with different temporal resolution parameters, and included in the bitstream corresponding to the layer.

[0334] In one embodiment, a sub-image sequence having a frame rate F may be encoded and included in the encoded bitstream corresponding to layer 0, where F*S t,k A temporally upsampled (or downsampled) sub-image sequence from the original sub-image sequence, having a sub-image sequence S t,k denotes the temporal sampling ratio for layer k. t,k If the value of S is greater than 1, the temporal resampling process corresponds to a frame rate up-conversion. t,k If the value of is less than 1, the temporal resampling process corresponds to a frame rate down-conversion.

[0335] In one embodiment, when a sub-picture with CSPS layer a is referenced by a sub-picture with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b, the decoded pixels in CSPS layer a are resampled and used for reference. The resampling process may require upsampling filtering or downsampling filtering.

[0336] FIG. 23 shows an example of a video stream including a background video CSPS having a layer_id equal to 0 and a plurality of foreground CSPS layers. The encoded sub-image can include one or more CSPS layers, but the background area that does not belong to any foreground CSPS layer can include a base layer. The base layer can include a background area and a foreground area. The enhancement CSPS layer can include a foreground area. The enhancement CSPS layer can have better visual quality than the base layer in the same area. The enhancement CSPS layer can refer to the reconstructed pixels and the motion vectors of the base layer corresponding to the same area.

[0337] In one embodiment, the video bitstream corresponding to the base layer is included in a track, and the CSPS layer corresponding to each sub-image is included in a separate track within the video file.

[0338] In one embodiment, the video bitstream corresponding to the base layer is included in a track, and the CSPS layers having the same layer_id are included in separate tracks. In one example, the track corresponding to layer k includes only the CSPS layer corresponding to layer k.

[0339] In one embodiment, each CSPS layer of each sub-image is stored in a separate track. Each track may or may not have any syntax analysis or decoding dependency from one or more other tracks.

[0340] In one embodiment, each track can include a bitstream corresponding to the CSPS layers from layer i to layer j of all or a subset of the sub-images, where 0 < i ≤ j ≤ k, and k is the top layer of the CSPS.

[0341] In one embodiment, the image includes one or more associated media data such as a depth map, an alpha map, 30 geometry data, an occupancy map, etc. Such associated time-limited media data can each be divided into one or more data sub-streams corresponding to one sub-image.

[0342] In one embodiment, Figure 24 shows an example of a video conference based on a multi-layer sub-image method. The video stream includes one base layer video bitstream corresponding to a background image and one or more enhancement layer video bitstreams corresponding to a foreground sub-image. Each enhancement layer video bitstream corresponds to a CSPS layer. On the display, the image corresponding to the base layer is displayed by default. t includes the image of one or more users in the picture (PIP). When a particular user is selected by the client control, the enhanced CSPS layer corresponding to the selected user is decoded and displayed with enhanced quality or spatial resolution.

[0343] 25 shows a diagram for the above-mentioned operation. The operation starts from (S2501) and proceeds to (S2510).

[0344] At (S2510), a video bitstream having multiple layers can be decoded.

[0345] At (S2520), a background region and one or more foreground sub-images may be identified.

[0346] At (S2530), it may be determined whether a particular sub-image region has been selected. If it is determined that a particular sub-image region has been selected, processing continues at (S2540). Otherwise, processing continues at (S2550).

[0347] In (S2540), the extended sub-image can be decoded and displayed. Then, the process proceeds to (S2599) where the operation ends.

[0348] In (S2550), the background region can be decoded and displayed. Then, the process proceeds to (S2599) and ends the operation.

[0349] In one embodiment, a network middlebox (such as a router) can select a subset of layers to send to a user depending on its bandwidth. The image / sub-image organization can be used for bandwidth adaptation. For example, if a user does not have the bandwidth, the router can delete or select some sub-images due to their importance or based on the settings used, and the operation can be done dynamically to adapt to the bandwidth.

[0350] FIG. 26 shows a use case of 360 video. When a spherical 360 image is projected onto a planar image, the projected 360 image can be divided into multiple sub-images as a base layer. The enhancement layer of a particular sub-image can be encoded and sent to the client. The decoder can decode both the base layer containing all the sub-images and the enhancement layer of the selected sub-image. If the current viewport is the same as the selected sub-image, the displayed image can have higher quality with the decoded sub-image with the enhancement layer. Otherwise, the decoded image with the base layer can be displayed with lower quality.

[0351] In one embodiment, any suitable layout information for display can be present in the file as auxiliary information (such as SEI messages or metadata). One or more decoded sub-images can be rearranged and displayed according to the signaled layout information. The layout information may be signaled by a streaming server or broadcaster, regenerated by a network entity or cloud server, or determined by a user's customized settings.

[0352] In one embodiment, if an input image is divided into one or more sub-regions (e.g., rectangular sub-regions), each sub-region may be coded as an independent layer. Each independent layer corresponding to a local region may have a unique layer_id value. For each independent layer, sub-image size and position information may be signaled. For example, image size (e.g., width and / or height of a picture), top left corner offset information (e.g., x_offset, y_offset). Figure 27 shows an example of a layout of divided sub-images, sub-image size, position information, and corresponding image prediction structure. The layout information including sub-image size(s) and sub-image position(s) may be signaled in a high-level syntax structure such as parameter set(s), slice or tile group header, or SEI message.

[0353] In one embodiment, each sub-image corresponding to an independent layer may have a unique POC value within an AU. When a reference image among the images stored in the DPB is indicated using a syntax element of the RPS or RPL structure, the POC value of each sub-image corresponding to a layer may be used.

[0354] In one embodiment, the layer_id may not be used and a POC value (eg, a POC delta value) may be used to indicate the prediction structure (eg, an inter-layer prediction structure).

[0355] In one embodiment, a sub-image having a POC value equal to N corresponding to a layer (or local region) may or may not be used as a reference image for a sub-image having a POC value equal to N+K corresponding to the same layer (or same local region) for motion compensation prediction. In various examples, the value of the number K may be equal to the maximum number of layers (e.g., independent layers), which may be the same for several sub-regions.

[0356] In one embodiment, Figure 28 shows an extended case of Figure 27. When the input image is divided into multiple (e.g., 4) sub-regions, each local region can be coded with one or more layers. With reference to Figure 28, the number of independent layers may be equal to the number of sub-regions, and one or more layers may correspond to a sub-region. Thus, each sub-region can be coded with one or more independent layers and zero or more dependent layers.

[0357] In one embodiment, in Figure 28, the input image may be divided into four sub-regions. The top right sub-region may be coded as two layers (e.g., layer 1 and layer 4), and the bottom right sub-region may be coded as two layers (e.g., layer 3 and layer 5). In Figure 28, layer 4 may refer to layer 1 for motion compensation prediction, and layer 5 may refer to layer 3 for motion compensation.

[0358] In one embodiment, in-loop filtering (e.g., deblocking filters, adaptive in-loop filtering, reshapers, bilateral filters, or any deep learning based filtering) across layer boundaries can be (optionally) disabled.

[0359] In one embodiment, motion compensated prediction or intrablock copying across layer boundaries may (optionally) be disabled.

[0360] In one embodiment, border padding for motion compensated prediction or in-loop filtering at sub-image boundaries may be optionally processed. A flag indicating whether border padding is processed may be signaled in a high-level syntax structure such as a parameter set (VPS, SPS, PPS, or APS), a slice or tile group header, or an SEI message.

[0361] In one embodiment, layout information of sub-regions (or sub-pictures) may be signaled in a VPS or an SPS. FIG. 29A-FIG. 29B show examples of syntax elements in a VPS and an SPS. Referring to FIG. 29A, a flag (e.g., vps_sub_picture_dividing_flag) is signaled in a VPS. The flag (e.g., vps_sub_picture_dividing_flag) may indicate whether an input image is divided into multiple sub-regions. When the value of the flag (e.g., vps_sub_picture_dividing_flag) is equal to 0, an input image in a coded video sequence corresponding to a current VPS may not be divided into multiple sub-regions. Thus, the input image size may be equal to the coded image size (e.g., pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in an SPS. When the value of the flag (e.g., vps_sub_picture_dividing_flag) is equal to 1, an input image may be divided into multiple sub-regions. Therefore, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled in the VPS. The values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may be equal to the width and height of the input image, respectively.

[0362] In one embodiment, the values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may not be used for decoding, but may be used for compositing and display.

[0363] In one embodiment, when the value of a flag (e.g., vps_sub_picture_dividing_flag) is equal to 1, the syntax elements pic_offset_x and pic_offset_y may be signaled in (a) an SPS corresponding to a particular layer. Thus, the coded picture size (e.g., pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to a particular layer. The location of the upper left corner of the sub-region (e.g., syntax elements pic_offset_x and pic_offset_y) may be signaled in the SPS.

[0364] In one embodiment, the position information of the top left corner of the sub-region (eg, syntax elements pic_offset_x, pic_offset_y) may not be used for decoding, but may be used for synthesis and display.

[0365] In one embodiment, layout information (e.g., size and position) and inter-layer dependency information of all or a subset sub-region of an input image may be signaled in a parameter set or SEI message. Figure 30 shows an example of syntax elements indicating information of sub-region layout, inter-layer dependency, and relationship between sub-region and one or more layers. With reference to Figure 30, the syntax element num_sub_region may indicate the number of (rectangular) sub-regions in the current coded video sequence. The syntax element num_layers may indicate the number of layers in the current coded video sequence. The value of num_layers may be equal to or greater than the value of num_sub_region. If any sub-region is coded as a single layer, the value of num_layers may be equal to the value of num_sub_region. When one or more sub-regions are coded as multiple layers, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] may indicate a dependency from the jth layer to the ith layer. The syntax element num_layers_for_region[i] may indicate the number of layers associated with the i-th subregion. The syntax element sub_region_layer_id[i][j] may indicate the layer_id of the j-th layer associated with the i-th subregion. sub_region_offset_x[i] and sub_region_offset_y[i] may indicate the horizontal and vertical positions, respectively, of the top-left corner of the i-th subregion. sub_region_width[i] and sub_region_height[i] may indicate the width and height, respectively, of the i-th subregion.

[0366] In one embodiment, one or more syntax elements specifying an output tier set configured to indicate one of multiple tiers to be output with or without PTL information may be signaled in a high level syntax structure, e.g., a VPS, DPS, SPS, PPS, APS or SEI message.

[0367] 31, a syntax element num_output_layer_sets indicating the number of output layer sets (OLS) in a coded video sequence that references the VPS may be signaled in the VPS. For each output layer set, a flag (e.g., output_layer_flag) may be signaled for each output layer.

[0368] In one embodiment, output_layer_flag[i] equal to 1 may specify that the i-th layer is an output. vps_output_layer_flag[i] equal to 0 may specify that the i-th layer is not an output.

[0369] In one embodiment, one or more syntax elements specifying the PTL information of each output tier set may be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS or SEI message. With reference to Figure 31, a syntax element num_profile_tile_level indicating the number of profile tier level information per OLS in the encoded video sequence that references the VPS may be signaled in the VPS. For each output tier set, a set of syntax elements of PTL information or an index indicating a particular PTL information among the entries in the PTL information may be signaled for each output tier.

[0370] In one embodiment, profile_tier_level_idx[i][j] specifies the index of the profile_tier_level() syntax structure that applies to the jth tier of the ith OLS in the list of profile_tier_level() syntax structures in the VPS.

[0371] In one embodiment, referring to FIG. 32, if the number of maximum layers is greater than 1 (vps_max_layers_minus1>0), the syntax elements num_profile_tile_level and / or num_output_layer_sets may be signaled.

[0372] In one embodiment, referring to FIG. 32, a syntax element vps_output_layers_mode[i] may be present in the VPS to indicate the mode of output layer notification for the i-th output layer set.

[0373] In one embodiment, vps_output_layers_mode[i] equal to 0 specifies that only the top layer is output with the i output layer set. vps_output_layer_mode[i] equal to 1 specifies that all layers are output with the i output layer set. vps_output_layer_mode[i] equal to 2 specifies that the layer that is output is the layer that has vps_output_layer_flag[i][j] equal to 1 set for the i output layer. Other values ​​may be reserved.

[0374] In one embodiment, output_layer_flag[i][j] may or may not be signaled depending on the value of vps_output_layers_mode[i] for the i-th output layer set.

[0375] In one embodiment, referring to Figure 32, there may be a flag vps_ptl_signal_flag [i] for the i-th output layer set. Depending on the value of vps_ptl_signal_flag [i], the PTL information of the i-th output layer set may or may not be notified.

[0376] In one embodiment, referring to FIG. 33, a number of sub-images (eg, max_subpics_minus1) in the current CVS may be signaled in a high-level syntax structure, such as, for example, a VPS, DPS, SPS, PPS, APS or SEI message.

[0377] In one embodiment, referring to FIG. 33, if the number of sub-images is greater than one (max_subpics_minus1>0), the sub-image identifier (eg, sub_pic_id[i]) of the i-th sub-image may be signaled.

[0378] In one embodiment, one or more syntax elements indicating sub-picture identifiers belonging to each layer of each output layer set may be signaled in the VPS. Referring to Figure 34, sub_pic_id_layer[i][j][k] indicates the kth sub-picture present in the jth layer of the ith output layer set. With the above information, the decoder can know which sub-pictures may be decoded and output for each layer of a particular output layer set.

[0379] In one embodiment, a picture header (PH) is a syntax structure that includes syntax elements that apply to all slices of a coded picture. A slice header can be part of a coded slice that includes data elements related to all tiles or CTU rows within the tiles represented in the slice. A picture unit (PU) is a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and include, for example, exactly one coded picture. A PU may include a picture header (PH) and one or more VCL NAL units that make up the coded picture.

[0380] In one embodiment, the SPS (RBSP) is available to the decoding process before the SPS is referenced and may be included in at least one AU with TemporalId equal to 0 or provided via external means.

[0381] In one embodiment, the SPS (RBSP) may be included in at least one AU with TemporalId equal to 0 in the CVS, which may include one or more PPSs that reference the SPS, or may be provided via external means, and may be available to the decoding process before the SPS is referenced.

[0382] In one embodiment, the SPS (RBSP) may be available to the decoding process before the SPS is referenced by one or more PPSs contained in at least one PU having a nuh_layer_id equal to the lowest nuh_layer_id value of a PPS NAL unit that references the SPS NAL unit in a CVS, including one or more PPSs that reference the SPS, or provided via external means.

[0383] In one embodiment, the SPS (RBSP) may be available to the decoding process before the SPS is referenced by one or more PPSs that are included in at least one PU with TemporalId equal to 0 and nuh_layer_id equal to the lowest nuh_layer_id value of a PPS NAL unit that references the SPS NAL unit, or that are provided via external means.

[0384] In one embodiment, an SPS (RBSP) may be available to the decoding process before the SPS is referenced by one or more PPSs, including one or more PPSs that reference the SPS, or provided via external means, and is contained in at least one PU whose PPS NAL unit referencing the SPS NAL unit in the CVS has TemporalId equal to 0 and whose nuh_layer_id is equal to the lowest nuh_layer_id value.

[0385] In one embodiment, the pps_seq_parameter_set_id specifies the value of sps_seq_parameter_set_id of the referenced SPS. In one example, the value of pps_seq_parameter_set_id may be the same in all PPSs referenced by a coded picture in the CLVS.

[0386] In one embodiment, all SPS NAL units with a particular value of sps_seq_parameter_set_id in the CVS may have the same content.

[0387] In one embodiment, regardless of the nuh_layer_id value, SPS NAL units may share the same value space of sps_seq_parameter_set_id.

[0388] In one embodiment, the nuh_layer_id value of an SPS NAL unit may be equal to the lowest nuh_layer_id value of a PPS NAL unit that references the SPS NAL unit.

[0389] In one embodiment, when an SPS with nuh_layer_id equal to m is referenced by one or more PPSs with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the (direct or indirect) reference layer of the layer with nuh_layer_id equal to n or the layer with nuh_layer_id equal to m.

[0390] In one embodiment, the PPS (RBSP) may be included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit or provided via external means and may be available to the decoding process before the PPS is referenced.

[0391] In one embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced, contained in at least one AU having a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS, which contains one or more PHs (or coded slice NAL units) that reference the PPS, or is provided via external means.

[0392] In one embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced by one or more PHs (or coded slice NAL units) contained in at least one PU having a nuh_layer_id equal to the lowest nuh_layer_id value of a coded slice NAL unit that references the PPS NAL unit in the CVS, or that is provided via external means.

[0393] In one embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced by one or more PHs (or coded slice NAL units) contained in at least one PU that contains one or more PHs (or coded slice NAL units) that reference the PPS or is provided via external means and has a TemporalId equal to the TemporalId of the PPS NAL unit and a nuh_layer_id equal to the lowest nuh_layer_id value of a coded slice NAL unit that references the PPS NAL unit in the CVS.

[0394] In one embodiment, the ph_pic_parameter_set_id in the PH specifies the value of pps_pic_parameter_set_id of the reference PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0395] In one embodiment, all PPS NAL units with a particular value of pps_pic_parameter_set_id within a PU may have the same content.

[0396] In one embodiment, regardless of the nuh_layer_id value, PPS NAL units may share the same value space of pps_pic_parameter_set_id.

[0397] In one embodiment, the nuh_layer_id value of a PPS NAL unit may be equal to the lowest nuh_layer_id value of a coded slice NAL unit that references the NAL unit.

[0398] In one embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the (direct or indirect) reference layer of the layer with nuh_layer_id equal to n or the layer with nuh_layer_id equal to m.

[0399] In one embodiment, the PPS (RBSP) may be included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit or provided via external means and may be available to the decoding process before the PPS is referenced.

[0400] In one embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced, contained in at least one AU having a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS, which contains one or more PHs (or coded slice NAL units) that reference the PPS, or is provided via external means.

[0401] In one embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced by one or more PHs (or coded slice NAL units) contained in at least one PU having a nuh_layer_id equal to the lowest nuh_layer_id value of a coded slice NAL unit that references the PPS NAL unit in the CVS, or that is provided via external means.

[0402] In one embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced by one or more PHs (or coded slice NAL units) contained in at least one PU that contains one or more PHs (or coded slice NAL units) that reference the PPS or is provided via external means and has a TemporalId equal to the TemporalId of the PPS NAL unit and a nuh_layer_id equal to the lowest nuh_layer_id value of a coded slice NAL unit that references the PPS NAL unit in the CVS.

[0403] In one embodiment, the ph_pic_parameter_set_id in the PH specifies the value of pps_pic_parameter_set_id of the reference PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0404] In one embodiment, all PPS NAL units with a particular value of pps_pic_parameter_set_id within a PU may have the same content.

[0405] In one embodiment, regardless of the nuh_layer_id value, PPS NAL units may share the same value space of pps_pic_parameter_set_id.

[0406] In one embodiment, the nuh_layer_id value of a PPS NAL unit may be equal to the lowest nuh_layer_id value of a coded slice NAL unit that references a NAL unit that references the PPS NAL unit.

[0407] In one embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the (direct or indirect) reference layer of the layer with nuh_layer_id equal to n or the layer with nuh_layer_id equal to m.

[0408] Output layer indicates a layer in the output layer set that is output. Output layer set (OLS) indicates a set of layers that includes a specified set of layers, where one or more layers in the set of layers are designated to be output layers. OLS layer index is an index of a layer in OLS. OLS layer index can point to a list of layers in OLS.

[0409] A sub-layer represents a temporal scalable layer of a temporal scalable bitstream that contains VCL NAL units and associated non-VCL NAL units with a particular value of the TemporalId variable. A sub-layer representation represents a subset of a bitstream that contains the NAL units of a particular sub-layer and the sub-layers below.

[0410] The VPS RBSP may be available to the decoding process before the VPS RBSP is referenced, either contained in at least one AU with TemporalId equal to 0 or provided via external means. All VPS NAL units with a particular value of vps_video_parameter_set_id in the CVS may have the same content.

[0411] The vps_video_parameter_set_id provides an identifier for the VPS for reference by other syntax elements. The value of vps_video_parameter_set_id may be greater than 0.

[0412] vps_max_layers_minus+1 specifies the maximum number of layers allowed within each CVS that references a VPS.

[0413] vps_max_sublayers_minus+1 specifies the maximum number of temporary sublayers that may exist in a layer in each CVS that references a VPS. The value of vps_max_sublayers_minus1 may range from 0 to 6, inclusive.

[0414] In one example, a flag (e.g., vps_all_layers_same_num_sublayers_flag) equal to 1 specifies that the number of temporary sublayers is the same for all layers in each CVS that references the VPS. A flag (e.g., vps_all_layers_same_num_sublayers_flag) equal to 0 specifies that layers in each CVS that references the VPS may or may not have the same number of temporary sublayers. If not present, the value of the flag (e.g., vps_all_layers_same_num_sublayers_flag) is inferred to be equal to 1.

[0415] A flag (e.g., vps_all_independent_layers_flag) equal to 1 specifies that all layers in the CVS are coded independently, without using inter-layer prediction. A flag (e.g., vps_all_independent_layers_flag) equal to 0 specifies that one or more layers in the CVS may use inter-layer prediction. If not present, the value of the flag (e.g., vps_all_independent_layers_flag) is inferred to be equal to 1.

[0416] vps_layer_id[i] may specify the value of nuh_layer_id for the i-th tier. For any two non-negative integer values ​​of m and n, the value of vps_layer_id[m] may be less than vps_layer_id[n] when m is less than n.

[0417] vps_independent_layer_flag[i] equal to 1 may specify that the layer with index i does not use inter-layer prediction. vps_independent_layer_flag[i] equal to 0 may specify that the layer with index i may use inter-layer prediction, and a syntax element vps_direct_ref_layer_flag[i][j] for j in the range 0 to i-1, inclusive, is present in the VPS. If not present, the value of vps_independent_layer_flag[i] is inferred to be equal to 1.

[0418] vps_direct_ref_layer_flag[i][j] equal to 0 may specify that the layer with index j is not a direct reference layer for the layer with index i. vps_direct_ref_layer_flag[i][j] equal to 1 may specify that the layer with index j is a direct reference layer for the layer with index i. For i and j in the range 0 to vps_max_layers_minus1, inclusive, vps_direct_ref_layer_flag[i][j] is inferred to be equal to 0 when not present. When vps_independent_layer_flag[i] is equal to 0, there may be at least one value of j in the range 0 to i-1, inclusive, such that the value of vps_direct_ref_layer_flag[i][j] is equal to 1.

[0419] The variables NumDirectRefLayers[i], DirectRefLayerldx[i][d], NumRefLayers[i], RefLayerldx[i][r], and LayerUsedAsRefLayerFlag[j] can be derived as follows: for(i=0;i <=vps_max_layers_minus1;i++) { for(j=0;j <=vps_max_layers_minus1;j++){ dependencyFlag[ i ][ j ]=vps_direct_ref_layer_flag[ i ][ j ] for(k=0; k < i; k++) if(vps_direct_ref_layer_flag[ i ][ k ]&&dependencyFlag[ k ][ j ]) dependencyFlag[ i ][ j ]=1 } LayerUsedAsRefLayerFlag[ i ]=0 } for(i=0;i <=vps_max_layers_minus1;i++){ for(j=0,d=0,r=0;j<=vps_max_layers_minus1;j++) {if(vps_direct_ref_layer_flag [i] [j]){ DirectRefLayerldx[ i ][ d++]=j LayerUsedAsRefLayerFlag[ j ]=1 } if(dependencyFlag[ i ][ j ]) RefLayerldx[i][r++]=j } NumDirectRefLayers[ i ]=d NumRefLayers[ i ]=r }

[0420] The variable GeneralLayerldx[i], which specifies the layer index of the layer with nuh_layer_id equal to vps_layer_id[i], can be derived as follows: for(i=0;i <=vps_max_layers_minus1;i++) GeneralLayerldx[ vps_layer_id[ i ] ]=i

[0421] In one example, for any two distinct values ​​of i and j, if the flag (e.g., dependencyFlag[i][j]) is equal to 1, then both are in the range from 0 to vps_max_layers_minus1. A bitstream compatibility requirement may be that the values ​​of chroma_format_idc and bit_depth_minus 8 applied to the i-th layer may be equal to the values ​​of chroma_format_idc and bit_depth_minus 8 applied to the j-th layer, respectively.

[0422] A max_tid_ref_present_flag[i] equal to 1 may specify that the syntax element max_tid_il_ref_pics_plus1[i] is present. A max_tid_ref_present_flag[i] equal to 0 may specify that the syntax element max_tid_il_ref_pics_plus1[i] is not present.

[0423] max_tid_il_ref_pics_plusl[i] equal to 0 may specify that no inter-layer prediction is used for non-IRAP pictures of the i-th layer. max_tid_il_ref_pics_plus1[i] greater than 0 may specify that pictures with TemporalId greater than max_tid_il_ref_pics_plus1[i]-1 are not used as ILRP for decoding pictures of the i-th layer. If not present, the value of max_tid_il_ref_pics_plus1[i] is inferred to be equal to 7.

[0424] each_layer_is_an_ols_flag equal to 1 may specify that each OLS contains only one layer, each layer in the CVS that references the VPS is itself an OLS, and the single contained layer is the only output layer. each_layer_is_an_ols_flag equal to 0 may specify that an OLS may contain two or more layers. If vps_max_layers_minus1 is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be equal to 1. Otherwise, when vps_all_independent_layers_flag is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be equal to 0.

[0425] ols_mode_idc equal to 0 specifies that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, where the i-th OLS includes layers with layer indices 0 to i, inclusive, and for each OLS, only the top layer of the OLS is output.

[0426] ols_mode_idc equal to 1 specifies that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, where the i-th OLS includes layers with layer indices 0 to i, inclusive. For each OLS, all layers within the OLS are output.

[0427] ols_mode_idc equal to 2 allows specifying that the total number of OLSs specified by the VPS is signaled explicitly, and for each OLS, the output layer is signaled explicitly, and that other layers are layers that are direct or indirect reference layers to the output layer of the OLS.

[0428] The value of ols_mode_idc may range from 0 to 2, inclusive. The value 3 of ols_mode_idc may be reserved for a particular video coding technique or standard, for example by ITU-T or ISO / IEC.

[0429] If a flag (e.g., vps_all_independent_layers_flag) is equal to 1 and a flag (e.g., each_layer_is_an_ols_flag) is equal to 0, the value of ols_mode_idc is inferred to be equal to 2.

[0430] num_output_layer_sets_minusl+1 allows specifying the total number of OLSs specified by the VPS when ols_mode_idc is equal to 2.

[0431] The variable TotalNumOlss, which specifies the total number of OLSs specified by the VPS, can be derived as follows: if(vps_max_layers_minus1==0) TotalNumOlss=1 else if(each_layer_is_an_ols_flag 1 1 ols_mode_idc==0 1 1 ols_mode_idc==1) TotalNumOlss=vps_max_layers_minus1+1 else if(ols_mode_idc==2) TotalNumOlss=num_output_layer_sets_minus1+1

[0432] If ols_output_layer_flag[i][j] is 1, it can specify that the layer with nuh_layer_id equal to vps_layer_id[j] is the output layer of the i-th OLS when ols_mode_idc is equal to 2, and if ols_output_layer_flag[i][j] is 0, it specifies that the layer with nuh_layer_id equal to vps_layer_id[j] is not the output layer of the i-th OLS when ols_mode_idc is equal to 2.

[0433] The variable NumOutputLayerslnOls[i] specifying the number of output layers in the ith OLS, the variable NumSubLayerslnLayerlnOLS[i][j] specifying the number of sublayers in the jth layer in the ith OLS, the variable OutputLayerldlnOls[i][j] specifying the nuh_layer_id value of the jth output layer in the ith OLS, and the variable LayerUsedAsOutputLayerFlag[k] specifying whether the kth layer is used as an output layer in at least one OLS can be derived as follows: NumOutputLayerslnOls

[0000] =1 OutputLayerldlnOls

[0000]

[0000] =vps_layer_id

[0000] NumSubLayerslnLayerlnOLS

[0000]

[0000] =vps_max_sub_layers_minus1+1 LayerUsedAsOutputLayerFlag

[0000] =1 for(i=1,i <=vps_max_layers_minus1;i++){ if(each_layer_is_an_ols_flag 1 1 ols_mode_idc < 2) LayerUsedAsOutputLayerFlag[ i ]=1 else / *(!each_layer_is_an_ols_flag&&ols_mode_idc==2)* / LayerUsedAsOutputLayerFlag[ i ]=0 } for(i=1;i < TotalNumOlss;i++) if(each_layer_is_an_ols_flag 1 1 ols_mode_idc==0) {NumOutputLayerslnOls[i]=1 OutputLayerldlnOls[i][0]=vps_layer_id[i] for(j=0;j < i&&(ols_mode_idc==0);j++) NumSubLayerslnLayerlnOLS[ i ][ j ]=max_tid_il_ref_pics_plus1[ i ] NumSubLayerslnLayerlnOLS[ i ][ i ]=vps_max_sub_layers_minus1+1 }else if(ols_mode_idc==1){ NumOutputLayerslnOls[ i ]=i+1 for(j=0;j < NumOutputLayerslnOls[ i ];j++) { OutputLayerldlnOls[ i ][ j ]=vps_layer_id[ j ] NumSubLayerslnLayerlnOLS[ i ][ j ]=vps_max_sub_layers_minus1+1 } }else if(ols_mode_idc==2){ for(j=0;j <=vps_max_layers_minus1;j++) { layerlncludedlnOlsFlag[ i ][ j ]=0 NumSubLayerslnLayerlnOLS[ i ][ j ]=0 } for(k=0,j=0;k <=vps_max_layers_minus1;k++) if(ols_output_layer_flag[ i ][ k ]){ layerlncludedlnOlsFlag[ i ][ k ]=1 LayerUsedAsOutputLayerFlag[ k ]=1 OutputLayerldx[ i ][ j ]=k OutputLayerldlnOls[ i ][ j++]=vps_layer_id[ k ] NumSubLayerslnLayerlnOLS[ i ][ j ]=vps_max_sub_layers_minus1+1 } NumOutputLayerslnOls[ i ]=j for(j=0;j < NumOutputLayerslnOls[ i ];j++) { idx=OutputLayerldx[ i ][ j ] for(k=0;k < NumRefLayers[ idx ];k++) { layerlncludedlnOlsFlag[ i ][ RefLayerldx[ idx ][ k ] ]=1 if(NumSubLayerslnLayerlnOLS[ i ][ RefLayerldx[ idx ][ k ] ] < max_tid_il_ref_pics_plus1[ OutputLayerldlnOls[ i ][ j ] ]) NumSubLayerslnLayerlnOLS[ i ][ RefLayerldx[ idx ][ k ] ]= max_tid_il_ref_pics_plus1[ OutputLayerldlnOls[ i ][ j ] ] } } }

[0434] For each value of i in the range 0 to vps_max_layers_minus1, inclusive, the values ​​of LayerUsedAsRefLayerFlag[i] and LayerUsedAsOutputLayerFlag[i] may not all be equal to 0. Thus, there may be no layers that are not the output layer of at least one OLS and a direct reference layer to another layer.

[0435] For each OLS, there may be at least one layer that is the output layer. Thus, for any value of i in the range 0 to TotalNumOlss-1, inclusive, the value of NumOutputLayerslnOls[i] may be 1 or greater.

[0436] The variable NumLayerslnOls[i], which specifies the number of layers in the i-th OLS, and the variable LayerldlnOls[i][j], which specifies the nuh_layer_id value of the j-th layer in the i-th OLS, are derived as follows: NumLayerslnOls

[0000] =1 LayerldlnOls

[0000]

[0000] =vps_layer_id

[0000] for(i=1;i < TotalNumOlss;i++){ if(each_layer_is_an_ols_flag) { NumLayerslnOls[ i ] = 1 LayerldlnOls[ i ]

[0000] =vps_layer_id[ i ] } else if(ols_mode_idc==0 1 1 ols_mode_idc==1){ NumLayerslnOls[ i ]=i+1 for(j=0;j < NumLayerslnOls[ i ];j++) LayerldlnOls[ i ][ j ]=vps_layer_id[ j ] }else if(ols_mode_idc==2){ for(k=0,j=0;k <=vps_max_layers_minus1;k++) if(layerlncludedlnOlsFlag[ i ][ k ]) LayerldlnOls[ i ][ j++]=vps_layer_id[ k ] NumLayerslnOls[ i ]=j } }

[0437] The variable OlsLayerldx[i][j], which specifies the OLS layer index of the layer whose nuh_layer_id is equal to LayerldlnOls[i][j], can be derived as follows: for(i=0;i < TotalNumOlss;i++) for j=0;j < NumLayerslnOls[i];j++) OlsLayerldx[ i ][ LayerldlnOls[i][j]]=j

[0438] The bottom layer in each OLS may be an independent layer, so for each i in the range 0 to TotalNumOlss-1, inclusive, the value of vps_independent_layer_flag[GeneralLayerldx[LayerldlnOls[i][0]]] may be equal to 1.

[0439] Each layer may be included in at least one OLS specified by the VPS. Thus, for each layer with a particular value of nuh_layer_id nuhLayerld equal to one of vps_layer_id[k] for k in the range of 0 to vps_max_layers_minus1, inclusive, there may be at least one pair of values ​​of i and j such that the value of LayerldlnOls[i][j] is equal to nuhLayerld, where i is in the range of 0 to TotalNumOlss-1, inclusive, and j is in the range of NumLayerslnOls[i]-1, inclusive.

[0440] In one embodiment, the decoding process operates as follows for the current image CurrPic. -PictureOutputFlag is set as follows: - PictureOutputFlag is set equal to 0 if one of the following conditions is true: - The current image is a RASL image and the associated lRAP image's NoOutputBeforeRecoveryFlag is equal to 1. -gdr_enabled_flag is equal to 1 and the current image is a GDR image that has NoOutputBeforeRecoveryFlag equal to 1. - gdr_enabled_flag is equal to 1, the current image is associated with a GDR image with NoOutputBeforeRecoveryFlag equal to 1, and the current image's PicOrderCntVal is less than the associated GDR image's RpPicOrderCntVal. -sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0, and the current AU contains an image picA that satisfies all of the following conditions: -PicA's PictureOutputFlag is 1. -PicA has a larger nuh_layer_id nuhLid than the current image. -PicA belongs to the output layer of OLS (i.e., OutputLayerldlnOls[TargetOlsldx][0] is equal to nuhLid). -sps_video_parameter_set_id is greater than 0 and ols_mode_idc is equal to 2, ols_output_layer_flag [TargetOlsldx] [GeneralLayerldx [nuh_layer_id] ] is equal to 0. - Otherwise, PictureOutputFlag is set equal to pic_output_flag.

[0441] After all slices of the current picture have been decoded, the current decoded picture is marked as "used for short-term reference" and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as "used for short-term reference".

[0442] In one embodiment, if each layer is an output layer set, PictureOutputFlag is set equal to pic_output_flag, regardless of the value of ols_mode_idc.

[0443] In one embodiment, PictureOutputFlag is set equal to 0 if sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is equal to 0, ols_mode_idc is equal to 0, and the current AU contains an image picA that satisfies all of the following conditions: PicA has PictureOutputFlag equal to 1, PicA has nuh_layer_id nuhLid greater than that of the current image, and PicA belongs to the output layer of OLS (i.e., OutputLayerldlnOls[TargetOlsldx][0] is equal to nuhLid).

[0444] In one embodiment, PictureOutputFlag is set equal to 0 if sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is equal to 0, ols_mode_idc is equal to 2, and ols_output_layer_flag[TargetOlsldx][GeneralLayerldx[nuh_layer_id]] is equal to 0.

[0445] An example of the syntax structure of a VPS RBSP is shown in Figure 35. The syntax structure shown in Figure 35 is similar to the syntax structure shown in Figure 34, and therefore, a detailed description is omitted for brevity.

[0446] The techniques described above (e.g., techniques for signaling constraint flags, adaptive resolution parameters, and / or the like) can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 36 illustrates a computer system (3600) suitable for implementing certain embodiments of the disclosed subject matter.

[0447] Computer software can be encoded using any suitable machine code or computer language that can undergo mechanisms such as assembly, compilation, linking, etc. to produce code containing instructions that can be executed directly, or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0448] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0449] The components illustrated in Figure 36 for the computer system (3600) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of the computer system (3600).

[0450] The computer system (3600) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), video (2D video, 3D video including stereoscopic video, etc.).

[0451] The input human interface devices may include one or more (only one of each is depicted) of a keyboard (3601), a mouse (3602), a trackpad (3603), a touch screen (3610), a data glove (not shown), a joystick (3605), a microphone (3606), a scanner (3607), and a camera (3608).

[0452] The computer system (3600) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, by tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touch screen (3610), data gloves (not shown), or joystick (3605), although there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (3609), headphones (not shown)), visual output devices (e.g., screens (3610), including CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touch screen input capability, each with or without tactile feedback capability, some of which may be capable of outputting two-dimensional visual output or output in more than three dimensions via means such as stereo output, virtual reality glasses (not shown), holographic displays, and smoke generator tanks (not shown)), and printers (not shown).

[0453] The computer system (3600) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (3620) with media such as CDs / DVDs (3621), thumb drives (3622), removable hard drives or solid state drives (3623), legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0454] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0455] The computer system (3600) may also include an interface (3654) to one or more communication networks (3655). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial including CANBus, etc. Certain networks generally require an external network interface adapter (e.g., a USB port on the computer system (3600)) attached to a particular general-purpose data port or peripheral bus (3649), while others are generally integrated into the core of the computer system (3600) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (3600) can communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast TV), unidirectional transmit only (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0456] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core (3640) of the computer system (3600).

[0457] The cores (3640) may include one or more central processing units (CPUs) (3641), graphics processing units (GPUs) (3642), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (3643), hardware accelerators for specific tasks (3644), graphics adapters (~50), and the like. These devices may connect via a system bus (3648) along with read only memory (ROM) (3645), random access memory (3646), internal mass storage devices such as internal hard drives that are not accessible to the user, SSDs, and the like (3647). In some computer systems, the system bus (3648) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus (3648) or via a peripheral bus (3649). In one example, a screen (3610) may be connected to a graphics adapter (3650). Peripheral bus architectures include PCI, USB, etc.

[0458] The CPU (3641), GPU (3642), FPGA (3643), and accelerator (3644) can execute certain instructions that, in combination, can constitute the computer code described above. The computer code can be stored in a ROM (3645) or a RAM (3646). Temporary data can also be stored in the RAM (3646), while persistent data can be stored, for example, in an internal mass storage (3647). Rapid storage and retrieval in any of the memory devices can be made possible by the use of cache memories that can be closely associated with one or more of the CPU (3641), GPU (3642), mass storage (3647), ROM (3645), RAM (3646), etc.

[0459] The computer-readable medium can bear computer code for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0460] By way of example and not limitation, the architecture, particularly the computer system (3600) having the cores (3640), can provide functionality as a result of processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of the cores (3640) of a non-transitory nature, such as the core internal mass storage (3647) or ROM (3645). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the cores (3640). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the cores (3640), and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM (3646) and modifying such data structures according to the processes defined by the software. Additionally, or alternatively, a computer system may provide functionality as a result of logic embodied in hardwired or otherwise circuitry (e.g., accelerator (3644)), which may operate in place of or in conjunction with software to perform particular processes or particular portions of particular processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0461] Appendix A: Acronyms JEM: joint exploration model VVC:versatile video coding BMS:benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI:Video Usability Information GOP: Groups of Pictures TU: Transform Units PU: Prediction Units CTU: Coding Tree Units CTB: Coding Tree Blocks PB: Prediction Blocks HRD:Hypothetical Reference Decoder SNR: Signal Noise Ratio CPU: Central Processing Units GPU: Graphics Processing Units CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Areas SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit

[0462] While this disclosure has described several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0463] 300 Communication Systems 310 Terminal Equipment 320 Terminal Equipment 330 Terminal Equipment 350 Communication Network 400 Communication Systems 401 Video Source 402 Stream 403 Video Encoder 404 Video Data, Video Bitstream 405 Streaming Server 406 Client Subsystem 407 Video Data 410 Video Decoder 411 Output Stream 412 Display 413 Capture Subsystem 420 Electronic equipment 430 Electronic equipment 501 Channel 510 Video Decoder 512 Rendering Equipment 515 Buffer Memory 520 Parser 521 Symbols 530 Electronic equipment 531 Receiver 551 Reverse conversion unit 552 Intra Image Prediction Unit 553 Motion Compensation Prediction Unit 555 Aggregation Device 556 Loop Filter Unit 557 Reference Image Memory 558 Image Buffer 601 Video Sources 603 Video Encoder 620 Electronic equipment 630 Source Coder 632 encoding engine 633 Decoder 634 Reference Image Memory, Reference Image Cache 635 Predictors 640 Transmitter 643 Video Sequences 645 Entropy Coder 650 Controller 660 Communication Channels 703 Video Encoder 721 General-purpose controller 722 Intra Encoder 723 Residual Calculation Unit 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 InterEncoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconstruction Module 880 Interdecoder 901 Image Header 902 ARC information 905 ARC Reference Information 906 Target Resolution Table 908 Tile Group Header 909 ARC information 911 Parameter Set 912 ARC information 913 ARC Reference Information 914 Tile Group Header 915 ARC information 916 Parameter Set 1001 Tile Group Header 1011 Flag 1012 If Statement 1013 Output Resolution 1015 Reference Image Dimensions 1016 Syntax Elements 1017 Syntax Elements 1201 Constraint Flags 1202 Constraint Flags 1400 General Constraint Information Syntax Structure 1401 Gate Flag 1402 Gate Flag 1403 Gate Flag 1404 Gate Flag 1405 Gate Flag 1406 Gate Flag 1407 Gate Flag 1408 Gate Flag 1410 Constraint Information Group 1411 Constraint Flags 1412 Constraint Flags 1413 Constraint Flags 1414 Constraint Flags 1420 Constraint Information Group 1430 while loop 1500 General Constraint Information Syntax Structure 1510 Constraint Information Group 1511 Constraint Information Group 1512 Constraint Information Group 1513 Constraint Information Group 1514 Constraint Information Group 1515 Constraint Information Group 1516 Constraint Information Group 1517 Constraint Information Group 1518 Constraint Information Group 1900 Process 3600 Computer Systems, Architecture 3601 Keyboard 3602 Mouse 3603 Trackpad 3605 Joystick 3606 Mike 3607 Scanner 3608 Camera 3609 Speaker 3610 Touch Screen 3620 CD / DVD ROM / RW 3621 Media 3622 Thumb Drive 3623 Removable Hard Drive or Solid State Drive 3640 cores 3641 CPU 3642 GPU 3643 FPGA 3644 Hardware Accelerator 3645 ROM 3646 Random Access Memory 3647 Internal Mass Storage 3648 System Bus 3649 Surrounding bus 3650 Graphics Adapter 3654 Interface 3655 Communication Networks

Claims

1. A method of video encoding performed by an encoder, comprising: generating constraint information for a subset of coding blocks, the constraint information including a gating flag for each of at least one constraint information group, each gating flag indicating whether a constraint information group of the at least one constraint information group corresponding to the respective gating flag is present in the constraint information, the constraint information group of the gating flag including a plurality of constraint flags for the subset of coding blocks, whether the constraint information group of the gating flag is present in the constraint information and the plurality of constraint flags in the constraint information group of the gating flag are used to determine prediction information for the subset of coding blocks, the prediction information being used to reconstruct the subset of coding blocks, an offset and a length being present in the constraint information, and the plurality of constraint flags in the constraint information group of the gating flag indicating the presence in the constraint information; generating an encoded video bitstream including the constraint information; A method comprising:

2. The method of claim 1 , wherein the constraint information group of the gate flag is determined to be absent in the constraint information.

3. The method of claim 1 , wherein each gate flag indicates that the constraint information group corresponding to the respective gate flag is not present in the constraint information and that no constraint flag is present in the constraint information.

4. The method of any one of claims 1 to 3, wherein the constraint information is in a high level syntax for at least one image, and the subset of coding blocks comprises coding blocks in the at least one image.

5. one or more constraint information groups of the at least one constraint information group are present in the encoded video bitstream; The method according to any one of claims 1 to 4, wherein a plurality of constraint flags in the one or more constraint information groups of the at least one constraint information group are signaled according to a predetermined order.

6. the plurality of constraint flags in the constraint information group of the gate flag include a plurality of constraint flags notified in a predetermined order, The method of claim 1 , wherein the plurality of constraint flags are run-decoded and used to determine prediction information.

7. A method of video encoding performed by an encoder, comprising: generating constraint information for a subset of coding blocks, the constraint information including a gating flag for each of at least one constraint information group, each gating flag indicating whether a constraint information group of the at least one constraint information group corresponding to the respective gating flag is present in the constraint information, the constraint information group of the gating flag including a plurality of constraint flags for the subset of coding blocks, whether the constraint information group of the gating flag is present in the constraint information is used to determine prediction information for the subset of coding blocks, the prediction information being used to reconstruct the subset of coding blocks, and for each of the one or more constraint information groups of the at least one constraint information group, an offset and a length are present in the constraint information, and a plurality of constraint flags in the respective one of the one or more constraint information groups of the at least one constraint information group indicate presence in the constraint information; generating an encoded video bitstream including the constraint information; A method comprising:

8. one or more of the at least one constraint information group are present in the encoded video bitstream; 2. The method of claim 1 , wherein one or more syntax elements in the constraint information are byte-aligned, and the one or more syntax elements include the gating flag for each of the at least one constraint information group and a plurality of constraint flags in the one or more constraint information groups of the at least one constraint information group.

9. Apparatus configured to perform the method according to any one of claims 1 to 8.

Images