Method and device for encoding and decoding video data

Abstract

The invention provides a method and a device for decoding video data. The method comprises the following steps. Video data including one or more sub-pictures is received. A hybrid network abstraction layer (NAL) unit type associated with each of the one or more sub-pictures is identified based on checking a flag corresponding to a NAL unit in the one or more sub-pictures. And decoding the video data based on the identified NAL unit type.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 003,137 filed with the U.S. Patent and Trademark Office on March 31, 2020 and U.S. Patent Application No. 17 / 095,289 filed with the U.S. Patent and Trademark Office on November 11, 2020, the entire contents of the above-mentioned U.S. patent applications are incorporated herein by reference.

[0003] This application is a divisional application for the Chinese patent application with application number 202180002889.5, application date March 16, 2021, and invention name “A method and device for decoding video data”. Technical Field

[0004] The present disclosure relates to the field of data processing technology, and more particularly to a method and device for decoding video data. Background Art

[0005] Video encoding and decoding using inter-picture prediction with motion compensation has been known for decades. Uncompressed digital video can be composed of a series of pictures, each picture having a spatial dimension of, for example, 1920x 1080 luminance samples and associated chrominance samples. This series of pictures can have a fixed or variable picture rate (also informally referred to as a frame rate) of, for example, 60 pictures per second or 60Hz. Uncompressed video has very high bit rate requirements. For example, 1080p60 4:2:0 video (1920x 1080 luminance sample resolution at 60Hz frame rate) at 8 bits per sample requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in the input video signal through compression. Compression can help reduce the bandwidth or storage space requirements mentioned above, in some cases by two orders of magnitude or more. Lossless compression and lossy compression and their combinations can be employed. Lossless compression refers to techniques that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely adopted. The amount of distortion tolerated depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of TV contribution applications. The achievable compression ratio can reflect that higher allowable / tolerable distortion can produce higher compression ratios.

[0007] Video encoders and decoders may utilize techniques from several broad categories, including, for example, motion compensation, transforms, quantization, and entropy coding, some of which are described below.

[0008] Historically, video encoders and decoders have tended to operate on a given picture size, which in most cases is defined and kept constant for a coded video sequence (CVS), a group of pictures (GOP), or a similar multi-picture time frame. For example, in MPEG-2, it is known that system designs change the horizontal resolution (and thus the picture size) according to factors such as the activity of the scene, but only at the I picture, and therefore usually for the GOP. For example, according to ITU-T Rec. H.263 Annex P, it is known to resample reference pictures in order to use different resolutions within a CVS. However, here the picture size does not change, only the reference picture is resampled, which may result in only part of the picture canvas being used (in the case of downsampling) or only part of the scene being captured (in the case of upsampling). In addition, H.263 Annex Q allows a single macroblock to be resampled up or down by a factor of 2 (in each dimension). Again, the picture size remains the same. The size of the macroblock is fixed in H.263 and therefore does not need to be signaled.

[0009] Changes in picture size in predicted pictures have become more mainstream in modern video coding. For example, VP9 allows reference picture resampling and resolution changes of the entire picture. Similarly, certain recommendations made for VVC (including, for example, Hendry et al., "On adaptive resolution change (ARC) for VVC", Joint Video Group document JVET-M0135-v1, January 9-19, 2019, the entire contents of which are incorporated herein) allow the entire reference picture to be resampled to a different (higher or lower) resolution. In this document, it is proposed that different candidate resolutions be encoded in a sequence parameter set and referenced by a per-picture syntax element in a picture parameter set.

[0010] Video encoders and decoders tend to operate on a given picture size, which in most cases is defined for a coded video sequence and remains constant. However, a picture can be divided into one or more sub-pictures. Each sub-picture can be further divided into one or more slices. Two or more independently coded sub-pictures can be merged into a coded picture, decoded by a decoder and displayed as a single output picture. When two or more independently coded pictures are merged into a coded picture, some coding or decoding constraints need to be specified, but there is currently no effective way to achieve this goal. Summary of the invention

[0011] Embodiments relate to methods, systems, and computer-readable media for decoding video data. According to one aspect, a method for decoding video data is provided. The method may include receiving video data including one or more sub-pictures. Based on checking a flag corresponding to a hybrid network abstraction layer (NAL) unit in the one or more sub-pictures, a NAL unit type associated with each of the one or more sub-pictures is identified. The video data is decoded based on the identified NAL unit type.

[0012] According to another aspect, a device for decoding video data is provided. The device includes: a receiving module for receiving video data including one or more sub-pictures; an identification module for identifying a hybrid network abstraction layer (NAL) unit type associated with each of the one or more sub-pictures based on checking a flag corresponding to a NAL unit in the one or more sub-pictures; and a decoding module for decoding the video data based on the identified NAL unit type.

[0013] According to another aspect, a computer device for decoding video data is provided. The computer device may include one or more processors, and one or more computer-readable non-transitory storage media. The one or more computer-readable non-transitory storage media are configured to store computer program code, which is used by the one or more computer processors to execute a method when the computer device executes the one or more sub-pictures. The method may include receiving video data including one or more sub-pictures. Based on checking a flag corresponding to a hybrid network abstraction layer (NAL) unit in the one or more sub-pictures, a NAL unit type associated with each of the one or more sub-pictures is identified. The video data is decoded based on the identified NAL unit type.

[0014] According to another aspect, a computer-readable medium for decoding video data is provided. The computer-readable medium may store a computer program for decoding video data, the computer program being configured to cause one or more computer processors to perform a method. The method may accordingly include receiving video data including one or more sub-pictures. Based on checking a flag corresponding to a hybrid network abstraction layer (NAL) unit in the one or more sub-pictures, a NAL unit type associated with each of the one or more sub-pictures is identified. The video data is decoded based on the identified NAL unit type.

[0015] Therefore, an embodiment of the present application provides a method and apparatus for decoding video data, identifying a hybrid network abstraction layer (NAL) unit type associated with each of one or more sub-pictures in the video data by checking a flag corresponding to a NAL unit in one or more sub-pictures, and decoding the video data based on the identified NAL unit type, thereby specifying encoding or decoding constraints for an encoded picture formed by merging two or more independently encoded pictures. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] These and other objects, features and advantages will become apparent from the following detailed description of illustrative embodiments which is to be read in conjunction with the accompanying drawings. The various features of the drawings are not drawn to scale as these illustrations are intended to be clearly understood by those skilled in the art in conjunction with the detailed description. In the drawings:

[0017] Figure 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment;

[0018] Figure 2 is a schematic diagram of a simplified block diagram of a communication system according to another embodiment;

[0019] Figure 3 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment;

[0020] Figure 4 is a schematic diagram of a simplified block diagram of an encoder according to another embodiment;

[0021] Figure 5 is a schematic diagram of options for signaling ARC parameters according to prior art or implementations as indicated;

[0022] Figure 6 is an example of a syntax table according to an embodiment;

[0023] Figure 7 is a schematic diagram of a computer system according to an embodiment;

[0024] Figure 8 is an example of a prediction structure for scalability with adaptive resolution changes;

[0025] Fig. 9 is an example of a syntax table according to an embodiment;

[0026] Fig.10 is a schematic diagram of a simplified block diagram for parsing and decoding a poc period and access unit count value per access unit;

[0027] Fig.11 is a schematic diagram of a video bitstream structure including multiple layers of sub-pictures;

[0028] Fig.12 is a schematic diagram of a display of a selected sub-picture with enhanced resolution;

[0029] Fig.13 is a block diagram of a decoding and display process for a video bitstream including multiple layers of sub-pictures;

[0030] Fig.14 is a schematic diagram of a 360-degree video display with an enhancement layer of a sub-picture;

[0031] Fig.15 is an example of sub-picture layout information and its corresponding layer and picture prediction structure;

[0032] Fig.16 is an example of layout information of sub-pictures of a spatial scalability modality with local regions and their corresponding layer and picture prediction structures;

[0033] Fig.17 is an example of a syntax table for sub-picture layout information;

[0034] Fig.18 is an example of a syntax table of a SEI message for sub-picture layout information;

[0035] Fig.19 is an example of a syntax table for indicating output layers and profile / layer / level information for each output layer set;

[0036] Fig. 20 is an example of a syntax table for indicating the output layer mode for each output layer set;

[0037] Fig.21 is an example of a syntax table for indicating a current sub-picture for each layer of each output layer set;

[0038] Fig. 22 is an example of a syntax table for indicating a sub-picture identifier;

[0039] Fig.23 is an example of a syntax table for indicating sub-picture partitioning information;

[0040] Fig.24 is an example of a syntax table for indicating a mixed NAL unit type and associated sub-picture partitioning information. DETAILED DESCRIPTION

[0041] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it is understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that may be implemented in various forms. However, those structures and methods may be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Instead, these exemplary embodiments are provided so that the disclosure will be thorough and complete and will fully convey the scope to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessary confusion in the presented embodiments.

[0042] As previously described, video encoders and decoders tend to operate on a given picture size, which in most cases is defined for a coded video sequence and remains constant. However, a picture can be divided into one or more sub-pictures. Each sub-picture can be further divided into one or more slices. Two or more independently coded sub-pictures can be merged into a coded picture, decoded by a decoder and displayed as a single output picture. Therefore, it may be advantageous to specify some coding or decoding constraints when two or more independently coded pictures are merged into a coded picture.

[0043] Various aspects are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer-readable media according to various embodiments. It should be understood that each block of the flowchart and / or block diagram and the combination of blocks in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0044] Figure 1 A simplified block diagram of a communication system (100) according to an embodiment of the present disclosure is shown. The system (100) may include at least two terminals (110-120) interconnected via a network (150). For unidirectional transmission of data, a first terminal (110) may encode video data at a local location for transmission to another terminal (120) via the network (150). A second terminal (120) may receive encoded video data of another terminal from the network (150), decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in media service applications, etc.

[0045] Figure 1A second pair of terminals (130, 140) is shown, which are provided to support bidirectional transmission of encoded video that may occur, for example, during a video conference. For bidirectional transmission of data, each terminal (130, 140) can encode captured video data at a local location for transmission to the other terminal via a network (150). Each terminal (130, 140) can also receive encoded video data transmitted by the other terminal, can decode the encoded data, and can display the recovered video data at a local display device.

[0046] exist Figure 1 In the embodiment of the present disclosure, the terminals (110-140) may be illustrated as servers, personal computers, and smart phones, but the principles of the present disclosure may not be limited thereto. Embodiments of the present disclosure may be applied to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (150) represents any number of networks that transmit coded video data between the terminals (110-140), including, for example, wired and / or wireless communication networks. The communication network (150) may exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (150) may be unimportant to the operation of the present disclosure unless otherwise described below.

[0047] Figure 2 The placement of a video encoder and decoder in a streaming environment is shown as an example of application of the disclosed subject matter. The disclosed subject matter can be equally applied to other video-enabled applications including, for example, video conferencing, digital television, storing compressed video on digital media including CDs, DVDs, memory sticks, etc., etc.

[0048] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a digital video camera, that creates, for example, an uncompressed video sample stream (202). The sample stream (202) - depicted as a thick line to emphasize the high data capacity compared to the encoded video bit stream - can be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement various aspects of the disclosed subject matter, as described in more detail below. The encoded video bit stream (204) - depicted as a thin line to emphasize the lower data capacity compared to the sample stream - can be stored on a streaming server (205) for future use. One or more streaming clients (206, 208) can access the streaming server (205) to retrieve a copy (207, 209) of the encoded video bit stream (204). The client (206) may include a video decoder (210) that decodes an input copy of an encoded video bitstream (207) and creates an output video sample stream (211) that can be presented on a display (212) or other presentation device (not depicted). In some streaming systems, the video bitstreams (204, 207, 209) may be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T Recommendation H.265. Under development is a video coding standard informally referred to as Versatile Video Coding or VVC. The disclosed subject matter may be used in the context of VVC.

[0049] Figure 3 It may be a functional block diagram of a video decoder (210) according to an embodiment.

[0050] The receiver (310) may receive one or more coded video sequences to be decoded by the decoder (210); in the same or another embodiment, one coded video sequence at a time, wherein the decoding of each coded video sequence is independent of the other coded video sequences. The coded video sequence may be received from a channel (312), which may be a hardware / software link to a storage device storing the coded video data. The receiver (310) may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective consuming entities (not depicted). The receiver (310) may separate the coded video sequence from the other data. To combat network jitter, a buffer memory (315) may be coupled between the receiver (310) and an entropy decoder / parser (320) (hereinafter referred to as "parser"). When the receiver (310) receives data from a store / forward device with sufficient bandwidth and controllability or from a synchronous network, the buffer (315) may not be needed, or the buffer (315) may be small. For use over a best effort packet network such as the Internet, a buffer (315) may be required, which may be relatively large and may advantageously have an adaptive size.

[0051] The video decoder (210) may include a parser (320) for reconstructing symbols (321) from an entropy coded video sequence. The types of these symbols include information for managing the operation of the decoder (210) and possibly for controlling a presentation device such as a display (212) that is not part of the decoder but may be coupled to the decoder, such as Figure 2As shown. The control information for the rendering device may be in the form of a supplementary enhancement information (SEI (Supplementary Enhancement Information) message) or a video usability information (VUI, Video Usability Information) parameter set fragment (not depicted). The parser (320) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be based on a video coding technique or standard and may follow principles known to those skilled in the art, including variable length coding with or without context sensitivity, Huffman coding, arithmetic coding, etc. The parser (320) may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP, Groups of Pictures), a picture, a tile, a slice, a macroblock, a coding unit (CU, Coding Unit), a block, a transform unit (TU, Transform Unit), a prediction unit (PU, Prediction Unit), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0052] The parser (320) may perform entropy decoding / parsing operations on the video sequence received from the buffer (315) to create symbols (321).

[0053] Depending on the type of coded video picture or part thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of the symbol (321) may involve a plurality of different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser (320). For clarity, such subgroup control information flow between the parser (320) and the following plurality of units is not depicted.

[0054] In addition to the functional blocks already mentioned, decoder 210 can be conceptually subdivided into a plurality of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the following functional units.

[0055] The first unit is a scaler / inverse transform unit (351). The scaler / inverse transform unit (351) receives quantized transform coefficients from the parser (320) as well as control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. as symbols (321). It can output blocks including sample values, which can be input into an aggregator (355).

[0056] In some cases, the output samples of the scaler / inverse transform (351) may belong to an intra-coded block; that is, a block that does not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) uses surrounding reconstructed information obtained from the current (partially reconstructed) picture (356) to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator (355) adds the prediction information already generated by the intra-prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351) on a per-sample basis.

[0057] In other cases, the output samples of the scaler / inverse transform unit (351) may relate to inter-coded and possibly motion compensated blocks. In such cases, the motion compensated prediction unit (353) may access a reference picture buffer (357) to obtain samples for prediction. After motion compensation of the obtained samples according to the symbols (321) associated with the block, these samples may be added by an aggregator (355) to the output of the scaler / inverse transform unit (in this case referred to as residual samples or residual signal) to generate output sample information. The addresses within the reference picture memory from which the motion compensation unit obtains the predicted samples may be controlled by motion vectors, which are available to the motion compensation unit in the form of symbols (321), which may have, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​obtained from the reference picture memory when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0058] The output samples of the aggregator (355) may be subjected to various loop filtering techniques in a loop filter unit (356). The video compression techniques may include loop filtering techniques controlled by parameters included in the coded video bitstream and available to the loop filter unit (356) as symbols (321) from the parser (320), but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of a coded picture or coded video sequence, and to previously reconstructed and loop filtered sample values.

[0059] The output of the loop filter unit (356) may be a sample stream that may be output to a rendering device (212) and stored in a reference picture buffer (357) for future inter-picture prediction.

[0060] Certain coded pictures, once fully reconstructed, can be used as reference pictures for future prediction. Once a coded picture is fully reconstructed and the coded picture has been identified as a reference picture (by, for example, the parser (320)), the current picture (356) can become part of the reference picture buffer (357) and a new current picture memory can be reallocated before starting to reconstruct a subsequent coded picture.

[0061] The video decoder 320 may perform decoding operations according to a predetermined video compression technique documented in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by the video compression technology or standard being used in the sense that the encoded video sequence follows the syntax of the video compression technology or standard specified in the video compression technology document or standard and in particular in the profile document in the video compression technology document or standard. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further constrained by the hypothetical reference decoder (HRD) specification signaled in the encoded video sequence and metadata for HRD buffer management.

[0062] In an embodiment, the receiver (310) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (320) to correctly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0063] Figure 4 It may be a functional block diagram of a video encoder (203) according to an embodiment of the present disclosure.

[0064] The encoder (203) may receive video samples from a video source (201) (which is not part of the encoder) that may capture video images to be encoded by the encoder (203).

[0065] The video source (201) may provide a source video sequence to be encoded by the encoder (203) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (201) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (201) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that, when viewed sequentially, impart motion. The picture itself may be organized as a spatial array of pixels, wherein each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0066] According to an embodiment, the encoder (203) can encode and compress the pictures of the source video sequence into an encoded video sequence (443) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is one of the functions of the controller (450). The controller controls other functional units as described below and is functionally coupled to these units. For clarity, the coupling is not depicted. The parameters set by the controller may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology...), picture size, picture group (GOP, group of pictures) layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller (450) because they may involve a video encoder (203) optimized for a certain system design.

[0067] Some video encoders operate in a manner that is readily recognizable to those skilled in the art as a "coding loop". As an oversimplified description, the coding loop may consist of an encoding portion of an encoder (430) (hereinafter referred to as a "source encoder"), which is responsible for creating symbols based on the input picture to be encoded and the reference picture, and a (local) decoder (433) embedded in the encoder (203), which reconstructs the symbols to create sample data that will also be created by the (remote) decoder (since in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). This reconstructed sample stream is input to a reference picture memory (434). Since the decoding of the symbol stream results in bit-accurate results that are independent of the decoder location (local or remote), the reference picture buffer contents are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values ​​that the decoder will "see" when using prediction during decoding. The basic principles of this reference picture synchronization (and the drift that results when synchronization cannot be maintained, for example due to channel errors) are well known to those skilled in the art.

[0068] The operation of the "local" decoder (433) can be combined with the above Figure 3 The operation of the "remote" decoder (210) described in detail is identical. Figure 3 , however, when symbols are available and encoding / decoding of the symbols into a coded video sequence can be lossless through the entropy encoder (445) and the parser (320), the entropy decoding portion of the decoder (210) including the channel (312), the receiver (310), the buffer (315) and the parser (320) may not be fully implemented in the local decoder (433).

[0069] At this point it can be observed that any decoder techniques other than parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter focuses on the decoder operation. The description of encoder techniques can be simplified because they are inverse to the fully described decoder techniques. A more detailed description is only required in certain areas and is provided below.

[0070] As part of its operation, the source encoder (430) may perform motion compensated predictive coding, which predictively encodes an input frame with reference to one or more previously encoded frames from a video sequence designated as "reference frames." In this manner, the encoding engine (432) encodes the differences between pixel blocks of an input frame and pixel blocks of a reference frame that may be selected as a prediction reference for the input frame.

[0071] The local video decoder (433) can decode the encoded video data of the frame that can be designated as the reference frame based on the symbols created by the source encoder (430). The operation of the encoding engine (432) can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 4 When decoded at a remote video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (433) replicates the decoding process that may be performed by the video decoder on the reference frame and may cause the reconstructed reference frame to be stored in the reference picture memory (434). In this way, the encoder (203) may store a copy of the reconstructed reference frame locally that has common content (absent transmission errors) with the reconstructed reference frame that will be obtained by the remote video decoder.

[0072] The predictor (435) may perform a prediction search for the encoding engine (432). That is, for a new frame to be encoded, the predictor (435) may search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (435) may operate on a pixel block by pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (435), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (434).

[0073] The controller (450) can manage encoding operations of the source encoder (430), including, for example, setting parameters and sub-group parameters for encoding video data.

[0074] The outputs of all the aforementioned functional units may be subjected to entropy coding in an entropy encoder (445). The entropy encoder converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0075] The transmitter (440) can buffer the encoded video sequence created by the entropy encoder (445) in preparation for transmission via a communication channel (460), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (440) can merge the encoded video data from the source encoder (430) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0076] The controller (450) can manage the operation of the encoder (203). During encoding, the controller (450) can assign a specific coded picture type to each coded picture, which can affect the coding techniques that can be applied to the corresponding picture. For example, a picture can generally be assigned one of the following frame types:

[0077] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of those variations of I pictures and their respective applications and features.

[0078] A predicted picture (P picture) may be a picture that may be encoded and decoded using intra prediction or inter prediction, which predicts sample values ​​of each block using at most one motion vector and a reference index.

[0079] Bidirectional prediction pictures (B pictures) can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-prediction pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0080] The source picture may typically be spatially subdivided into a plurality of blocks of samples (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded block by block. The block may be predictively encoded with reference to other (encoded) blocks determined by the coding assignment applied to the block's corresponding picture. For example, a block of an I picture may be non-predictively encoded, or a block of an I picture may be predictively encoded (spatial prediction or intra-frame prediction) with reference to an already encoded block of the same picture. A pixel block of a P picture may be non-predictively encoded via spatial prediction or via temporal prediction with reference to one previously encoded reference picture. A block of a B picture may be non-predictively encoded via spatial prediction or via temporal prediction with reference to one or two previously encoded reference pictures.

[0081] The video encoder (203) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T Rec. H.265. In its operation, the video encoder (203) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard being used.

[0082] In an embodiment, the transmitter (440) may transmit additional data along with the encoded video. The source encoder (430) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set segments, etc.

[0083] Before describing certain aspects of the disclosed subject matter in greater detail, some terminology needs to be introduced that will be referred to in the remainder of this description.

[0084] Sub-pictures hereafter refer in some cases to samples, blocks, macroblocks, coding units, or rectangular arrangements of similar entities that are semantically grouped and can be independently encoded at varying resolutions. One or more sub-pictures can be used for a picture. One or more coded sub-pictures can form a coded picture. One or more sub-pictures can be assembled into a picture, and one or more sub-pictures can be extracted from a picture. In some environments, one or more coded sub-pictures can be assembled into a coded picture in a compressed domain without transcoding to a sample level, and in the same or certain other cases, one or more coded sub-pictures can be extracted from a coded picture in a compressed domain.

[0085] Adaptive Resolution Change (ARC) hereinafter refers to the following mechanism: allows the resolution of pictures or sub-pictures within a coded video sequence to be changed by, for example, reference picture resampling. ARC parameters hereinafter refer to the control information required to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, resolution of output and / or reference pictures, various control flags, etc.

[0086] The above description focuses on encoding and decoding a single semantically independent coded video picture.Before describing the implications of encoding / decoding of multiple sub-pictures with independent ARC parameters and the additional complexity it implies, the options for signaling ARC parameters should be described.

[0087] Reference Figure 5, several novel options for signaling ARC parameters are shown. As noted for each option, they all have certain advantages and certain disadvantages from the perspective of coding efficiency, complexity, and architecture. A video coding standard or technology may select one or more of these options or options known from the prior art for signaling ARC parameters. These options may not be mutually exclusive, and it is conceivable that these options may be interchangeable based on application requirements, the standard technology involved, or the choice of encoder.

[0088] Categories of ARC parameters can include:

[0089] - Up / down sampling factors, separate or combined in X and Y dimensions

[0090] – Up / down sampling factor, adding a time dimension, indicating a constant speed of scaling up / down for a given number of pictures

[0091] - Either of the above may involve the encoding of one or more possibly short syntax elements, which may point to a table containing factors.

[0092] - The resolution of the input picture, output picture, reference picture, coded picture, combined or separated in the X or Y dimension in units of samples, blocks, macroblocks, CUs, or any other suitable granularity. If there is more than one resolution (e.g., one for the input picture and one for the reference picture), in some cases one set of values ​​can be inferred from the other. This can be gated, for example, using a flag. See below for a more detailed example.

[0093] - "warped" coordinates similar to those used in H.263 Annex P, also in units of a suitable granularity as described above. H.263 Annex P defines an efficient method for encoding such warped coordinates, but other conceivable, possibly more efficient methods may be devised. For example, the variable-length reversible "Huffman"-style encoding of the warped coordinates of Annex P may be replaced by a binary encoding of a suitable length, where the length of the binary codeword may be derived, for example, from the maximum picture size, possibly multiplied by a factor and offset by a value in order to allow "warping" beyond the bounds of the maximum picture size.

[0094] - Upsampling or downsampling filter parameters. In the simplest case, there may be only a single filter for upsampling and / or downsampling. However, in some cases, it may be advantageous to allow greater flexibility in the filter design, and this may require signaling the filter parameters. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g., by a list of filter coefficients, using a suitable entropy coding technique), the filter may be selected implicitly by the up / down sampling rate, which in turn may be signaled according to any of the mechanisms mentioned above, and so on.

[0095] Hereinafter, the description assumes encoding of a finite set of up / down sampling factors (the same factors used in both the X and Y dimensions) indicated by a codeword. The codeword may advantageously be variable length encoded, for example, using Ext-Golomb codes common to certain syntax elements in video coding specifications such as H.264 and H.265. For example, a suitable mapping of values ​​to up / down sampling factors may be according to the following table:

[0096] Codeword Ext-Golomb Code Original / target resolution 0 1 1 / 1 1 010 1 / 1.5 (increase by 50%) 2 011 1.5 / 1 (50% drop) 3 00100 1 / 2 (increase 100%) 4 00101 2 / 1 (down 100%)

[0097] Many similar mappings can be designed, depending on the needs of the application and the capabilities of the boost and drop mechanisms available in the video compression technology or standard. The table can be extended to more values. The values ​​can also be represented by entropy coding mechanisms other than Ext-Golomb codes, for example using a binary encoding. This may have certain advantages when the resampling factors are of interest outside the video processing engine (first of all the encoder and decoder) itself, such as a MANE. It should be noted that for the (probably) most common cases where no resolution change is required, a shorter Ext-Golomb code can be chosen; in the table above, only a single bit is chosen. For the most common cases, this has a coding efficiency advantage over using a binary code.

[0098] The number of entries in the table and their semantics may be fully or partially configurable. For example, the essentials of the table may be communicated in a "high" parameter set such as a sequence or decoder parameter set. Alternatively or additionally, one or more such tables may be defined in a video coding technique or standard, and one or more such tables may be selected by, for example, a decoder or sequence parameter set.

[0099] Hereafter, it is described how the upsampling / downsampling factors (ARC information) encoded as described above can be included in a video coding technique or standard syntax. Similar considerations can apply to one or several codewords controlling up / downsampling filters. See below for a discussion when filters or other data structures require relatively large amounts of data.

[0100] H.263 Annex P includes ARC information 502 in the form of four warp coordinates into the picture header 501, specifically in the H.263PLUSPTYPE (503) header extension. This may be a wise design choice when a) there is a picture header available and b) the ARC information is expected to change frequently. However, the overhead when using H.263 style signaling may be very high, and the scaling factor may not apply across picture boundaries because the picture header may have a transient nature.

[0101] The JVCET-M135-v1 referenced above includes ARC reference information (505) (index) located in a picture parameter set (504) and indexing a table (506) including a target resolution, which is in turn located in a sequence parameter set (507). According to the author's verbal statement, the placement of possible resolutions in the table (506) in the sequence parameter set (507) can be justified by using the SPS as an interoperability negotiation point during capability exchange. By referencing the appropriate picture parameter set (504), the resolution can vary from picture to picture within the limits set by the values ​​in the table (506).

[0102] Still refer to Figure 5 , there may be the following additional options to convey ARC information in the video bitstream. As described above, each of these options has certain advantages over the prior art. These options may exist simultaneously in the same video coding technology or standard.

[0103] In an embodiment, ARC information (509), such as a resampling (scaling) factor, may be present in a slice header, a GOB header, a tile header, or a tile group header (hereinafter referred to as a tile group header) (508). This may be sufficient for very small ARC information, such as a single variable length ue(v) or a fixed length codeword of several bits, such as shown above. Having the ARC information directly in the tile group header has the following additional advantage: the ARC information can be applied to, for example, a sub-picture represented by the tile group instead of the entire picture. See also below. In addition, even if the video compression technology or standard only envisions adaptive resolution changes for the entire picture (compared to, for example, adaptive resolution changes based on tile groups), from an error-tolerance perspective, placing the ARC information in the tile group header has certain advantages over placing the ARC information in the H.263 style picture header.

[0104] In the same or another embodiment, the ARC information (512) itself may be present in an appropriate parameter set (511), such as a picture parameter set, a header parameter set, a tile parameter set, an adaptation parameter set, etc. (depicted as an adaptation parameter set). The scope of the parameter set may advantageously be no larger than a picture, such as a tile group. The use of the ARC information is implicit through the activation of the relevant parameter set. For example, when a video coding technique or standard only considers picture-based ARC, then a picture parameter set or equivalent may be appropriate.

[0105] In the same or another embodiment, ARC reference information (513) may be present in a tile group header (514) or similar data structure. The reference information (513) may refer to a subset of ARC information (515) available in a parameter set (516) that extends beyond a single picture, such as a sequence parameter set or a decoder parameter set.

[0106] An additional level of indirect implicit activation of PPS, PPS, SPS from the tile group header as used in JVET-M0135-v1 seems unnecessary, since picture parameter sets can (and are in some standards such as RFC3984) be used for capability negotiation or announcement, just like sequence parameter sets. However, if the ARC information should apply, for example, to sub-pictures that are also represented by tile groups, parameter sets such as adaptive parameter sets or header parameter sets with an activation scope limited to tile groups may be a better choice. Furthermore, if the size of the ARC information exceeds a negligible size - for example containing filter control information such as many filter coefficients - then from a coding efficiency point of view, the parameters may be a better choice than using the header (508) directly, since the settings can be reused by future pictures or sub-pictures by referring to the same parameter set.

[0107] When using a sequence parameter set or another higher parameter set with a range spanning multiple pictures, certain considerations may apply:

[0108] 1. The parameter set (516) for storing the ARC information table may be a sequence parameter set in some cases, but is advantageously a decoder parameter set in other cases. The decoder parameter set may have an activation range of multiple CVSs, i.e., coded video streams, i.e., all coded video bits from the start of the session until the session is torn down. Such a range may be more appropriate because the possible ARC factors may be decoder features that can be implemented in hardware, and hardware features tend not to change with any CVS (which in at least some entertainment systems is a group of pictures with a length of one second or less). That is, placing the table in a sequence parameter set is explicitly included in the placement options described herein (especially in conjunction with point 2 below).

[0109] 2. The ARC reference information (513) can advantageously be placed directly in the picture / slice tile / GOB / tile group header (hereinafter the tile group header) (514) instead of being placed in the picture parameter set as in JVCET-M0135-v1 for the following reasons: When the encoder wants to change a single value in a picture parameter set, such as the ARC reference information, it must create a new PPS and reference the new PPS. Assume that only the ARC reference information changes, while other information, such as the quantization matrix information in the PPS remains unchanged. The size of such information may be large and will need to be retransmitted to make the new PPS complete. Since the ARC reference information (513) may be a single codeword, such as an index in a table and will be the only value that changes, it would be cumbersome and wasteful to retransmit all, such as the quantization matrix information. So far, from a coding efficiency perspective, it may be much better to avoid indirection through the PPS as proposed in JVET-M0135-v1. Similarly, placing the ARC reference information in the PPS has the following additional disadvantage: the ARC information referenced by the ARC reference information (513) must be applied to the entire picture rather than a sub-picture, because the scope of picture parameter set activation is the picture.

[0110] In the same or another embodiment, the signaling of ARC parameters may follow the following: Figure 6 A detailed example is outlined in . Figure 6 Depicted is a syntax diagram in a representation used in video coding standards since at least 1993. The notation of such a syntax diagram roughly follows C-style programming. Bold lines indicate syntax elements present in the bitstream, lines without bold typically indicate control flow or variable settings.

[0111] The tile group header (601), which is an exemplary syntax structure that may be used for a header for a (possibly rectangular) portion of a picture, may conditionally contain a variable length, Exp-Golomb coded syntax element dec_pic_size_idx (602) (depicted in bold). The presence of this syntax element in the tile group header may be gated according to the use of adaptive resolution (603) - here, the value of the flag is not depicted in bold, meaning that the flag is present in the bitstream when it appears in the syntax diagram. Whether adaptive resolution is used for the picture or portion thereof may be signaled in any high-level syntax structure, either internal or external to the bitstream. In the example shown, this is signaled in a sequence parameter set, as outlined below.

[0112] Still refer to Figure 6, an excerpt of a sequence parameter set (610) is also shown. The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, this flag may indicate the use of adaptive resolution, which in turn may require certain control information. In this example, such control information is conditionally present based on the value of the flag based on the parameter set (612) and the if() statement in the tile group header (601).

[0113] When adaptive resolution is used, in this example, the encoding is the output resolution in units of samples (613). Reference numeral 613 refers to both output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can define the resolution of the output picture. Elsewhere in the video coding technique or standard, certain restrictions on any value may be defined. For example, the level definition may limit the number of total output samples, which may be the product of the values ​​of these two syntax elements. In addition, certain video coding techniques or standards or external techniques or standards such as system standards may limit the range of values ​​(e.g., one or two dimensions must be divisible by a power of 2) or aspect ratio (e.g., the width and height must be in a 4:3 or 16:9 relationship). Such restrictions may be introduced to facilitate hardware implementation or for other reasons and are well known in the art.

[0114] In some applications, it is desirable for the encoder to instruct the decoder to use a certain reference picture size rather than implicitly assuming that size to be the output picture size. In this example, the syntax element reference_pic_size_present_flag (614) gates the conditional presence of the reference picture size (615) (again, this reference number refers to both width and height).

[0115] Finally, a table of possible decoded picture widths and heights is shown. For example, such a table may be represented by a table indication (num_dec_pic_size_in_luma_samples_minus1) (616). "Minus1" may refer to the interpretation of the value of this syntax element. For example, if the coded value is zero, there is one table entry. If the value is 5, there are six table entries. For each "row" in the table, the decoded picture width and height are then included in the syntax (617).

[0116] The presented table entry (617) may be indexed using the syntax element dec_pic_size_idx (602) in the tile group header, allowing each tile group to have a different decode size - effectively a scaling factor.

[0117] Some video coding techniques or standards such as VP9 support spatial scalability by implementing some form of reference picture resampling (signaled entirely differently from the disclosed subject matter) in conjunction with temporal scalability to achieve spatial scalability. In particular, ARC-style techniques may be used to cause certain reference pictures to be upsampled to a higher resolution to form the basis of a spatial enhancement layer. Those upsampled pictures may be improved using normal prediction mechanisms at a higher resolution to add detail.

[0118] The disclosed subject matter can be used in such an environment. In some cases, in the same embodiment or another embodiment, a value in a NAL unit header, such as a time ID field, can be used to indicate not only a temporal layer, but also a spatial layer. Doing so has certain advantages for certain system designs; for example, existing selected forwarding units (SFUs) created and optimized for temporal layers that are selected for forwarding based on NAL unit header time ID values ​​can be used in a scalable environment without modification. To achieve this, it may be necessary that the mapping between the coded picture size and the temporal layer be indicated by the time ID field in the NAL unit header.

[0119] In some video coding technologies, an access unit (AU) may refer to a coded picture, slice, tile, NAL unit, etc., which is captured and synthesized into a corresponding picture / slice / tile / NAL unit bitstream at a given time. The time may be the synthesis time.

[0120] In HEVC and certain other video coding techniques, a picture order count (POC) value can be used to indicate a reference picture selected from multiple reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) includes one or more pictures, slices or tiles, each picture, slice or tile belonging to the same AU can carry the same POC value, from which it can be derived that they are created from content at the same synthesis time. In other words, in a scenario where two pictures / slices / tiles carry the same given POC value, this can indicate two pictures / slices / tiles belonging to the same AU and having the same synthesis time. In contrast, two pictures / slices / tiles with different POC values ​​can indicate those pictures / slices / tiles belonging to different AUs and having different synthesis times.

[0121] In embodiments of the disclosed subject matter, the aforementioned strict relationship may be relaxed because an access unit may include pictures, slices, or tiles with different POC values. By allowing different POC values ​​within an AU, the POC values ​​may be used to identify potentially independently decodable pictures / slices / tiles with the same presentation time. This in turn may enable support for multiple scalable layers without changing reference picture selection signaling (e.g., reference picture set signaling or reference picture list signaling), as described in more detail below.

[0122] However, it is still desirable to be able to identify the AU to which a picture / slice / tile belongs based solely on the POC value relative to other pictures / slices / tiles having different POC values. This can be achieved as described below.

[0123] In the same or other embodiments, an access unit count (AUC) may be signaled in a high-level syntax structure such as a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter. The value of the AUC may be used to identify which NAL units, pictures, slices, or tiles belong to a given AU. The value of the AUC may correspond to different synthesis moments. The AUC value may be equal to a multiple of the POC value. The AUC value may be calculated by dividing the POC value by an integer value. In some cases, the division operation may impose a certain burden on the decoder implementation. In such a case, a small restriction in the numerical space of the AUC value may allow a shift operation to be used instead of a division operation. For example, the AUC value may be equal to the most significant bit (MSB) value of the POC value range.

[0124] In the same embodiment, the value of the POC cycle (poc_cycle_au) for each AU may be signaled in a high-level syntax structure such as a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter. poc_cycle_au may indicate how many different and consecutive POC values ​​may be associated with the same AU. For example, if the value of poc_cycle_au is equal to 4, pictures, slices, or tiles with POC values ​​equal to 0-3 (including endpoints) are associated with AUs with AUC values ​​equal to 0, and pictures, slices, or tiles with POC values ​​equal to 4-7 (including endpoints) are associated with AUs with AUC values ​​equal to 1. Therefore, the value of AUC may be inferred by dividing the POC value by the value of poc_cycle_au.

[0125] In the same or another embodiment, the value of poc_cycle_au can be obtained from information located in, for example, a video parameter set (VPS) that identifies the number of spatial or SNR layers in the coded video sequence. Such a possible relationship is briefly described below. Although the derivation described above can save some bits in the VPS and can therefore improve coding efficiency, it may be advantageous to explicitly encode poc_cycle_au in an appropriate high-level syntax structure hierarchically below the video parameter set so that poc_cycle_au can be minimized for a given small portion of the bitstream, such as a picture. Since the POC value (and / or the value of a syntax element that indirectly references the POC) can be encoded in a low-level syntax structure, this optimization can save more bits than can be saved by the above-described derivation process.

[0126] The above-described techniques for signaling adaptive resolution parameters may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Figure 7 A computer system 700 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0127] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly by a computer central processing unit (CPU), a graphics processing unit (GPU), or the like, or by interpretation, microcode execution, or the like.

[0128] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0129] Figure 7 The components shown in the computer system 700 are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Neither should the configuration of components be interpreted as having any dependency or requirement related to any one component or combination of components shown in the exemplary embodiment of the computer system 700.

[0130] Computer system 700 may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input from one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, for example, audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0131] The input human interface devices may include one or more of the following (only one of each is depicted): keyboard 701 , mouse 702 , trackpad 703 , touch screen, data glove 704 , joystick 705 , microphone 706 , scanner 707 , camera 708 .

[0132] The computer system 700 may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen, data gloves 704, or joystick 705, but there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers 709, headphones (not depicted)), visual output devices (e.g., screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may be able to output two-dimensional visual outputs or more than three-dimensional outputs through means such as stereoscopic image output; virtual reality glasses (not depicted), holographic displays, and cigarette cans (not depicted)) and printers (not depicted).

[0133] The computer system 700 may also include human-accessible storage devices and their associated media such as optical media, including CD / DVD ROM / RW 720 with CD / DVD or similar media 721, thumb drive 722, removable hard drive or solid state drive 723, traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security software dogs (not depicted), etc.

[0134] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0135] The computer system 700 may also include an interface to one or more communication networks. The network may be, for example, wireless, wired, optical. The network may also be local, wide, metropolitan, vehicle-mounted and industrial, real-time, delay-tolerant, etc. Examples of networks include: local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired connections or wireless wide-area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicles and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter that is attached to some universal data ports or peripheral buses (749) (e.g., a USB port of the computer system 700); other networks are typically integrated into the kernel of the computer system 700 by attaching to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smart phone computer system). Using any of these networks, the computer system 700 can communicate with other entities. Such communications can be one-way receive-only (e.g., broadcast television), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way, such as to other computer systems using local or wide area digital networks. Certain protocols and protocol stacks may be used on each of these networks and the above-mentioned network interfaces.

[0136] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the kernel 740 of the computer system 700 .

[0137] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, dedicated programmable processing units 743 in the form of field programmable gate areas (FPGAs), hardware accelerators 744 for specific tasks, etc. These devices, along with read-only memory (ROM) 745, random access memory 746, and internal mass storage devices (e.g., internal non-user accessible hard drives, SSDs, etc.) 747, may be connected via a system bus 748. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to enable expansion of additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus 748 or may be attached to the system bus 748 via a peripheral bus 749. The architecture of the peripheral bus includes PCI, USB, etc.

[0138] The CPU 741, GPU 742, FPGA 743, and accelerator 744 may execute certain instructions, which in combination may form the aforementioned computer code. The computer code may be stored in ROM 745 or RAM 746. Transient data may also be stored in RAM 746, while permanent data may be stored in, for example, an internal mass storage device 747. Fast storage and retrieval of any memory device may be achieved by using a cache memory that may be closely associated with one or more CPUs 741, GPUs 742, mass storage devices 747, ROM 745, RAM 746, etc.

[0139] The computer readable medium may have thereon computer code for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0140] As an example and not limitation, a computer system having architecture 700 and specifically having kernel 740 can provide functions as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be a medium associated with a user-accessible mass storage device as introduced above and certain storage devices of the kernel 740 as a non-transient nature, such as a kernel internal mass storage device 747 or ROM 745. Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the kernel 740. Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the kernel 740 and specifically the processor therein (including CPU, GPU, FPGA, etc.) to perform a specific process described herein or a specific part of a specific process, including defining a data structure stored in RAM 746 and modifying such a data structure according to a process defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise contained in a circuit (e.g., accelerator 744) that may run in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may encompass circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits containing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0141] Figure 8An example of a video sequence structure with a combination of temporal_id, layer_id, POC and AUC values ​​with adaptive resolution change is shown. In this example, a picture, slice or tile in the first AU with AUC=0 may have temporal_id=0 and layer_id=0 or 1, while a picture, slice or tile in the second AU with AUC=1 may have temporal_id=1 and layer_id=0 or 1, respectively. Regardless of the values ​​of temporal_id and layer_id, the value of POC increases by 1 for each picture. In this example, the value of poc_cycle_au may be equal to 2. Preferably, the value of poc_cycle_au may be set equal to the number of layers (spatial scalability). Therefore, in this example, the value of POC increases by 2, while the value of AUC increases by 1.

[0142] In the above-mentioned embodiments, all or a subset of the inter-picture or inter-layer prediction structure and reference picture indication can be supported by using the existing reference picture set (RPS) signaling or reference picture list (RPL) signaling in HEVC. In the RPS or RPL, the selected reference picture is indicated by signaling the POC value or the incremental value of the POC between the current picture and the selected reference picture. For the disclosed subject matter, RPS and RPL can be used to indicate the inter-picture or inter-layer prediction structure without changing the signaling, but with the following restrictions. If the value of the temporal_id of the reference picture is greater than the value of the temporal_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction. If the value of the layer_id of the reference picture is greater than the value of the layer_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction.

[0143] In the same embodiment and other embodiments, motion vector scaling based on POC differences for temporal motion vector prediction may be disabled across multiple pictures within an access unit. Thus, although each picture may have a different POC value within an access unit, the motion vector is not scaled and used for temporal motion vector prediction within an access unit. This is because reference pictures with different POCs in the same AU are considered to be reference pictures with the same time instant. Therefore, in this embodiment, the motion vector scaling function may return 1 when the reference picture belongs to the AU associated with the current picture.

[0144] In the same embodiment and other embodiments, when the spatial resolution of the reference picture is different from the spatial resolution of the current picture, motion vector scaling based on POC difference for temporal motion vector prediction can be optionally disabled across multiple pictures. When motion vector scaling is allowed, the motion vector is scaled based on the POC difference and the spatial resolution ratio between the current picture and the reference picture.

[0145] In the same or another embodiment, in particular when poc_cycle_au has a non-uniform value (when vps_contant_poc_cycle_per_au==0), the motion vector can be scaled based on the AUC difference instead of the POC difference for temporal motion vector prediction. Otherwise (when vps_contant_poc_cycle_per_au==1), the motion vector scaling based on the AUC difference can be the same as the motion vector scaling based on the POC difference.

[0146] In the same or another embodiment, when motion vectors are scaled based on AUC difference, reference motion vectors in the same AU (with the same AUC value) as the current picture are not scaled based on the AUC difference and are used for motion vector prediction without scaling or with scaling based on the spatial resolution ratio between the current picture and the reference picture.

[0147] In the same embodiment and other embodiments, the AUC value is used to identify the boundary of the AU and is used for hypothetical reference decoder (HRD) operation, which requires input and output timing at the AU granularity. In most cases, the decoded picture of the highest layer in the AU can be output for display. The AUC value and the layer_id value can be used to identify the output picture.

[0148] In an embodiment, a picture may be composed of one or more sub-pictures. Each sub-picture may cover a partial area or the entire area of ​​the picture. The area supported by a sub-picture may overlap or not overlap with the area supported by another sub-picture. The area composed of one or more sub-pictures may or may not cover the entire area of ​​the picture. If a picture is composed of sub-pictures, the area supported by the sub-picture is the same as the area supported by the picture.

[0149] In the same embodiment, a sub-picture may be encoded by an encoding method similar to that used to encode a picture. A sub-picture may be encoded independently or may be encoded based on another sub-picture or an encoded picture. A sub-picture may or may not have any analytical dependency on another sub-picture or an encoded picture.

[0150] In the same embodiment, the coded sub-picture may be included in one or more layers. The coded sub-pictures in the layers may have different spatial resolutions. The original sub-picture may be spatially resampled (upsampled or downsampled), encoded using different spatial resolution parameters, and included in the bitstream corresponding to the layer.

[0151] In the same or another embodiment, a sub-picture with (W, H) may be encoded and included in a coded bitstream corresponding to layer 0, where W indicates the width of the sub-picture and H indicates the height of the sub-picture, while a sub-picture with (W*S*H) from the sub-picture with the original spatial resolution may be encoded and included in a coded bitstream corresponding to layer 0, where W indicates the width of the sub-picture and H indicates the height of the sub-picture. w,k , H*S h,k ) and includes it in the coded bitstream corresponding to the kth layer, where S w,k , S h,k Indicates the horizontal resampling rate and the vertical resampling rate. If S w,k , S h,k If the value of is greater than 1, resampling is equal to upsampling. w,k , S h,k If the value of is less than 1, resampling is equal to downsampling.

[0152] In the same or another embodiment, a coded sub-picture in a layer may have a different visual quality than a coded sub-picture in another layer in the same sub-picture or in a different sub-picture. i,n Encode the sub-image i in the nth layer, and use the quantization parameter Q j,m Encode sub-picture j in layer m.

[0153] In the same or another embodiment, a coded sub-picture in a layer may be independently decodable without any parsing or decoding dependency with a coded sub-picture in another layer of the same local region. A sub-picture layer that can be independently decoded without reference to another sub-picture layer of the same local region is an independent sub-picture layer. A coded sub-picture in an independent sub-picture layer may or may not have a decoding or parsing dependency with a previously coded sub-picture in the same sub-picture layer, but the coded sub-picture may not have any dependency with a coded picture in another sub-picture layer.

[0154] In the same or another embodiment, a coded sub-picture in a layer may be dependency decodable, having any parsing or decoding dependency with a coded sub-picture in another layer of the same local region. A sub-picture layer that can be dependency decoded by referencing another sub-picture layer of the same local region is a dependent sub-picture layer. A coded sub-picture in a dependent sub-picture may reference a coded sub-picture belonging to the same sub-picture, a previously coded sub-picture in the same sub-picture layer, or both reference sub-pictures.

[0155] In the same or another embodiment, a coded sub-picture is composed of one or more independent sub-picture layers and one or more dependent sub-picture layers. However, for a coded sub-picture, there may be at least one independent sub-picture layer. An independent sub-picture layer may have a value of a layer identifier (layer_id) that may be present in a NAL unit header or other high-level syntax structure equal to 0. A sub-picture layer with layer_id equal to 0 is a base sub-picture layer.

[0156] In the same or another embodiment, a picture may be composed of one or more foreground sub-pictures and one background sub-picture. The area supported by the background sub-picture may be equal to the area of ​​the picture. The area supported by the foreground sub-picture may overlap with the area supported by the background sub-picture. The background sub-picture may be a base sub-picture layer and the foreground sub-picture may be a non-base (enhanced) sub-picture layer. One or more non-base sub-picture layers may reference the same base layer for decoding. Each non-base sub-picture layer with layer_id equal to a may reference a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0157] In the same or another embodiment, a picture may consist of one or more foreground sub-pictures, with or without background sub-pictures. Each sub-picture may have its own base sub-picture layer and one or more non-base (enhancement) layers. Each base sub-picture layer may be referenced by one or more non-base sub-picture layers. Each non-base sub-picture layer with layer_id equal to a may reference a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0158] In the same or another embodiment, a picture may be composed of one or more foreground sub-pictures, with or without background sub-pictures. Each coded sub-picture in a (base or non-base) sub-picture layer may be referenced by one or more non-base layer sub-pictures belonging to the same sub-picture and one or more non-base layer sub-pictures not belonging to the same sub-picture.

[0159] In the same or another embodiment, a picture may be composed of one or more foreground sub-pictures, with or without background sub-pictures. A sub-picture in layer a may be further divided into multiple sub-pictures in the same layer. One or more coded sub-pictures in layer b may reference a divided sub-picture in layer a.

[0160] In the same or another embodiment, a coded video sequence (CVS) may be a set of coded pictures. A CVS may be composed of one or more coded sub-picture sequences (CSPS), where a CSPS may be a set of coded sub-pictures covering the same local area of ​​a picture. A CSPS may have a temporal resolution that is the same as or different from the temporal resolution of the coded video sequence.

[0161] In the same embodiment or another embodiment, the CSPS may be encoded and included in one or more layers. The CSPS may be composed of one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS may reconstruct a sub-picture sequence corresponding to the same local area.

[0162] In the same embodiment or another embodiment, the number of CSPS layers corresponding to a CSPS may be the same as or different from the number of CSPS layers corresponding to another CSPS.

[0163] In the same or another embodiment, a CSPS layer may have a different temporal resolution (e.g., frame rate) than another CSPS layer. The original (uncompressed) sub-picture sequence may be temporally resampled (upsampled or downsampled), encoded with a different temporal resolution parameter, and included in the bitstream corresponding to the layer.

[0164] In the same or another embodiment, a sub-picture sequence having a frame rate F may be encoded and included in the coded bitstream corresponding to layer 0, while a sub-picture sequence having a frame rate F*S may be encoded and included in the coded bitstream corresponding to layer 0. t,k A temporally upsampled (or downsampled) sub-picture sequence from the original sub-picture sequence is encoded and included in the coded bitstream corresponding to layer k, where S t,k indicates the temporal sampling rate of layer k. If S t,k If the value of S is greater than 1, the temporal resampling process is equivalent to frame rate upconversion. t,k If the value of is less than 1, the temporal resampling process is equivalent to frame rate down-conversion.

[0165] In the same embodiment or another embodiment, when a sub-picture with CSPS layer a is referenced by a sub-picture with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b, the decoded pixels in CSPS layer a are resampled and used for reference. The resampling process may require upsampling filtering or downsampling filtering.

[0166] In the same embodiment or another embodiment, Fig. 9 An example of a syntax table for signaling a syntax element of vps_poc_cycle_au in a VPS (or SPS) and a syntax element of slice_poc_cycle_au in a slice header is shown, vps_poc_cycle_au indicating the poc_cycle_au used for encoding all pictures / slices in a video sequence, and slice_poc_cycle_au indicating the poc_cycle_au of the current slice. If the POC value increases uniformly for each AU, vps_contant_poc_cycle_per_au in the VPS is set equal to 1, and vps_poc_cycle_au is signaled in the VPS. In this case, slice_poc_cycle_au is not explicitly signaled, and the AUC value for each AU is calculated by dividing the POC value by vps_poc_cycle_au. If the POC value does not increase uniformly for each AU, vps_contant_poc_cycle_per_au in the VPS is set equal to 0. In this case, vps_access_unit_cnt is not signaled, but slice_access_unit_cnt is signaled in the slice header of each slice or picture. Each slice or picture may have a different slice_access_unit_cnt value. The AUC value for each AU is calculated by dividing the POC value by slice_poc_cycle_au. Fig.10 A block diagram illustrating the relevant workflow is shown.

[0167] In the same embodiment or other embodiments, even if the POC values ​​of the pictures, slices or tiles may be different, the pictures, slices or tiles corresponding to the AU with the same AUC value may be associated with the same decoding or output time. Therefore, there is no internal parsing / decoding dependency between the pictures, slices or tiles in the same AU, all or a subset of the pictures, slices or tiles associated with the same AU may be decoded in parallel, and all or a subset of the pictures, slices or tiles associated with the same AU may be output at the same time.

[0168] In the same or other embodiments, even if the POC values of pictures, slices, or tiles may be different, pictures, slices, or tiles corresponding to an AU with the same AUC value can be associated with the same synthesis / display time. When the synthesis time is included in the container format, even if pictures correspond to different AUs, if the pictures have the same synthesis time, they can be displayed at the same time.

[0169] In the same or other embodiments, each picture, slice, or tile can have the same temporal identifier (temporal_id) within the same AU. All or a subset of pictures, slices, or tiles corresponding to a time instance can be associated with the same temporal sublayer. In the same or other embodiments, each picture, slice, or tile can have the same or different spatial layer id (layer_id) within the same AU. All or a subset of pictures, slices, or tiles corresponding to a time instance can be associated with the same or different spatial layers.

[0170] Fig.11 An example video stream is shown, which includes a background video CSPS with layer_id equal to 0 and multiple foreground CSPS layers. Although an encoded subpicture can be composed of one or more CSPS layers, the background region that does not belong to any foreground CSPS layer can be composed of a base layer. The base layer can contain both the background region and the foreground region, while the enhanced CSPS layer contains the foreground region. At the same region, the enhanced CSPS layer can have better visual quality than the base layer. The enhanced CSPS layer can refer to the reconstructed pixels and motion vectors of the base layer corresponding to the same region.

[0171] In the same or another embodiment, in a video file, the video bitstream corresponding to the base layer is included in a track, while the CSPS layer corresponding to each subpicture is included in a separate track.

[0172] In the same or another embodiment, the video bitstream corresponding to the base layer is included in a track, while the CSPS layers with the same layer_id are included in separate tracks. In this example, the track corresponding to layer k only includes the CSPS layers corresponding to layer k.

[0173] In the same or another embodiment, each CSPS layer of each subpicture is stored in a separate track. Each track may or may not have any parsing or decoding correlation with one or more other tracks.

[0174] In the same or another embodiment, each track can contain the bitstreams corresponding to layer i to layer j of the CSPS layers of all subpictures or a subset of subpictures, where 0 < i =< j =< k, and k is the highest layer of CSPS.

[0175] In the same or another embodiment, a picture is composed of one or more associated media data, including a depth map, an alpha map, 3D geometry data, an occupancy map, etc. Such associated timed media data may be divided into one or more data substreams, each of the one or more data substreams corresponding to a sub-picture.

[0176] In the same embodiment or another embodiment, Fig.12 An example of a video conference based on a multi-layer sub-picture method is shown. In the video stream, a base layer video bitstream corresponding to a background picture and one or more enhancement layer video bitstreams corresponding to a foreground sub-picture are included. Each enhancement layer video bitstream corresponds to a CSPS layer. In the display, the picture corresponding to the base layer is displayed by default. It contains a picture-in-picture (PIP) of one or more users. When a specific user is selected through a client control, the enhanced CSPS layer corresponding to the selected user is decoded and displayed with enhanced quality or spatial resolution. Fig.13 An operational diagram is shown.

[0177] In the same or another embodiment, a network middlebox (e.g., a router) can select a subset of layers to send to a user based on its bandwidth. Picture / sub-picture organization can be used for bandwidth adaptation. For example, if a user has no bandwidth, the router strips layers or selects some sub-pictures due to their importance or based on usage settings, and this can be done dynamically to adapt to bandwidth.

[0178] Fig.14 The use of 360-degree video is shown. When a spherical 360-degree picture is projected onto a flat picture, the projected 360-degree picture can be divided into multiple sub-pictures as a base layer. The enhancement layer of a specific sub-picture can be encoded and transmitted to the client. The decoder can decode both the base layer including all sub-pictures and the enhancement layer of the selected sub-picture. When the current viewport is the same as the selected sub-picture, the displayed picture can have a higher quality using the decoded sub-picture with the enhancement layer. Otherwise, the decoded picture with the base layer can be displayed at a low quality.

[0179] In the same or another embodiment, any layout information for display may be present in the file as supplemental information (e.g., SEI messages or metadata). One or more decoded sub-pictures may be repositioned and displayed based on the signaled layout information. The layout information may be signaled by the streaming server or broadcaster, or may be regenerated by a network entity or cloud server, or may be determined by user-defined settings.

[0180] In an embodiment, when the input picture is divided into one or more (rectangular) sub-regions, each sub-region can be encoded as an independent layer. Each independent layer corresponding to a local area can have a unique layer_id value. For each independent layer, the sub-picture size and position information can be sent by signal. For example, the picture size (width, height), the offset information of the upper left corner (x_offset, y_offset). Fig.15 An example of the layout of the divided sub-pictures, their sub-picture size and position information and their corresponding picture prediction structure is shown. The layout information including sub-picture size and sub-picture position can be signaled in a high-level syntax structure such as a parameter set, a header of a slice or a tile group, or an SEI message.

[0181] In the same embodiment, each sub-picture corresponding to an independent layer may have its unique POC value within the AU. When indicating a reference picture among pictures stored in the DPB by using a syntax element in an RPS or RPL structure, the POC value of each sub-picture corresponding to a layer may be used.

[0182] In the same or another embodiment, to indicate the (inter-layer) prediction structure, instead of using layer_id, a POC (delta) value may be used.

[0183] In the same embodiment, a sub-picture with a POC value equal to N corresponding to a layer (or local area) may or may not be used as a reference picture for a sub-picture with a POC value equal to N+K corresponding to the same layer (or the same local area) for motion compensated prediction. In most cases, the value of the number K may be equal to the maximum number of (independent) layers, which may be the same as the number of sub-regions.

[0184] In the same embodiment or another embodiment, Fig.16 Shows Fig.15 When the input picture is divided into multiple (e.g., four) sub-regions, each local region may be encoded with one or more layers. In this case, the number of independent layers may be equal to the number of sub-regions, and one or more layers may correspond to the sub-regions. Therefore, each sub-region may be encoded with one or more independent layers and zero or more dependency layers.

[0185] In the same embodiment, Fig.16In , the input picture can be divided into four sub-regions. The upper right sub-region can be encoded into two layers, namely layer 1 and layer 4, and the lower right sub-region can be encoded into two layers, namely layer 3 and layer 5. In this case, layer 4 can refer to layer 1 for motion compensation prediction, and layer 5 can refer to layer 3 for motion compensation.

[0186] In the same or another embodiment, loop filtering (e.g., deblocking filtering, adaptive loop filtering, shaper, bilateral filtering, or any deep learning based filtering) across layer boundaries may (optionally) be disabled.

[0187] In the same or another embodiment, motion compensated prediction or intra-block copy across layer boundaries may (optionally) be disabled.

[0188] In the same or another embodiment, boundary filling for motion compensated prediction or loop filtering at the boundaries of sub-pictures may be optionally processed. A flag indicating whether boundary filling is processed may be signaled in a high-level syntax structure such as a parameter set (VPS, SPS, PPS, or APS), a slice or tile group header, or an SEI message.

[0189] In the same or another embodiment, the layout information of the sub-regions (or sub-pictures) may be signaled in the VPS or SPS. Fig.17 Examples of syntax elements in VPS and SPS are shown. In this example, vps_sub_picture_dividing_flag is signaled in the VPS. This flag may indicate whether the input picture is divided into multiple sub-regions. When the value of vps_sub_picture_dividing_flag is equal to 0, the input picture in the coded video sequence corresponding to the current VPS may not be divided into multiple sub-regions. In this case, the input picture size may be equal to the coded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples), which is signaled in the SPS. When the value of vps_sub_picture_dividing_flag is equal to 1, the input picture may be divided into multiple sub-regions. In this case, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled in the VPS. The values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may be equal to the width and height of the input picture, respectively.

[0190] In the same embodiment, the values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may not be used for decoding but may be used for synthesis and display.

[0191] In the same embodiment, when the value of vps_sub_picture_dividing_flag is equal to 1, the syntax elements pic_offset_x and pic_offset_y may be signaled in the SPS corresponding to the specific layer. In this case, the coded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to the specific layer. In addition, the position of the upper left corner of the sub-region (pic_offset_x, pic_offset_y) may be signaled in the SPS.

[0192] In the same embodiment, the position information (pic_offset_x, pic_offset_y) of the upper left corner of the sub-region may not be used for decoding but may be used for synthesis and display.

[0193] In the same embodiment or another embodiment, layout information (size and position) of all sub-regions or a subset of sub-regions of an input picture, dependency information between layers may be signaled in a parameter set or SEI message. Fig.18An example of a syntax element for indicating information about the layout of sub-regions, dependencies between layers, and relationships between sub-regions and one or more layers is shown. In this example, the syntax element num_sub_region indicates the number of (rectangular) sub-regions in the current coded video sequence. The syntax element num_layers indicates the number of layers in the current coded video sequence. The value of num_layers may be equal to or greater than the value of num_sub_region. When any sub-region is coded as a single layer, the value of num_layers may be equal to the value of num_sub_region. When one or more sub-regions are coded as multiple layers, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] indicates a dependency from the jth layer to the ith layer. num_layers_for_region[i] indicates the number of layers associated with the i-th sub-region. sub_region_layer_id[i][j] indicates the layer_id of the jth layer associated with the i-th sub-region. sub_region_offset_x[i] and sub_region_offset_y[i] indicate the horizontal position and vertical position of the upper left corner of the ith sub-region, respectively. sub_region_width[i] and sub_region_height[i] indicate the width and height of the ith sub-region, respectively.

[0194] In one embodiment, one or more syntax elements may be signaled in a high-level syntax structure such as a VPS, DPS, SPS, PPS, APS, or SEI message that specifies an output layer set indicating one or more layers to be output with or without profile level information. Fig.19 The syntax element num_output_layer_sets indicating the number of output layer sets (OLS) in the coded video sequence may be signaled in the VPS. For each output layer set, as many output_layer_flags as the number of output layers may be signaled.

[0195] In the same embodiment, output_layer_flag[i] equal to 1 specifies that the i-th layer is output. vps_output_layer_flag[i] equal to 0 specifies that the i-th layer is not output.

[0196] In the same or another embodiment, one or more syntax elements specifying profile level information for each output layer set may be signaled in a high-level syntax structure such as a VPS, DPS, SPS, PPS, APS, or SEI message. Fig.19 The syntax element num_profile_tile_level indicating the number of profile level information for each OLS in the coded video sequence may be signaled in the VPS with reference to the VPS. For each output layer set, a set of syntax elements for profile level information as many as the number of output layers may be signaled or an index indicating a specific profile level information among the entries in the profile level information.

[0197] In the same embodiment, profile_tier_level_idx[i][j] specifies the index of the profile_tier_level() syntax structure of the jth layer in the list of profile_tier_level() syntax structures in the VPS that applies to the i-th OLS.

[0198] In the same embodiment or another embodiment, reference is made to Fig. 20 , when the maximum number of layers is greater than 1 (vps_max_layers_minus1>0), the syntax elements num_profile_tile_level and / or num_output_layer_sets may be signaled.

[0199] In the same embodiment or another embodiment, reference is made to Fig. 20 , the syntax element vps_output_layers_mode[i] indicating the mode of output layer signaling of the i-th output layer set may be present in the VPS.

[0200] In the same embodiment, vps_output_layers_mode[i] equal to 0 specifies that only the topmost layer of the i-th output layer set is output. vps_output_layer_mode[i] equal to 1 specifies that all layers of the i-th output layer set are output. vps_output_layer_mode[i] equal to 2 specifies that the layers output are the layers of the i-th output layer set for which vps_output_layer_flag[i][j] is equal to 1. More values ​​may be reserved.

[0201] In the same embodiment, output_layer_flag[i][j] may or may not be signaled depending on the value of vps_output_layers_mode[i] for the i-th output layer set.

[0202] In the same embodiment or another embodiment, reference is made to Fig. 20 , a flag vps_ptl_signal_flag[i] may be present for the i-th output layer set. Depending on the value of vps_ptl_signal_flag[i], the profile level information of the i-th output layer set may or may not be signaled.

[0203] In the same embodiment or another embodiment, reference is made to Fig.21 , the number of sub-pictures in the current CVS, max_subpics_minus1, can be signaled in a high-level syntax structure such as VPS, DPS, SPS, PPS, APS or SEI message.

[0204] In the same embodiment, reference Fig.21 , when the number of sub-pictures is greater than 1 (max_subpics_minus1>0), the sub-picture identifier sub_pic_id[i] of the i-th sub-picture may be sent by a signal.

[0205] In the same or another embodiment, one or more syntax elements indicating sub-picture identifiers for each layer belonging to each output layer set may be signaled in the VPS. Fig. 22 , sub_pic_id_layer[i][j][k] indicates the kth sub-picture present in the jth layer of the i-th output layer set. With this information, the decoder can identify which sub-picture can be decoded and output for each layer of a specific output layer set.

[0206] In an embodiment, a picture header (PH) is a syntax structure containing syntax elements that apply to all slices of a coded picture. A picture unit (PU) is a set of NAL units that are associated with each other according to a specified classification rule, are continuous in decoding order, and contain exactly one coded picture. A PU may contain a picture header (PH) and one or more video coding layer (VCL) NAL units that constitute a coded picture.

[0207] In an embodiment, the SPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with TemporalId equal to 0, or provided by external means.

[0208] In an embodiment, the SPS (RBSP) is available to the decoding process before it is referenced, included in at least one AU with TemporalId equal to 0 in the CVS containing one or more PPSs referencing the SPS, or provided by external means.

[0209] In an embodiment, the SPS (RBSP) is made available to the decoding process before it is referenced by one or more PPSs, is included in at least one PU with a nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL unit that references the SPS NAL unit in the CVS, which contains one or more PPSs that reference the SPS, or the SPS (RBSP) is provided by external means.

[0210] In an embodiment, the SPS (RBSP) is available to the decoding process before it is referenced by one or more PPSs, is included in at least one PU with TemporalId equal to 0 and nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL units referencing the SPS NAL unit, or is provided by external means.

[0211] In an embodiment, the SPS (RBSP) is available to the decoding process before it is referenced by one or more PPSs, is included in at least one PU with TemporalId equal to 0 and nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL unit that references the SPS NAL unit in the CVS, which CVS contains one or more PPSs that reference the SPS, or the SPS (RBSP) is provided by external means.

[0212] In the same or another embodiment, pps_seq_parameter_set_id specifies the value of sps_seq_parameter_set_id for the referenced SPS. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0213] In the same or another embodiment, all SPS NAL units in a CVS with a particular sps_seq_parameter_set_id value may have the same content.

[0214] In the same or another embodiment, SPS NAL units may share the same value space of sps_seq_parameter_set_id regardless of the nuh_layer_id value.

[0215] In the same or another embodiment, the nuh_layer_id value of the SPS NAL unit may be equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL unit.

[0216] In an embodiment, when an SPS with nuh_layer_id equal to m is referenced by one or more PPSs with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0217] In an embodiment, the PPS (RBSP) should be available to the decoding process before it is referenced, be included in at least one AU with TemporalId equal to the TemporalId of the PPS NAL unit, or be provided by external means.

[0218] In an embodiment, the PPS (RBSP) should be available to the decoding process before it is referenced, be included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS containing one or more PHs (or coded slice NAL units) that reference the PPS, or the PPS (RBSP) is provided by external means.

[0219] In an embodiment, the PPS (RBSP) shall be available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), be included in at least one PU with the lowest nuh_layer_id value of the coded slice NAL units whose nuh_layer_id is equal to the PPS NAL unit in the referenced CVS, which contains one or more PHs (or coded slice NAL units) that reference the PPS, or the PPS (RBSP) is provided by external means.

[0220] In an embodiment, the PPS (RBSP) shall be available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), be included in at least one PU with TemporalId equal to the TemporalId of the PPS NAL unit and nuh_layer_id equal to the lowest nuh_layer_id value of the coded slice NAL unit that references the PPS NAL unit in the CVS that contains one or more PHs (or coded slice NAL units) that reference the PPS, or the PPS (RBSP) is provided by external means.

[0221] In the same or another embodiment, ph_pic_parameter_set_id in the PH specifies the value of pps_pic_parameter_set_id for the referenced PPS used. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0222] In the same or another embodiment, all PPS NAL units within a PU with a specific pps_pic_parameter_set_id value shall have the same content.

[0223] In the same or another embodiment, PPS NAL units may share the same value space of pps_pic_parameter_set_id regardless of the nuh_layer_id value.

[0224] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL unit,

[0225] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n. The layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0226] In an embodiment, the PPS (RBSP) should be available to the decoding process before it is referenced, be included in at least one AU with TemporalId equal to the TemporalId of the PPS NAL unit, or be provided by external means.

[0227] In an embodiment, the PPS (RBSP) should be available to the decoding process before it is referenced, be included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS containing one or more PHs (or coded slice NAL units) referencing the PPS, or the PPS (RBSP) is provided by external means.

[0228] In an embodiment, the PPS (RBSP) shall be available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), be included in at least one PU with the lowest nuh_layer_id value of the coded slice NAL units whose nuh_layer_id is equal to the PPS NAL unit in the referenced CVS, which contains one or more PHs (or coded slice NAL units) that reference the PPS, or the PPS (RBSP) is provided by external means.

[0229] In an embodiment, the PPS (RBSP) shall be available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), included in at least one PU with TemporalId equal to the TemporalId of the PPS NAL unit and nuh_layer_id equal to the lowest nuh_layer_id value of the coded slice NAL unit that references the PPS NAL unit in the CVS containing one or more PHs (or coded slice NAL units) that reference the PPS, or the PPS (RBSP) shall be provided by external means.

[0230] In the same or another embodiment, ph_pic_parameter_set_id in the PH specifies the value of pps_pic_parameter_set_id for the referenced PPS used. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0231] In the same or another embodiment, all PPS NAL units within a PU with a specific pps_pic_parameter_set_id value shall have the same content.

[0232] In the same or another embodiment, PPS NAL units may share the same value space of pps_pic_parameter_set_id regardless of the nuh_layer_id value.

[0233] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL unit.

[0234] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n. The layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0235] In an embodiment, if Fig. 22 As shown, pps_subpic_id[i] in the picture parameter set specifies the sub-picture ID of the i-th sub-picture. The length of the pps_subpic_id[i] syntax element is pps_subpic_id_len_minus1+1 bits.

[0236] For each value of i in the range from 0 to sps_num_subpics_minus1 (inclusive), the variable SubpicIdVal[i] is derived as follows:

[0237]

[0238] In the same or another embodiment, for any two different values ​​of i and j in the range of 0 to sps_num_subpics_minus1 (inclusive), SubpicIdVal[i] may not be equal to SubpicIdVal[j].

[0239] In the same embodiment or another embodiment, when the current picture is not the first picture of CLVS, for each value of i in the range of 0 to sps_num_subpics_minus1 (including the endpoints), if the value of SubpicIdVal[i] is not equal to the value of SubpicIdVal[i] of the previous picture in the same layer in decoding order, the nal_unit_type of all coded slice NAL units of the sub-pictures in the current picture with sub-picture index i can be equal to a specific value in the range of IDR_W_RADL to CRA_NUT (including the endpoints).

[0240] In the same or another embodiment, when the current picture is not the first picture of the CLVS, for each value of i in the range of 0 to sps_num_subpics_minus1 (including the endpoints), if the value of SubpicIdVal[i] is not equal to the value of SubpicIdVal[i] of the previous picture in the same layer in decoding order, sps_independent_subpics_flag can be equal to 1.

[0241] In the same or another embodiment, when the current picture is not the first picture of the CLVS, for each value of i in the range of 0 to sps_num_subpics_minus1 (including the endpoints), if the value of SubpicIdVal[i] is not equal to the value of SubpicIdVal[i] of the previous picture in the same layer in decoding order, subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] can be equal to 1.

[0242] In the same or another embodiment, when the current picture is not the first picture of the CLVS, for each value of i in the range of 0 to sps_num_subpics_minus1 (including the endpoints), if the value of SubpicIdVal[i] is not equal to the value of SubpicIdVal[i] of the previous picture in the same layer in decoding order, then sps_independent_subpics_flag shall be equal to 1 or subpic_treat_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] shall be equal to 1.

[0243] In the same or another embodiment, when a sub-picture is encoded independently without reference to another sub-picture, the value of the sub-picture identifier of a region may be changed within the encoded video sequence.

[0244] Samples are processed in units of CTB. The array size of each luma CTB is CtbSizeY in samples in terms of width and height. The width and height of the array of each chroma CTB are CtbWidthC and CtbHeightC in samples, respectively. Each CTB is assigned partition signaling to identify the block size used for intra or inter prediction and for transform coding. The partition is a recursive quadtree partition. The root of the quadtree is associated with the CTB. The quadtree is split until a leaf is reached, which is called a quadtree leaf. When the component width is not an integer of the CTB size, the CTB at the right component boundary is incomplete. When the component height is not an integer multiple of the CTB size, the CTB at the bottom component boundary is incomplete.

[0245] The width and height information of each sub-picture can be signaled in the SPS in units of CtbSizeY. Fig.23In, for example, subpic_width_minus1[i] plus 1 specifies the width of the i-th sub-picture in CtbSizeY units. The length of the syntax element is Ceil(Log2((pic_width_max_in_luma_samples+CtbSizeY-–1)>>CtbLog2SizeY)) bits. When not present, the value of subpic_width_minus1[i] is inferred to be equal to ((pic_width_max_in_luma_samples+CtbSizeY-1)>>CtbLog2SizeY)-subpic_ctu_top_left_x[i]–1. subpic_height_minus1[i] plus 1 specifies the height of the i-th sub-picture in CtbSizeY units. The length of the syntax element is Ceil(Log2((pic_height_max_in_luma_samples+CtbSizeY-–1)>>CtbLog2SizeY)) bits. When not present, the value of subpic_height_minus1[i] is inferred to be equal to ((pic_height_max_in_luma_samples+CtbSizeY-1)>>CtbLog2SizeY)-subpic_ctu_top_left_y[i]-1.

[0246] When the image width is greater than or equal to CtbSizeY, the width of each sub-image can be greater than or equal to CtbSizeY. When the image height is greater than or equal to CtbSizeY, the height of each sub-image can be greater than or equal to CtbSizeY.

[0247] If the picture width is not greater than CtbSizeY and the picture height is not greater than CtbSizeY, then the picture may not be divided into more than one sub-picture. In this case, the number of sub-pictures can be equal to 1.

[0248] When pic_width_max_in_luma_samples is not greater than CtbSizeY and pic_height_max_in_luma_samples is not greater than CtbSizeY, the value of subpic_info_present_flag shall be equal to 0. When subpic_info_present_flag is equal to 0, there is no explicit signaling for sub-picture partitioning information and the number of sub-pictures in the picture is equal to one.

[0249] In the same or another embodiment, sps_subpic_id_len_minus1 plus 1 specifies the number of bits used to represent the syntax element sps_subpic_id[i], the syntax element pps_subpic_id[i] (when present), and the syntax element slice_subpic_id (when present). The value of sps_subpic_id_len_minus1 may be in the range of 0 to 15, inclusive. The value of 1<<(sps_subpic_id_len_minus1) may be greater than or equal to sps_num_subpics_minus1+1.

[0250] In the same or another embodiment, when the number of sub-pictures is equal to 1, subpic_info_present_flag may be equal to 0 and the sub-picture partitioning information may not be explicitly signaled, because in this case the sub-picture width and height information are equal to the picture width and height information, and the upper left position of the sub-picture is equal to the upper left position of the picture.

[0251] For example, subpic_ctu_top_left_x[i] specifies the horizontal position of the top-left CTU of the i-th sub-picture in units of CtbSizeY. The length of the syntax element is Ceil(Log2((pic_width_max_in_luma_samples+CtbSizeY-–1)>>CtbLog2SizeY)) bits. When not present, the value of subpic_ctu_top_left_x[i] is inferred to be equal to 0. subpic_ctu_top_left_y[i] specifies the vertical position of the top-left CTU of the i-th sub-picture in units of CtbSizeY. The length of the syntax element is Ceil(Log2((pic_height_max_in_luma_samples+CtbSizeY-–1)>>CtbLog2SizeY)) bits. When not present, the value of subpic_ctu_top_left_y[i] is inferred to be equal to 0. subpic_width_minus1[i] plus 1 specifies the width of the i-th sub-picture in CtbSizeY units. The length of the syntax element is Ceil(Log2((pic_width_max_in_luma_samples+CtbSizeY-–1)>>CtbLog2SizeY)) bits. When not present, the value of subpic_width_minus1[i] is inferred to be equal to ((pic_width_max_in_luma_samples+CtbSizeY-1)>>CtbLog2SizeY)-subpic_ctu_top_left_x[i]-1. subpic_height_minus1[i] plus 1 specifies the height of the i-th sub-picture in CtbSizeY units. The length of the syntax element is Ceil(Log2((pic_height_max_in_luma_samples+CtbSizeY-–1)>>CtbLog2SizeY)) bits. When not present, the value of subpic_height_minus1[i] is inferred to be equal to ((pic_height_max_in_luma_samples+CtbSizeY-–1)>>CtbLog2SizeY)-subpic_ctu_top_left_y[i]-1.

[0252] In the same or another embodiment, when the number of sub-pictures is greater than 1, subpic_info_present_flag may be equal to 1, and the sub-picture partitioning information may be explicitly signaled in the parameter set, such as Fig.23 shown.

[0253] For example, in Fig.23 In , sps_num_subpics_minus2 plus 2 specifies the number of subpictures in each picture in the CLVS. The value of sps_num_subpics_minus2 can be in the range of 0 to Ceil(pic_width_max_in_luma_samples ÷ CtbSizeY)*Ceil(pic_height_max_in_luma_samples ÷ CtbSizeY)-2 (inclusive). When not present, the value of sps_num_subpics_minus2 is inferred to be equal to 0.

[0254] In the same embodiment, the lists SubpicWidthInTiles[i] and SubpicHeightInTiles[i] (i ranges from 0 to sps_num_subpics_minus1 (inclusive)) that specify the width and height of the i-th sub-picture in the tile column and row, respectively, and the list subpicHeightLessThanOneTileFlag[i] (i ranges from 0 to sps_num_subpics_minus1 (inclusive)) that specifies whether the height of the i-th sub-picture is less than one tile row are derived as follows:

[0255]

[0256] When rect_slice_flag is equal to 1, a list NumCtusInSlice[i] specifying the number of CTUs in the i-th slice (i ranges from 0 to num_slices_in_pic_minus1 (inclusive)), a list SliceTopLeftTileIdx[i] specifying the tile index of the tile containing the first CTU in the slice (i ranges from 0 to num_slices_in_pic_minus1 (inclusive)), and a matrix CtbAddrInSlice[i][j] specifying the picture raster scan address of the j-th CTB in the i-th slice (i ranges from 0 to num_slices_in_pic_minus1 (inclusive), and j ranges from 0 to NumCtusInSlice-1 (inclusive)), and a variable specifying the number of slices in the tile containing the i-th slice

[0257]

[0258]

[0259]

[0260] Two or more independently encoded sub-pictures may be merged into a coded picture so that the coded picture may be decoded and output as a single picture.

[0261] When two or more independently coded sub-pictures are merged into a coded picture, the coded picture may be composed of VCL NAL units with two or more different NAL unit types.

[0262] exist Fig.23 In the parameter set (e.g., PPS, SPS), the flag mixed_nalu_types_in_pic_flag can be signaled. mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture of the referenced PPS has more than one VCL NAL unit, and the VCL NAL units do not have the same nal_unit_type value. mixed_nalu_types_in_pic_flag equal to 0 specifies that each picture of the referenced PPS has one or more VCL NAL units, and the VCL NAL units of each picture of the referenced PPS have the same nal_unit_type value.

[0263] When mixed_nalu_types_in_pic_flag in the PPS is equal to 1, each picture with mixed_nalu_types_in_pic_flag equal to 1 is processed as a drag picture. Therefore, a coded picture with two or more different NAL unit types can be decoded as a drag picture. When the picture is referenced by a subsequent picture in decoding order, the picture can be processed as a drag picture.

[0264] exist Fig.23 sps_independent_subpics_flag equal to 1 specifies that all sub-picture boundaries in the CLVS are considered picture boundaries and there is no loop filtering across sub-picture boundaries. sps_independent_subpics_flag equal to 0 does not impose such a constraint. When not present, the value of sps_independent_subpics_flag is inferred to be equal to 0.

[0265] exist Fig.23In the OLS, subpic_treated_as_pic_flag[i] equal to 1 specifies that the i-th subpicture of each coded picture in the CLVS is to be treated as a picture in the decoding process except for loop filtering operations. subpic_treated_as_pic_flag[i] equal to 0 specifies that the i-th subpicture of each coded picture in the CLVS is not to be treated as a picture in the decoding process except for loop filtering operations. When not present, the value of subpic_treated_as_pic_flag[i] is inferred to be equal to sps_independent_subpics_flag. When subpic_treated_as_pic_flag[i] is equal to 1, the bitstream conformance requirement is that for each output layer in the OLS that includes the layer containing the i-th subpicture as an output layer and its reference layers, all of the following are true:

[0266] All pictures in the output layer and its reference layers shall have the same value of pic_width_in_luma_samples and the same value of pic_height_in_luma_samples.

[0267] All SPSs referenced by the output layer and its reference layers shall have the same sps_num_subpics_minus1 value, and for each value of j in the range of 0 to sps_num_subpics_minus1 (inclusive), shall have the same values ​​of subpic_ctu_top_left_x[j], subpic_ctu_top_left_y[j], subpic_width_minus1[j], subpic_height_minus1[j], and loop_filter_across_subpic_enabled_flag[j], respectively.

[0268] For each value of j in the range 0 to sps_num_subpics_minus1 (inclusive), all pictures in every access unit in the output layer and its reference layers shall have the same value of SubpicIdVal[j].

[0269] exist Fig.23In the CLVS, loop_filter_across_subpic_enabled_flag[i] equal to 1 specifies that loop filtering operations may be performed across the boundaries of the i-th subpic in each coded picture in the CLVS. loop_filter_across_subpic_enabled_flag[i] equal to 0 specifies that loop filtering operations will not be performed across the boundaries of the i-th subpic in each coded picture in the CLVS. When not present, the value of loop_filter_across_subpic_enabled_pic_flag[i] is inferred to be equal to 1-sps_independent_subpics_flag.

[0270] When two or more coded sub-pictures are merged into a coded picture, those coded sub-pictures may not have any parsing or decoding dependencies between each other.

[0271] In an embodiment, when mixed_nalu_types_in_pic_flag in a PPS is equal to 1, the value of subpic_treated_as_pic_flag[] of a sub-picture referring to the PPS may be equal to 1.

[0272] In an embodiment, when sps_independent_subpics_flag is equal to 0 and one or more subpic_treated_as_pic_flag[] values ​​are not equal to 1, mixed_nalu_types_in_pic_flag may be equal to 0.

[0273] In an embodiment, when mixed_nalu_types_in_pic_flag is equal to 1, the value of sps_independent_subpics_flag may be equal to 1.

[0274] In an embodiment, when mixed_nalu_types_in_pic_flag in a PPS is equal to 1, the value of subpic_treated_as_pic_flag[] of the sub-picture referencing the PPS is inferred to be equal to 1.

[0275] In an embodiment, when two or more adjacent sub-pictures with different NAL unit types within a picture shall have the value of subpic_treated_as_pic_flag[] equal to 1.

[0276] In an embodiment, Fig.24In the PPS, sub-picture partitioning information can be signaled in the PPS. For example, pps_independent_subpics_flag equal to 1 specifies that all boundary sub-pictures of the referenced PPS are considered picture boundaries and there is no loop filtering across sub-picture boundaries. pps_independent_subpics_flag equal to 0 does not impose such a constraint. When not present, the value of pps_independent_subpics_flag is inferred to be equal to 0. pps_subpic_treated_as_pic_flag[i] equal to 1 specifies that the i-th sub-picture of each coded picture of the referenced PPS is treated as a picture in the decoding process except for loop filtering operations. pps_subpic_treated_as_pic_flag[i] equal to 0 specifies that the i-th sub-picture of each coded picture of the referenced PPS is not treated as a picture in the decoding process except for loop filtering operations. When not present, the value of pps_subpic_treated_as_pic_flag[i] is inferred to be equal to pps_independent_subpics_flag. pps_loop_filter_across_subpic_enabled_flag[i] equal to 1 specifies that loop filtering operations may be performed across the boundaries of the i-th subpic in each coded picture that references the PPS. pps_loop_filter_across_subpic_enabled_flag[i] equal to 0 specifies that loop filtering operations will not be performed across the boundaries of the i-th subpic in each coded picture that references the PPS. When not present, the value of pps_loop_filter_across_subpic_enabled_pic_flag[i] is inferred to be equal to 1-pps_independent_subpics_flag.

[0277] In the same embodiment, when mixed_nalu_types_in_pic_flag in PPS is equal to 1, the value of pps_subpic_treated_as_pic_flag[] shall be equal to 1.

[0278] In the same or another embodiment, when mixed_nalu_types_in_pic_flag is equal to 1, pps_independent_subpics_flag shall be equal to 1.

[0279] In the same or another embodiment, when mixed_nalu_types_in_pic_flag is equal to 1, pps_subpic_treated_as_pic_flag[] shall be equal to 1.

[0280] In an embodiment, when mixed_nalu_types_in_pic_flag is equal to 1 and at least the VCL NAL unit of the picture has nal_unit_type equal to CRA_NUT, a CRA sub-picture or picture may not be processed as a CVS start picture.

[0281] In an embodiment, when mixed_nalu_types_in_pic_flag is equal to 1 and at least the VCL NAL unit of the picture has nal_unit_type equal to CRA_NUT, the leading picture associated with the CRA sub-picture or picture may be output.

[0282] In the same embodiment, when mixed_nalu_types_in_pic_flag is equal to 1 and at least the VCL NAL unit of a picture has nal_unit_type equal to CRA_NUT, HandleCraAsCvsStartFlag and NoOutputBeforeRecoveryFlag of the picture are both set equal to 0.

[0283] The embodiment of the present application also provides a device for decoding video data. The device includes: a receiving module for receiving video data including one or more sub-pictures; an identification module for identifying a hybrid network abstraction layer (NAL) unit type associated with each of the one or more sub-pictures based on checking a flag corresponding to a NAL unit in the one or more sub-pictures; and a decoding module for decoding the video data based on the identified NAL unit type.

[0284] The embodiment of the present application also provides a computer device for decoding video data. The computer device may include one or more processors and one or more computer-readable non-transitory storage media. The one or more computer-readable non-transitory storage media are configured to store computer program codes, and the computer program codes are used by the one or more computer processors to enable the computer device to perform a method when they are executed. The method may include receiving video data including one or more sub-pictures. Based on checking a flag corresponding to a hybrid network abstraction layer (NAL) unit in one or more sub-pictures, the NAL unit type associated with each of the one or more sub-pictures is identified. The video data is decoded based on the identified NAL unit type.

[0285] The embodiment of the present application also provides a computer-readable medium for decoding video data. The computer-readable medium may store a computer program for decoding video data, and the computer program is configured to cause one or more computer processors to perform a method. The method may accordingly include receiving video data including one or more sub-pictures. Based on checking a flag corresponding to a hybrid network abstraction layer (NAL) unit in one or more sub-pictures, a NAL unit type associated with each of the one or more sub-pictures is identified. The video data is decoded based on the identified NAL unit type.

[0286] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various alternative equivalents that fall within the scope of the present disclosure. It should therefore be understood that those skilled in the art will be able to design a variety of systems and methods that, although not explicitly shown or described herein, implement the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A method for decoding video data, characterized in that: include: Receiving video data including one or more sub-pictures; identifying a NAL unit type associated with each sub-picture based on examining a flag corresponding to a hybrid network abstraction layer NAL unit in one or more sub-pictures; as well as The video data is decoded based on the identified NAL unit type, wherein a mixed_nalu_types_in_pic_flag flag of the picture is determined to be equal to 1 and a sps_independent_subpics_flag flag is determined to be equal to 1 based on that NAL unit types of at least two sub-pictures in one picture are different.

2. The method according to claim 1, characterized in that When the value of the HandleCraAsCvsStartFlag flag is 1, it is used to indicate that the current picture is the starting point of the current encoded video sequence.

3. The method according to claim 2, characterized in that Based on having a mixed NAL unit type and the video coding layer NAL unit having a clean random access type, both the HandleCraAsCvsStartFlag flag and the NoOutputBeforeRecoveryFlag flag for the video data are set equal to 0.

4. The method according to claim 3, characterized in that Based on the Handle CraAsCvsStartFlag being set equal to 0, the current sub-picture is not processed as the start picture of the coded video sequence.

5. The method according to claim 2, characterized in that: Based on the mixed NAL unit type and the video coding layer NAL unit having a clean random access type, the leading picture associated with the video data is output.

6. The method according to claim 5, characterized in that Both the HandleCraAsCvsStartFlag flag and the NoOutputBeforeRecoveryFlag flag for video data are set equal to 0.

7. The method according to claim 6, characterized in that Based on having a mixed NAL unit type, the pps_subpic_treated_as_pic_flag flag is set equal to 1.

8. A method for encoding video data, characterized in that include: Receiving video data including one or more sub-pictures; identifying a NAL unit type associated with each sub-picture based on examining a flag corresponding to a hybrid network abstraction layer NAL unit in one or more sub-pictures; as well as The video data is decoded based on the identified NAL unit type, wherein a mixed_nalu_types_in_pic_flag flag of the picture is set to be equal to 1 and a sps_independent_subpics_flag flag is set to be equal to 1 based on that NAL unit types of at least two sub-pictures in one picture are different.

9. A device for decoding video data, characterized in that: The device comprises: A receiving module, used for receiving video data including one or more sub-pictures; an identification module for identifying a NAL unit type associated with each sub-picture based on examining a flag corresponding to a hybrid network abstraction layer NAL unit in one or more sub-pictures; and A decoding module is used to decode video data based on the identified NAL unit type, wherein based on the fact that the NAL unit types of at least two sub-pictures in a picture are different, the mixed_nalu_types_in_pic_flag flag of the picture is determined to be equal to 1, and the sps_independent_subpics_flag flag is determined to be equal to 1.

10. A computer device for processing video data, comprising: one or more computer processors; as well as one or more computer-readable non-transitory storage media configured to store computer program code; The method is characterized in that the computer program code is used to cause the computer device to perform the method according to any one of claims 1 to 8 when the computer program code is executed by the one or more computer processors.

11. A non-transitory computer-readable medium storing a computer program for processing video data, characterized in that: The computer program is configured to cause one or more computer processors to perform the method according to any one of claims 1 to 7.

12. A non-transitory computer-readable medium storing a computer program for processing video data, characterized in that: The computer program is configured to cause one or more computer processors to execute the method according to claim 8 to generate and store a video bitstream.

13. A method for generating a video bitstream, characterized in that: The video bit stream is generated according to the encoding method according to claim 8.

Images