Video code stream coding and decoding method and device, processing method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides video code stream encoding and decoding methods and devices, a processing method, electronic equipment and a storage medium. The decoding method comprises: obtaining a first flag from a video stream, the first flag indicating that reference picture resampling is enabled for an encoded video sequence (CVS); in response to the first flag indicating that reference picture resampling is enabled for the CVS, determining whether a picture resolution change is allowed in the CVS based on a second flag; when the second flag indicates that a picture resolution change is allowed in the CVS, decoding the CVS by resampling a reference picture for spatial scalability and adaptive resolution change (ARC); and when the value of the second flag indicates that the picture resolution does not change in the CVS, decoding the CVS by resampling the reference picture for spatial scalability, but not for ARC, and obtaining a third flag indicating whether sub-picture information of the CVS will appear in the video bitstream.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority under Section 119 of the United States Patent Act to U.S. Provisional Patent Application No. 63 / 027,835 filed on May 20, 2020, U.S. Provisional Patent Application No. 63 / 035,647 filed on June 5, 2020, U.S. Provisional Patent Application No. 63 / 036,174 filed on June 8, 2020, U.S. Provisional Patent Application No. 63 / 036,342 filed on June 8, 2020, and U.S. Patent Application No. 17 / 211,236 filed on March 24, 2021, and the disclosures of the above applications are incorporated by reference into this application in their entirety. Technical Field

[0003] The present application relates to video encoding and decoding, and more specifically to a method and apparatus for encoding and decoding a video code stream, a processing method, an electronic device, and a storage medium. Background Art

[0004] Video encoding and decoding using inter-picture prediction with motion compensation is known. An uncompressed digital video may consist of a series of pictures, each picture having a certain spatial dimension, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures may have a fixed or variable picture rate (also informally referred to as a frame rate), for example, 60 pictures per second or 60 Hertz (Hz). Uncompressed video has significant bit rate requirements. For example, 1080p60 4:2:0 video (1920×1080 luma sample resolution at 60 Hz frame rate) with 8 bits per sample requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0005] One purpose of video encoding and decoding can be to reduce redundancy in the input video signal through compression. Compression can help reduce the bandwidth or storage space requirements mentioned above, in some cases, by two or more orders of magnitude. Both lossless compression and lossy compression and combinations thereof can be used for video encoding and decoding. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original signal and the reconstructed signal is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used in video. The amount of distortion allowed by lossy compression depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that the higher the allowable / tolerable distortion, the higher the compression ratio that can be produced.

[0006] There are several broad categories of techniques that video encoders and decoders can use, including, for example, motion compensation, transforms, quantization, and entropy coding. Some of the techniques in these categories are described below.

[0007] Historically, video encoders and decoders have tended to operate on a given picture size, which in most cases is defined and kept constant for a Coded Video Sequence (CVS), Group of Pictures (GOP) or similar multi-picture temporal frame. For example, in MPEG-2, the system design changes the horizontal resolution (and thus the picture size) depending on factors such as scene activity, but only at I pictures, so this generally applies to GOPs. Resampling reference pictures to allow different resolutions to be used in CVS is known, for example from ITU-T Rec. H.263 Annex P. However, since the picture size does not change here, only the reference picture is resampled, which may result in only a portion of the picture canvas being used (in the case of downsampling) or only a portion of the scene being captured (in the case of upsampling). In addition, H.263 Annex Q allows individual macroblocks to be resampled up or down by a factor of two (in each dimension). Again, the picture size remains constant. The size of the macroblock is fixed in H.263 and therefore does not need to be signaled.

[0008] Changes in picture size in predicted pictures have become more mainstream in modern video coding. For example, VP9 allows reference picture resampling and resolution changes for the entire picture. Similarly, certain proposals for VVC (including, for example, Hendry et al., "On adaptive resolution change (ARC) for VVC", Joint Video Team document JVET-M0135-v1, January 9-19, 2019, which is incorporated herein in its entirety) allow the entire reference picture to be resampled to a different (higher or lower) resolution. In this document, it is proposed that different candidate resolutions be encoded in a sequence parameter set and referenced by a per-picture syntax element in the picture parameter set. Summary of the invention

[0009] In an embodiment, a method for decoding a video stream is provided, the method comprising: obtaining a first flag from the video stream, the first flag indicating that reference picture resampling is enabled for a coded video sequence (CVS); in response to the first flag indicating that reference picture resampling is enabled for the CVS, determining whether picture resolution change is allowed in the CVS based on a second flag; when the second flag indicates that picture resolution change is allowed in the CVS, decoding the CVS by using reference picture resampling for spatial scalability and adaptive resolution change (ARC); and when the value of the second flag indicates that picture resolution does not change in the CVS, decoding the CVS by using reference picture resampling for spatial scalability but not using reference picture resampling for ARC, and obtaining a third flag, the third flag indicating whether sub-picture information of the CVS will appear in the video stream.

[0010] In an embodiment, a method for encoding a video bitstream is provided, the method comprising: signaling a first flag, the first flag indicating that reference picture resampling is enabled for a coded video sequence (CVS); in response to the first flag indicating that reference picture resampling is enabled for the CVS, signaling a second flag, the second flag indicating whether picture resolution change is allowed in the CVS; when the second flag indicates that picture resolution change is allowed in the CVS, encoding the CVS by using reference picture resampling for spatial scalability and adaptive resolution change (ARC); and when the value of the second flag indicates that picture resolution does not change in the CVS, encoding the CVS by using reference picture resampling for spatial scalability but not using reference picture resampling for ARC, and signaling a third flag, the third flag indicating whether sub-picture information of the CVS will appear in the video bitstream.

[0011] In an embodiment, a method for processing a video code stream is provided. The video code stream is decoded according to the aforementioned decoding method, or is generated based on the aforementioned encoding method.

[0012] In an embodiment, a video code stream decoding device is provided, comprising a processing circuit configured to execute the above-mentioned video code stream decoding method.

[0013] In an embodiment, a video code stream encoding device is provided, comprising a processing circuit configured to execute the above-mentioned video code stream decoding method and encoding method.

[0014] In an embodiment, an electronic device is provided, including a memory for storing computer-readable instructions; a processor for reading the computer-readable instructions and executing the aforementioned method as instructed by the computer-readable instructions.

[0015] In an embodiment, a non-volatile computer-readable storage medium is provided, on which a video code stream is stored. The video code stream is decoded according to the aforementioned decoding method, or is generated based on the aforementioned encoding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, nature and various advantages of the subject matter of the present application will become more apparent from the following detailed description and accompanying drawings, in which:

[0017] Figure 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0018] Figure 2 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0019] Figure 3 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0020] Figure 4 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0021] Figures 5A-5E is a diagram of options for signaling ARC parameters according to an embodiment.

[0022] Figure 6A-6B is a diagram of an example of a syntax table according to an embodiment.

[0023] Figure 7 is an example of a prediction structure for scalability with adaptive resolution change according to an embodiment.

[0024] Figure 8 is an example of a syntax table according to an embodiment.

[0025] Fig. 9is a schematic diagram of a simplified block diagram for parsing and decoding a picture order count (poc) period and an access unit count value for each access unit according to an embodiment.

[0026] Fig.10 is a schematic diagram of a video bitstream structure including multiple layers of sub-pictures according to an embodiment.

[0027] Fig.11 is a schematic diagram of displaying a selected sub-picture with enhanced resolution according to an embodiment.

[0028] Fig.12 is a block diagram of a decoding and display process of a video stream including multiple layers of sub-pictures according to an embodiment.

[0029] Fig.13 is a schematic diagram of displaying a 360-degree video with an enhancement layer of a sub-picture according to an embodiment.

[0030] Fig.14 is an example of layout information of sub-pictures and their corresponding layers and picture prediction structures according to an embodiment.

[0031] Fig.15 is an example of layout information of sub-pictures and their corresponding layers and picture prediction structures in a form of spatial scalability of local areas according to an embodiment.

[0032] Figures 16A-16B is an example of a syntax table of sub-picture layout information according to an embodiment.

[0033] Fig.17 is an example of a syntax table of a SEI message of sub-picture layout information according to an embodiment.

[0034] Fig.18 is an example of a syntax table for indicating output layers and profile / tier / level information for each output layer set according to an embodiment.

[0035] Fig.19 is an example of a syntax table for indicating an output layer mode for each output layer set according to an embodiment.

[0036] Fig. 20 is an example of a syntax table for indicating a sub-picture existing in each layer of each output layer set according to an embodiment.

[0037] Fig.21 is an example of a syntax table for a video parameter set raw byte sequence payload (RBSP).

[0038] Fig. 22 is an example of a syntax table indicating an output layer set having an output layer set mode.

[0039] Fig.23 is an example of a syntax table for indicating reference picture resampling and resolution change within a coded video sequence in a sequence parameter set (SPS).

[0040] Fig.24 is an example of a syntax table indicating the picture size in a picture parameter set (PPS).

[0041] FIG. 25A to FIG. 25C is a flowchart of an example method for decoding an encoded video stream according to an embodiment.

[0042] Fig.26 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION

[0043] Figure 1 A simplified block diagram of a communication system (100) according to an embodiment disclosed in the present application is illustrated. The system (100) may include at least two terminals (110-120) interconnected by a network (150). For unidirectional transmission of data, a first terminal (110) may encode video data at a local location for transmission to another terminal (120) via the network (150). A second terminal (120) may receive encoded video data from another terminal from the network (150), decode the encoded video data, and display the recovered video data. Unidirectional data transmission may be common in applications such as media services.

[0044] Figure 1 A second pair of terminals (130, 140) is illustrated for supporting bidirectional transmission of encoded video, which may occur, for example, during a video conference. For bidirectional transmission of data, each terminal (130, 140) may encode video data collected at a local location for transmission to the other terminal via a network (150). Each terminal (130, 140) may also receive encoded video data transmitted by the other terminal, may decode the encoded data, and display the recovered video data on a local display device.

[0045] exist Figure 1In the embodiment, the terminals (110-140) may be illustrated as servers, personal computers, and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (150) represents any number of networks that transmit encoded video data between the terminals (110-140), including, for example, wired and / or wireless communication networks. The communication network (150) may exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet. For the purposes of the present application discussion, unless explained below, the architecture and topology of the network (150) may be irrelevant to the operations disclosed in the present application.

[0046] As an example of the application of the subject matter disclosed in this application, Figure 2 The placement of the video encoder and video decoder in a streaming environment is shown. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0047] The streaming system may include an acquisition subsystem (213), which may include a video source (201) such as a digital camera, which creates, for example, an uncompressed video sample stream (202). The sample stream (202) is depicted as a thick line to emphasize that it has a higher data volume than an encoded video stream, and the sample stream (202) may be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the subject matter disclosed in the present application as described in more detail below. The encoded video stream (204) is depicted as a thin line to emphasize that it has a lower data volume than the sample stream, and it may be stored on a streaming server (205) for future use. One or more streaming clients (206, 208) may access the streaming server (205) to retrieve a copy (207, 209) of the encoded video stream (204). The client (206) may include a video decoder (210) that decodes an incoming copy (207) of an encoded video stream and produces an output video sample stream (211) that may be presented on a display (212) or another presentation device (not depicted). In some streaming systems, the video streams (204, 207, 209) may be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T Recommendation H.265. The video coding standard being developed is informally referred to as Versatile Video Coding or VVC. The subject matter disclosed herein may be used in the context of the VVC standard.

[0048] Figure 3 It may be a functional block diagram of a video decoder (210) according to an embodiment disclosed in the present application.

[0049] The receiver (310) may receive one or more coded video sequences to be decoded by the decoder (210); in the same or another embodiment, one coded video sequence is received at a time, wherein the decoding of each coded video sequence is independent of the other coded video sequences. The coded video sequence may be received from a channel (312), which may be a hardware / software link to a storage device storing the coded video data. The receiver (310) may receive the coded video data as well as other data, such as coded audio data and / or auxiliary data streams that may be forwarded to their respective consuming entities (not depicted). The receiver (310) may separate the coded video sequence from the other data. To prevent network jitter, a buffer memory (315) may be coupled between the receiver (310) and the entropy decoder / parser (320) (hereinafter referred to as the "parser"). When the receiver (310) receives data from a storage / forward device with sufficient bandwidth and controllability, or from an isochronous synchronous network, it may not be necessary to configure the buffer (315), or it may be made smaller. For use on a best effort traffic packet network such as the Internet, a buffer (315) may be required, which may be relatively large and advantageously may be of an adaptive size.

[0050] The video decoder (210) may include a parser (320) to reconstruct symbols (321) from an entropy coded video sequence. The types of symbols include information for managing the operation of the decoder (210) and potentially information for controlling a presentation device, such as a display (212), which is not part of the decoder but may be coupled to the decoder, such as Figure 3As shown. The control information for the presentation device may be a parameter set fragment (not depicted) of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI). The parser (320) may parse and / or entropy decode the received coded video sequence. The coding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (320) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a sub-picture, a tile, a slice, a brick, a macroblock, a Coding Tree Unit (CTU), a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. A tile may refer to a rectangular area of ​​a CU / CTU within a special tile row in a picture. A brick may refer to a rectangular area of ​​a CU / CTU row within a special tile. A slice may refer to at least one brick of a picture contained in a NAL unit. A sub-picture may refer to a rectangular area of ​​at least one slice in a picture. The entropy decoder / parser may also extract information from an encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and the like. The entropy decoder / parser may also extract information from an encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and the like.

[0051] The parser (320) may perform entropy decoding and / or parsing operations on the video sequence received from the buffer (315) to create symbols (321).

[0052] Depending on the type of coded video picture or portion of coded video picture (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of the symbol (321) may involve multiple different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by the parser (320) from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser (320) and the multiple units below is not described.

[0053] In addition to the functional blocks already mentioned, decoder 210 may be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these units closely interact with each other and may be integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the functional units below.

[0054] The first unit is a sealer and / or inverse transform unit (351). The sealer and / or inverse transform unit (351) receives quantized transform coefficients as symbols (321) from the parser (320) and control information including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer and / or inverse transform unit (351) may output a block including sample values, which may be input into an aggregator (355).

[0055] In some cases, the output samples of the scaler and / or inverse transform unit (351) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) uses surrounding reconstructed information extracted from the current (partially reconstructed) picture (358) to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator (355) adds the prediction information generated by the intra-prediction unit (352) to the output sample information provided by the scaler and / or inverse transform unit (351) on a per-sample basis.

[0056] In other cases, the output samples of the sealer and / or inverse transform unit (351) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (353) may access the reference picture memory (357) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (321) belonging to the block, these samples may be added to the output of the sealer and / or inverse transform unit (in this case referred to as residual samples or residual signal) by the aggregator (355) to generate output sample information. The motion compensation unit's retrieval of prediction samples from addresses within the reference picture memory may be controlled by motion vectors, and the motion vectors are provided to the motion compensation unit in the form of the symbols (321), for example, including X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.

[0057] The output samples of the aggregator (355) may be used by various loop filtering techniques in a loop filter unit (356). The video compression techniques may include in-loop filter techniques controlled by parameters included in the encoded video bitstream and available to the loop filter unit (356) as symbols (321) from the parser (320), and may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop filtered sample values.

[0058] The output of the loop filter unit (356) may be a sample stream that may be output to a rendering device (212) and stored in a reference picture memory for subsequent inter-picture prediction.

[0059] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. Once a coded picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (320)), the current reference picture (358) may become part of the reference picture buffer (357), and a new current picture memory may be reallocated before starting reconstruction of a subsequent coded picture.

[0060] The video decoder 210 may perform decoding operations according to a predetermined video compression technique as documented in a standard such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technique or standard, as specified in the video compression technology document or standard, particularly the profile therein. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, mega samples per second), maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0061] In an embodiment, the receiver (310) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0062] Figure 4 It may be a block diagram of a video encoder (203) according to an embodiment disclosed in the present application.

[0063] The encoder (203) may receive video samples from a video source (201) (not a part of the encoder) that may capture video images to be encoded by the encoder (203).

[0064] The video source (201) may provide a source video sequence in the form of a digital video sample stream to be encoded by the encoder (203), wherein the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (201) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (203) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be constructed as a spatial pixel array, wherein each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0065] According to an embodiment, the encoder (203) may encode and compress pictures of a source video sequence into an encoded video sequence (443) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of a controller (450). The controller controls other functional units as described below and is functionally coupled to these units. For the sake of brevity, couplings are not shown in the figure. Parameters set by the controller may include rate control related parameters (picture skipping, quantizer, lambda value of rate-distortion optimization technology, etc.), picture size, GOP layout, maximum motion vector search range, etc. A person skilled in the art can easily identify other functions of the controller (450) that may be related to a video encoder (203) optimized for a certain system design.

[0066] Some video decoders operate in what is readily recognizable to those skilled in the art as a "coding loop". As a simplified description, the coding loop may include an encoding portion of an encoder (430) (hereinafter referred to as a "source encoder"), which is responsible for creating symbols based on an input picture to be encoded and a reference picture, and a (local) decoder (433) embedded in the encoder (203), which reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because any compression between the symbols and the encoded video code stream is lossless in the video compression techniques contemplated by the subject matter disclosed herein). The reconstructed sample stream is input to a reference picture memory (434). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the reference picture buffer contents are also bit-accurately corresponding between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values ​​that the decoder will "see" when using prediction during decoding. This basic principle of reference picture synchronization (and the drift that results when synchronization cannot be maintained, for example due to channel errors) is well known to those skilled in the art.

[0067] The operation of the "local" decoder (433) can be combined with the above Figure 3 The "remote" decoder (210) is identical to the one described in detail. However, additional brief reference is made to Figure 4 , when symbols are available and the entropy encoder (445) and the parser (320) are capable of losslessly encoding and / or decoding the symbols into an encoded video sequence, the entropy decoding portion of the decoder (210), including the channel (312), the receiver (310), the buffer (315) and the parser (320), may not be fully implemented in the local decoder (433).

[0068] At this point, it can be observed that any decoder technology, except for the parsing and / or entropy decoding present in the decoder, must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on the decoder operation. The description of the encoder technology can be simplified because the encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only needed in certain areas and is provided below.

[0069] As part of its operation, the source encoder (430) may perform motion compensated predictive coding that predictively encodes an input frame with reference to one or more previously encoded frames from a video sequence designated as "reference frames." In this manner, the encoding engine (432) encodes the differences between pixel blocks of an input frame and pixel blocks of a reference frame that may be selected as a prediction reference for the input frame.

[0070] The local video decoder (433) may decode the encoded video data of the frame that may be designated as the reference frame based on the symbol created by the source encoder (430). The operation of the encoding engine (432) may advantageously be a lossy process. When the encoded video data is available at the video decoder ( Figure 4 When the video sequence is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (433) replicates the decoding process that may be performed by the video decoder on the reference frame and may cause the reconstructed reference frame to be stored in the reference picture cache (434). In this way, the encoder (203) may store a copy of the reconstructed reference frame locally that has common content (absent transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0071] The predictor (435) may perform a prediction search for the encoding engine (432). That is, for a new frame to be encoded, the predictor (435) may search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (435) may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (435), it may be determined that the input picture may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (434).

[0072] The controller (450) may manage encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters for encoding video data.

[0073] The outputs of all the above functional units may be entropy encoded in an entropy encoder (445). The entropy encoder performs lossless compression on the symbols generated by the various functional units according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0074] The transmitter (440) may buffer the encoded video sequence created by the entropy encoder (445) in preparation for transmission over a communication channel (460), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (440) may combine the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0075] The controller (450) may manage the operation of the encoder (203). During encoding, the controller (450) may assign a certain coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following frame types:

[0076] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0077] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0078] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0079] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or the blocks may be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be non-predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be non-predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0080] The video encoder (203) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (203) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0081] In an embodiment, the transmitter (440) may transmit additional data when transmitting the encoded video. The video encoder (430) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, auxiliary enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.

[0082] Recently, there has been some attention on the compression domain aggregation or extraction of multiple semantically independent picture parts into a single video picture. In particular, in the context of, for example, 360-degree codecs, multiple semantically independent source pictures (e.g., six cube faces of a 360-degree scene of a cube projection, or individual camera inputs in a multi-camera setup) may require separate adaptive resolution settings to handle each scene activity at a given point in time. In other words, at a given point in time, the encoder may choose to use different resampling factors for the multiple semantically independent pictures that make up the entire 360-degree scene. This in turn requires reference picture resampling to be performed when combining multiple parts of an encoded picture into a single picture, and adaptive resolution encoding signaling is available.

[0083] The following is an introduction to several terms that will be mentioned in the rest of this specification.

[0084] "Sub-picture" refers, in some cases, to a semantically grouped rectangular arrangement of samples, blocks, macroblocks, coding units, or similar entities that can be independently encoded with varying resolutions. One or more sub-pictures may form a picture. One or more coded sub-pictures may form a coded picture. One or more sub-pictures may be assembled into a picture, and one or more sub-pictures may be extracted from a picture. In some circumstances, one or more coded sub-pictures may be assembled into a coded picture in the compressed domain without transcoding them to the sample level, and in the same or other circumstances, one or more coded sub-pictures may be extracted from a coded picture in the compressed domain.

[0085] "Adaptive Resolution Change (ARC)" may refer to a mechanism that allows the resolution of a picture or sub-picture within a coded video sequence to be changed, for example, by resampling a reference picture. "ARC parameters" hereinafter refer to the control information required to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, resolution of output pictures and / or reference pictures, various control flags, and the like.

[0086] In an embodiment, encoding and decoding may be performed on a single, semantically independent coded video picture.Before describing the implications of encoding / decoding of multiple sub-pictures with independent ARC parameters and the additional complexity it implies, options for signaling ARC parameters will be described.

[0087] refer to Figures 5A-5E , several embodiments for signaling ARC parameters are shown. As indicated in each embodiment, they may have certain advantages and disadvantages from the perspective of coding efficiency, complexity, and architecture. Video coding standards or technologies may select one or more of these embodiments, or select options known in the prior art, for signaling ARC parameters. These embodiments may not be mutually exclusive and may be interchangeable depending on application needs, the standard technologies involved, or the selection of encoders.

[0088] Categories of ARC parameters can include:

[0089] - Upsampling factors and / or downsampling factors in the X and Y dimensions, either separately or combined

[0090] - Add upsampling factor and / or downsampling factor of the time dimension to indicate constant speed upscaling and / or downscaling of a given number of images

[0091] - Either of the above may involve encoding one or more possibly shorter syntax elements, which may point to a table containing the factor(s).

[0092] - "resolution": the resolution of an input picture, output picture, reference picture, coded picture, in samples, blocks, macroblocks, CUs, or any other suitable granularity, combined or separate, in either the X or Y dimensions. If more than one resolution is present (e.g., one for input pictures and one for reference pictures), then in some cases one set of values ​​may be inferred from the other. For example, the resolution may be gated using a flag. See below for more detailed examples.

[0093] - "Warping" coordinates: similar to the coordinates used in Annex P of the H.263 standard, which can have an appropriate granularity as described above. Annex P of the H.263 standard defines an efficient way to encode such warping coordinates, but it is conceivable that other potentially more efficient methods can also be envisioned. For example, the variable-length reversible "Huffman" encoding of the warping coordinates of Annex P is replaced by a binary encoding of an appropriate length, where the length of the binary codeword can be derived, for example, from the maximum picture size (possibly multiplied by a factor and offset by a value) to allow "warping" outside the boundaries of the maximum picture size).

[0094] - Upsampling filter parameters and / or downsampling filter parameters: In embodiments, only a single filter may be used for upsampling and / or downsampling. However, in embodiments, it may be desirable to allow greater flexibility in the design of the filters, which may be achieved by signaling filter parameters. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g., by a list of filter coefficients using an appropriate entropy coding technique), the filter may be selected implicitly by an upsampling ratio and / or a downsampling ratio, which in turn may be signaled according to any of the mechanisms mentioned above, etc.

[0095] In the following, this specification assumes that a finite set of upsampling factors and / or downsampling factors (using the same factors in the X and Y dimensions) indicated by codewords is encoded. The codewords can be variable-length encoded, for example, using Exp-Golomb coding common to certain syntax elements in video coding specifications (such as H.264 and H.265). A suitable mapping of values ​​to upsampling factors and / or downsampling factors can be, for example, shown in Table 1 below:

[0096] Table 1

[0097] Codeword Exp-Golomb encoding Original resolution / target resolution 0 1 1 / 1 1 010 1 / 1.5 (50% zoom) 2 011 1.5 / 1 (50% smaller) 3 00100 1 / 2 (100% zoom) 4 00101 2 / 1 (reduced by 100%)

[0098] Many similar mappings can be designed, depending on the needs of the application and the capabilities of the upscaling and downscaling mechanisms available in the video compression technology or standard. The table can be extended to more values. The values ​​can also be represented using entropy coding mechanisms other than Exp-Golomb codes, for example using binary encoding. Using binary encoding may have certain advantages when the resampling factors are of interest outside the video processing engine (most importantly the encoder and decoder) itself, such as the MANE (Media Aware Network Element). It should be noted that for cases where no resolution change is required, a shorter Exp-Golomb code can be chosen; in the above table, only a single bit. For this most common case, using Exp-Golomb codes can have an advantage in coding efficiency over using binary codes.

[0099] The number of entries in the table and their semantics may be fully or partially configurable. For example, the basic form of the table may be conveyed in a "higher layer" parameter set such as a sequence parameter set or a decoder parameter set. In an embodiment, one or more such tables may be defined in a video coding technology or standard and may be selected by, for example, a decoder or sequence parameter set.

[0100] The following describes how to include the upsampling factors and / or downsampling factors (ARC information) encoded as described above in a video coding technique or standard syntax. Similar considerations can be applied to one or more codewords that control upsampling filters and / or downsampling filters. See the discussion below for when a filter or other data structure requires a relatively large amount of data.

[0101] like Figure 5A As shown, Annex P of the H.263 standard includes ARC information (502) in the form of four deformation coordinates in the picture header (501), more specifically, in the H.263 standard PLUSPTYPE (503) header extension. This may be a wise design choice when a) there is a picture header available, and b) the ARC information is expected to change frequently. However, the overhead of using H.263-type signaling may be quite high, and the scaling factor of the picture boundary may not be relevant because the picture header may have an instantaneous nature.

[0102] like Figure 5B As shown, JVCET-M135-v1 includes ARC reference information (505) (index) located in a picture parameter set (504) for indexing a table (506) including target resolutions, and the table (506) is located in a sequence parameter set (507). According to the author's verbal statement, the location of possible resolutions in the table (506) in the sequence parameter set (SPS) (507) can be rationalized by using the SPS as an interoperability negotiation point during capability exchange. Within the limits set by the values ​​in the table (506), the resolution can be changed from picture to picture by referencing the appropriate picture parameter set (Picture Parameter Set, PPS) (504).

[0103] refer to Figure 5C-5E , there may be the following embodiments to transmit ARC information in a video bitstream. Each of these options has certain advantages over the above embodiments. These embodiments may exist simultaneously in the same video coding technology or standard.

[0104] In an embodiment, for example Figure 5C In the illustrated embodiment, ARC information (509) such as resampling (scaling) factors may appear in a slice header, a GOB (Group of Blocks) header, a tile header, or a tile group header. Figure 5CAn embodiment is shown in which a tile group header (508) is used. This may be sufficient if the ARC information is small, for example, the ARC information is a single variable length ue(v) codeword of a few bits or a fixed length codeword, for example, as shown above. The ARC information can be applied to, for example, a sub-picture represented by a tile group, rather than for the entire picture, and then there is the additional advantage of including the ARC information directly in the tile group header. See also below. In addition, even if the video compression technology or standard only envisions adaptive resolution changes for the entire picture (as opposed to, for example, adaptive resolution changes based on tile groups), from a fault tolerance perspective, placing the ARC information in the tile group header has certain advantages over placing it in an H.263-type picture header.

[0105] In an embodiment, for example Figure 5D In the illustrated embodiment, the ARC information (512) itself may reside in an appropriate parameter set, such as a picture parameter set, a header parameter set, a tile parameter set, an adaptation parameter set, and so on. Figure 5D An embodiment is shown in which an adaptation parameter set (511) is used. The scope of the parameter set may advantageously be no larger than one picture, such as a group of tiles. The use of ARC information may be implicit by activating the relevant parameter set. For example, when a video coding technique or standard only considers picture-based ARC, a picture parameter set or an equivalent parameter set may be applicable.

[0106] In an embodiment, for example Figure 5E In the illustrated embodiment, ARC reference information (513) may be present in a tile group header (514) or similar data structure. The ARC reference information (513) may refer to a subset of the ARC information (515) available in a parameter set (516) (e.g., a sequence parameter set or a decoder parameter set) that extends beyond a single picture.

[0107] like Fig. 6A As shown, a tile group header (601), as an exemplary syntax structure of a header applicable to a (possibly rectangular) portion of a picture, may conditionally contain a variable length Exp-Golomb coded syntax element dec_pic_size_idx (602) (shown in bold). The presence of this syntax element in the tile group header may be gated by using adaptive resolution (603). Here, the value of the flag is not shown in bold, which means that the point at which the flag appears in the codestream is when it appears in the syntax table. Whether adaptive resolution is used for the picture or a portion thereof may be signaled in any high-level syntax structure inside or outside the codestream. In the example shown, adaptive resolution is signaled in a sequence parameter set, as described below.

[0108] refer to Figure 6B, an excerpt of a sequence parameter set (610) is also shown. The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, this flag may indicate that adaptive resolution is used, which in turn may require specific control information. In this example, such control information is conditionally present based on the value of the flag, which is based on an if() statement in the sequence parameter set (612) and the tile group header (601).

[0109] When adaptive resolution is used, the output resolution in samples (613) is encoded. Reference numeral 613 refers to output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can define the resolution of the output picture. Elsewhere in the video coding technique or standard, certain restrictions on any value may be defined. For example, the level definition may limit the number of total output samples, which may be the product of the values ​​of the two syntax elements described above. In addition, certain video coding techniques or standards, or external techniques or standards (e.g., system standards) may limit the range of values ​​(e.g., one or both dimensions must be divisible by a power of 2) or aspect ratio (e.g., width and height must have a relationship such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementation or for other reasons, as is well known in the art.

[0110] In some applications, it is recommended that the encoder instruct the decoder to use a certain reference picture size, rather than implicitly assuming it to be the output picture size. In this example, the syntax element reference_pic_size_present_flag (614) enables the conditional presence of the reference picture size (615) (again, this number refers to both width and height).

[0111] Finally, a possible table of decoded picture width and height is shown. Such a table may be indicated, for example, by a table indication (num_dec_pic_size_in_luma_samples_minus1) (616). "Minus1" (minus 1) may refer to the interpretation of the value of this syntax element. For example, if the coded value is zero, there is one table entry. If the value is five, there are six table entries. For each "row" in the table, the decoded picture width and height are then contained in the syntax (617).

[0112] The presented table entry (617) can be indexed using the syntax element dec_pic_size_idx (602) in the tile group header, allowing each tile group to have a different decode size (effectively a scaling factor).

[0113] Some video coding techniques or standards, such as VP9, ​​support spatial scalability by implementing some form of reference picture resampling (which is signaled in a very different way than disclosed in this application) in conjunction with temporal scalability to achieve spatial scalability. More specifically, certain reference pictures can be upsampled to a higher resolution using ARC-type techniques to form the basis of a spatial enhancement layer. These upsampled pictures can be refined using standard prediction mechanisms at a higher resolution to increase detail.

[0114] The embodiments discussed in this application can be used in such environments. In some cases, in the same or another embodiment, the value in the Network Abstract Layer (NAL) unit header (e.g., the Temporal ID field) can be used to indicate not only the temporal layer, but also the spatial layer. Doing so may have certain advantages for certain system designs; for example, an existing Selected Forwarding Unit (SFU) created and optimized for forwarding selected for the temporal layer based on the Temporal ID value in the NAL unit header can be used in a scalable environment without modification. To achieve this, it may be necessary to map between the coded picture size and the temporal layer indicated by the Temporal ID field in the NAL unit header.

[0115] In some video coding technologies, an access unit (AU) may refer to a coded picture, slice, tile, NAL unit, etc., which is collected at a given time and constitutes a corresponding picture, slice, tile, and / or NAL unit stream. Such a time may be, for example, a composition time.

[0116] In HEVC and certain other video coding techniques, a Picture Order Count (POC) value may be used to indicate a reference picture selected from a plurality of reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) includes one or more pictures, slices, or tiles, each picture, slice, or tile belonging to the same AU may carry the same POC value, from which it may be inferred that they are created based on content with the same composition time. In other words, where two pictures / slices / tiles carry the same given POC value, this may indicate that the two pictures / slices / tiles belong to the same AU and have the same composition time. Conversely, two pictures / slices / tiles having different POC values ​​may indicate that the pictures / slices / tiles belong to different AUs and have different composition times.

[0117] In an embodiment, the aforementioned rigid relationship can be relaxed because an access unit may include pictures, slices, or tiles with different POC values. By allowing different POC values ​​within an AU, the POC value can be used to identify potentially independently decodable pictures / slices / tiles with the same representation time. This in turn enables support for multiple scalable layers without changing the reference picture selection signaling (e.g., reference picture set signaling or reference picture list signaling), as described in detail below.

[0118] However, compared with other pictures / slices / tiles with different POC values, it is still desirable to be able to identify the AU to which a picture / slice / tile belongs based solely on the POC value. This can be achieved as described below.

[0119] In an embodiment, an access unit count (AUC) may be signaled in a high-level syntax structure (e.g., a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter). The value of the AUC may be used to identify which NAL units, pictures, slices, or tiles belong to a given AU. The value of the AUC may correspond to different synthesis time instances. The AUC value may be equal to a multiple of the POC value. The AUC value may be calculated by dividing the POC value by an integer value. In some cases, the division operation may impose a certain burden on the decoder implementation. In such cases, a slight restriction on the numbering space of the AUC value may allow a shift operation to be used instead of a division operation. For example, the AUC value may be equal to the value of the most significant bit (MSB) of the POC value range.

[0120] In an embodiment, the value of the POC cycle (poc_cycle_au) for each AU may be signaled in a high-level syntax structure (e.g., a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter). poc_cycle_au may be used to indicate how many different and consecutive POC values ​​may be associated with the same AU. For example, if the value of poc_cycle_au is equal to 4, pictures, slices, or tiles with POC values ​​equal to 0-3 (including 0 and 3) may be associated with AUs with AUC values ​​equal to 0, and pictures, slices, or tiles with POC values ​​equal to 4-7 (including 4 and 7) may be associated with AUs with AUC values ​​equal to 1. Therefore, the value of AUC may be inferred by dividing the POC value by the value of poc_cycle_au.

[0121] In an embodiment, the value of poc_cycle_au can be derived from information located, for example, in a video parameter set (VPS), which is used to identify the number of spatial layers or SNR layers in a coded video sequence. An example of this possible relationship is briefly described below. Although the derivation as described above can save several bits in the VPS and thus improve coding efficiency, in some embodiments poc_cycle_au can be explicitly encoded in an appropriate high-level syntax structure at a level lower than the video parameter set, so that poc_cycle_au can be minimized for a given small portion of the codestream (e.g., a picture). Since the POC value and / or the value of the syntax element that indirectly references the POC can be encoded with a low-level syntax structure, this optimization can save more bits compared to the above-mentioned derivation process.

[0122] In an embodiment, Figure 8 An example of providing a syntax table is shown to signal a syntax element vps_poc_cycle_au in a VPS (or SPS), which indicates the poc_cycle_au for all pictures / slices in a coded video sequence and a syntax element slice_poc_cycle_au in a slice header, which is used to indicate the poc_cycle_au of the current slice. If the POC value increases uniformly according to the AU, the vps_contant_poc_cycle_per_au in the VPS can be set to 1 and vps_poc_cycle_au is signaled in the VPS. In this case, slice_poc_cycle_au may not be explicitly signaled, and the AUC value for each AU may be calculated by dividing the POC value by vps_poc_cycle_au. If the POC value does not increase uniformly according to the AU, the vps_contant_poc_cycle_per_au in the VPS may be set to 0. In this case, vps_access_unit_cnt may not be signaled, but slice_access_unit_cnt may be signaled in the slice header of each slice or picture. Each slice or picture may have a different slice_access_unit_cnt value. The AUC value of each AU is calculated by dividing the POC value by slice_poc_cycle_au.

[0123] Fig. 9A block diagram illustrating an example of the above process is shown. For example, in operation S910, the VPS (or SPS) can be parsed, and it can be determined in operation S920 whether the POC period of each AU is constant within the encoded video sequence. If the POC period of each AU is constant ("yes" in operation S920), then in operation S930, the value of the access unit count of the specific access unit can be calculated based on the poc_cycle_au signaled for the encoded video sequence and the POC value of the specific access unit. If the POC period of each AU is not constant ("no" in operation S920), then in operation S940, the value of the access unit count of the specific access unit can be calculated based on the poc_cycle_au signaled at the picture level and the POC value of the specific access unit. In operation S950, the new VPS (or SPS) can be parsed.

[0124] In an embodiment, pictures, slices or tiles corresponding to AUs with the same AUC value may be associated with the same decoding or output time instance, even though the POC values ​​of the pictures, slices or tiles may be different. Therefore, in the absence of any inter-frame parsing / decoding dependencies between pictures, slices or tiles in the same AU, all or a subset of pictures, slices or tiles associated with the same AU may be decoded in parallel and may be output at the same time instance.

[0125] In an embodiment, pictures, slices or tiles corresponding to AUs with the same AUC value may be associated with the same composition / display time instance even though the POC values ​​of the pictures, slices or tiles may be different. When the composition time is included in the container format, multiple pictures may be displayed at the same time instance even if they correspond to different AUs if they have the same composition time.

[0126] In an embodiment, each picture, slice or tile may have the same temporal identifier (temporal_id) in the same AU. All or a subset of pictures, slices or tiles corresponding to one time instance may be associated with the same temporal sublayer. In an embodiment, each picture, slice or tile may have the same or different spatial layer id (layer_id) in the same AU. All or a subset of pictures, slices or tiles corresponding to one time instance may be associated with the same or different spatial layers.

[0127] Figure 7An example of a video sequence structure with a combination of temporal_id, layer_id, and POC and AUC values ​​with adaptive resolution variation is shown. In this example, a picture, strip or tile in the first AU with AUC=0 may have temporal_id=0 and layer_id=0 or 1, while a picture, strip or tile in the second AU with AUC=1 may have temporal_id=1 and layer_id=0 or 1, respectively. Regardless of the values ​​of temporal_id and layer_id, the POC value increases by 1 with each picture. In this example, the value of poc_cycle_au may be equal to 2. Preferably, the value of poc_cycle_au may be set equal to the number of (spatial scalability) layers. Therefore, in an embodiment, the POC value increases by 2, while the AUC value increases by 1.

[0128] In the above-mentioned embodiments, all or a subset of the inter-picture prediction structure or inter-layer prediction structure and reference picture indication may be supported by using the existing Reference Picture Set (RPS) signaling or Reference Picture List (RPL) signaling in HEVC. In the RPS or RPL, the selected reference picture is indicated by signaling the POC value or the difference in POC between the current picture and the selected reference picture. In an embodiment, the RPS and RPL may be used to indicate the inter-picture prediction structure or inter-layer prediction structure without changing the signaling, but with the following limitations. If the value of the temporal_id of the reference picture is greater than the value of the temporal_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction. If the value of the layer_id of the reference picture is greater than the value of the layer_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction.

[0129] In an embodiment, motion vector scaling for temporal motion vector prediction based on POC difference may be disabled across multiple pictures within an access unit. Thus, although each picture may have a different POC value within an access unit, the motion vector is not scaled and used for temporal motion vector prediction within an access unit. This is because, in the same AU, reference pictures with different POCs are considered to be reference pictures with the same temporal instance. Therefore, in an embodiment, the motion vector scaling function may return 1 when the reference picture belongs to the AU associated with the current picture.

[0130] In an embodiment, when the spatial resolution of the reference picture is different from the spatial resolution of the current picture, motion vector scaling for temporal motion vector prediction based on POC difference may be optionally disabled on multiple pictures. When motion vector scaling is allowed, the motion vector may be scaled based on the POC difference and the ratio of the spatial resolution between the current picture and the reference picture.

[0131] In an embodiment, the motion vector may be scaled based on the AUC difference instead of the POC difference for temporal motion vector prediction, especially when poc_cycle_au has a non-uniform value (e.g., when vps_contant_poc_cycle_per_au==0). Otherwise (e.g., when vps_contant_poc_cycle_per_au==1), the motion vector scaling based on the AUC difference may be the same as the motion vector scaling based on the POC difference.

[0132] In an embodiment, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU (with the same AUC value) as the current picture is not scaled based on the AUC difference, and is used for motion vector prediction without or based on the ratio of the spatial resolution between the current picture and the reference picture.

[0133] In an embodiment, the AUC value can be used to identify the boundaries of AUs and for hypothetical reference decoder (HRD) operations, which require input and output timing with AU granularity. In an embodiment, the decoded picture of the highest layer in the AU can be output for display. The AUC value and the layer_id value can be used to identify the output picture.

[0134] In an embodiment, a picture may include one or more sub-pictures. Each sub-picture may cover a partial area or the entire area of ​​the picture. The area supported by one sub-picture may overlap or not overlap with the area supported by another sub-picture. The area covered by one or more sub-pictures may cover or not cover the entire area of ​​the picture. If a picture includes a sub-picture, the area supported by the sub-picture may be the same as the area supported by the picture.

[0135] In an embodiment, a sub-picture may be encoded by a codec similar to that used by an encoded picture. A sub-picture may be independently encoded or may be encoded based on another sub-picture or an encoded picture. A sub-picture may or may not have a resolution dependency on any resolution from another sub-picture or an encoded picture.

[0136] In an embodiment, the coded sub-picture may be included in one or more layers. The coded sub-pictures in a layer may have different spatial resolutions. The original sub-picture may be spatially resampled (e.g., upsampled or downsampled), encoded with different spatial resolution parameters, and included in a codestream corresponding to the layer.

[0137] In an embodiment, a sub-picture with (W, H) may be encoded and included in an encoded bitstream corresponding to layer 0, wherein W indicates the width of the sub-picture, H indicates the height of the sub-picture, and the upsampled (or downsampled) sub-picture with the original spatial resolution and having (W*S w,k ,H*S h,k ) can be encoded and included in the encoded bitstream corresponding to layer k, where S w,k , S h,k Indicates the horizontal resampling rate and the vertical resampling rate. If S w,k , S h,k If the value of is greater than 1, the resampling can be upsampling. However, if S w,k , S h,k If the value of is less than 1, the resampling can be downsampling.

[0138] In an embodiment, a coded sub-picture in one layer may have a different visual quality than a coded sub-picture in another layer in the same sub-picture or in a different sub-picture. For example, a sub-picture i in layer n may be quantized with quantization parameter Q i,n For encoding, the sub-image j in layer m can be quantized with the quantization parameter Q j,m to encode.

[0139] In an embodiment, a coded sub-picture in one layer may be independently decodable without any parsing dependency or decoding dependency on a coded sub-picture in another layer of the same local region. A sub-picture layer that is independently decodable without reference to another sub-picture layer of the same local region may be an independent sub-picture layer. A coded sub-picture in an independent sub-picture layer may or may not have a decoding dependency or a parsing dependency on a previously coded sub-picture from the same sub-picture layer, but the coded sub-picture may not have any dependency on a coded picture from another sub-picture layer.

[0140] In an embodiment, a coded sub-picture in one layer may be dependently decodable, having any parsing dependency or decoding dependency on a coded sub-picture in another layer from the same local region. A sub-picture layer that is dependency decodable with reference to another sub-picture layer of the same local region may be a dependent sub-picture layer. A coded sub-picture in a dependent sub-picture may reference a coded sub-picture belonging to the same sub-picture, a previously coded sub-picture in the same sub-picture layer, or both reference sub-pictures.

[0141] In an embodiment, a coded sub-picture may include one or more independent sub-picture layers and one or more dependent sub-picture layers. However, for a coded sub-picture, at least one independent sub-picture layer may exist. The value of the layer identifier (layer_id) of the independent sub-picture layer may be equal to 0, which may be present in the NAL unit header or another high-level syntax structure. The sub-picture layer with layer_id equal to 0 may be a base sub-picture layer.

[0142] In an embodiment, a picture may include one or more foreground sub-pictures and one background sub-picture. The area supported by the background sub-picture may be equal to the area of ​​the picture. The area supported by the foreground sub-picture may overlap with the area supported by the background sub-picture. The background sub-picture may be a base sub-picture layer and the foreground sub-picture may be a non-base (enhanced) sub-picture layer. One or more non-base sub-picture layers may be decoded with reference to the same base layer. Each non-base sub-picture layer with layer_id equal to a may reference a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0143] In an embodiment, a picture may include one or more foreground sub-pictures with or without background sub-pictures. Each sub-picture may have its own base sub-picture layer and one or more non-base (enhancement) layers. Each base sub-picture layer may be referenced by one or more non-base sub-picture layers. Each non-base sub-picture layer with layer_id equal to a may reference a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0144] In an embodiment, a picture may include one or more foreground sub-pictures with or without background sub-pictures. Each coded sub-picture in a (base or non-base) sub-picture layer may be referenced by one or more non-base layer sub-pictures belonging to the same sub-picture and one or more non-base layer sub-pictures not belonging to the same sub-picture.

[0145] In an embodiment, a picture may include one or more foreground sub-pictures with or without background sub-pictures. A sub-picture in layer a may be further divided into multiple sub-pictures in the same layer. One or more coded sub-pictures in layer b may refer to the divided sub-pictures in layer a.

[0146] In an embodiment, a coded video sequence (CVS) may be a set of coded pictures. A CVS may include one or more coded sub-picture sequences (CSPS), where a CSPS may be a set of coded sub-pictures covering the same local area of ​​a picture. A CSPS may have the same or different temporal resolution as the coded video sequence.

[0147] In an embodiment, the CSPS may be encoded and included in one or more layers. The CSPS may include one or more CSPS layers. By decoding the one or more CSPS layers corresponding to the CSPS, a sub-picture sequence corresponding to the same area may be reconstructed.

[0148] In an embodiment, the number of CSPS layers corresponding to one CSPS may be the same as or different from the number of CSPS layers corresponding to another CSPS.

[0149] In an embodiment, one CSPS layer may have a different temporal resolution (e.g., frame rate) than another CSPS layer. The original (uncompressed) sub-picture sequence may be temporally resampled (e.g., upsampled or downsampled), encoded with different temporal resolution parameters, and included in the codestream corresponding to the layer.

[0150] In an embodiment, a sub-picture sequence with a frame rate of F may be encoded and included in a coded bitstream corresponding to layer 0, and a temporally upsampled (or downsampled) sub-picture sequence with a frame rate of F*S may be encoded and included in a coded bitstream corresponding to layer 0. t,k The sub-picture sequence can be encoded and included in the encoded bitstream corresponding to layer k, where S t,k indicates the temporal sampling rate of layer k. If S t,k If the value of S is greater than 1, the temporal resampling process may be a frame rate upconversion. t,k If the value of is less than 1, the temporal resampling process is equivalent to frame rate down-conversion.

[0151] In an embodiment, when a sub-picture with CSPS layer a is referenced by a sub-picture with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b, the decoded pixels in CSPS layer a are resampled and used as reference. The resampling process may use upsampling filtering or downsampling filtering.

[0152] Fig.10An example video stream is shown that includes a background video CSPS with layer_id equal to 0 and multiple foreground CSPS layers. Although the encoded sub-pictures can include one or more CSPS layers, the background area that does not belong to any foreground CSPS layer can include a base layer. The base layer can contain both the background area and the foreground area, while the enhanced CSPS layer can contain the foreground area. At the same region, the visual quality of the enhanced CSPS layer can be better than that of the base layer. The enhanced CSPS layer can refer to the reconstructed pixels and motion vectors of the base layer corresponding to the same region as it.

[0153] In an embodiment, the video bitstream corresponding to the base layer is contained in one track in the video file, while the CSPS layers corresponding to each sub-picture are contained in separate tracks.

[0154] In an embodiment, the video bitstream corresponding to the base layer is contained in one track, while the CSPS layers with the same layer_id are contained in separate tracks. In this example, the track corresponding to layer k includes only the CSPS layer corresponding to layer k.

[0155] In an embodiment, each CSPS layer of each sub-picture is stored in a separate track. Each track may or may not have any parsing dependencies or decoding dependencies on one or more other tracks.

[0156] In an embodiment, each track can contain the bitstream corresponding to layers i to j of the CSPS layers of all or a subset of the sub-pictures, where 0 < i = < j = < k, and k is the highest layer of the CSPS.

[0157] In an embodiment, a picture includes one or more associated media data, and these associated media data include depth maps, alpha maps, 3D geometric data, occupancy maps, etc. Such associated timed media data can be divided into one or more data sub-bitstreams, and each data sub-bitstream corresponds to a sub-picture.

[0158] Fig.11 An example of a video conference based on the multi-layer sub-picture method is shown. In the video stream, it includes a base layer video bitstream corresponding to the background picture and one or more enhanced layer video bitstreams corresponding to the foreground sub-pictures. Each enhanced layer video bitstream can correspond to a CSPS layer. On the display, the picture corresponding to the base layer is displayed by default. It includes picture-in-picture (PIP) of one or more users. When a specific user is selected through the control of the client, the enhanced CSPS layer corresponding to the selected user can be decoded and displayed with enhanced quality or spatial resolution.

[0159] Fig.12A block diagram illustrating an example of the above process is shown. For example, in operation S1210, a video code stream having multiple layers can be decoded. In operation S1220, a background area and one or more foreground sub-pictures can be identified. In operation S1230, it can be determined whether a specific sub-picture area, such as one of the foreground sub-pictures, is selected. If a specific sub-picture area is selected ("yes" in operation S1240), the enhanced sub-picture can be decoded and displayed. If a specific sub-picture area is not selected ("no" in operation S1240), the background area can be decoded and displayed.

[0160] In an embodiment, a network middlebox (such as a router) can select a subset of layers to send to a user based on its bandwidth. The organization of pictures / sub-pictures can be used for bandwidth adaptation. For example, if a user does not have the bandwidth, the router strips off some layers or selects some sub-pictures based on the importance of the layer or sub-picture or based on the settings used, which can be done dynamically to adapt to the bandwidth.

[0161] Fig.13 An embodiment related to a use case of 360-degree video is shown. When a spherical 360-degree picture, such as picture 1301, is projected onto a flat picture, the projected 360-degree picture can be used as a base layer and divided into multiple sub-pictures. For example, the multiple sub-pictures can include a rear sub-picture, a top sub-picture, a right sub-picture, a left sub-picture, a front sub-picture, and a bottom sub-picture. The enhancement layer of a specific sub-picture, such as the front sub-picture, can be encoded and transmitted to a client. The decoder is capable of decoding a base layer including all sub-pictures and an enhancement layer of a selected sub-picture. When the current viewport is the same as the selected sub-picture, the quality of the displayed picture can be higher than the quality of the decoded sub-picture with the enhancement layer. Otherwise, the decoded picture with the base layer can be displayed at a lower quality.

[0162] In an embodiment, any layout information for display may be present in the file as supplemental information (such as SEI messages or metadata). One or more decoded sub-pictures may be repositioned and displayed based on the signaled layout information. The layout information may be signaled by the streaming server or broadcaster, or may be regenerated by a network entity or cloud server, or may be determined by a user's customized settings.

[0163] In an embodiment, when the input picture is divided into one or more (rectangular) sub-regions, each sub-region can be encoded as an independent layer. Each independent layer corresponding to a local area can have a unique layer_id value. For each independent layer, the sub-picture size and position information can be signaled. For example, the picture size (width, height), the offset information of the upper left corner (x_offset, y_offset). Fig.14An example of the layout of the divided sub-pictures, their sub-picture size and position information, and their corresponding picture prediction structure is shown. The layout information may be signaled in a high-level syntax structure (such as one or more parameter sets, a slice header or a tile group header, or an SEI message), and the layout information includes one or more sub-picture sizes and one or more sub-picture positions.

[0164] In an embodiment, each sub-picture corresponding to an independent layer may have its unique POC value within the AU. When indicating a reference picture in a picture stored in the DPB by using one or more syntax elements in the RPS structure or the RPL structure, one or more POC values ​​for each sub-picture corresponding to the layer may be used.

[0165] In an embodiment, to indicate the (inter-layer) prediction structure, layer_id may not be used and the POC (delta) value may be used.

[0166] In an embodiment, a sub-picture corresponding to a layer (or a local area) with a POC value equal to N may or may not be used as a reference picture for motion compensated prediction of a sub-picture corresponding to the same layer (or the same local area) with a POC value equal to N+K. In most cases, the value of the number K may be equal to the maximum number of (independent) layers, which may be the same as the number of sub-regions.

[0167] In an embodiment, Fig.15 Shows Fig.14 When the input picture is divided into multiple (e.g., four) sub-regions, each local region can be encoded with one or more layers. In this case, the number of independent layers can be equal to the number of sub-regions, and one or more layers can correspond to the sub-regions. Therefore, each sub-region can be encoded with one or more independent layers and zero or more dependent layers.

[0168] In an embodiment, Fig.15 In , the input picture can be divided into four sub-regions. As an example, the upper right sub-region can be encoded into two layers, namely layer 1 and layer 4, and the lower right sub-region can be encoded into two layers, namely layer 3 and layer 5. In this case, layer 4 can refer to layer 1 for motion compensation prediction, and layer 5 can refer to layer 3 for motion compensation.

[0169] In an embodiment, in-loop filtering (such as deblocking filtering, adaptive in-loop filtering, shaper, bilateral filtering or any deep learning based filtering) across layer boundaries may (optionally) be disabled.

[0170] In an embodiment, motion compensated prediction or intra-block copy across layer boundaries may (optionally) be disabled.

[0171] In an embodiment, boundary filling for motion compensated prediction or in-loop filtering at sub-picture boundaries may be optionally processed. A flag may be signaled in a high-level syntax structure (such as one or more parameter sets (VPS, SPS, PPS, or APS), a slice or tile group header, or a SEI message) to indicate whether boundary filling is processed.

[0172] In an embodiment, layout information of one or more sub-regions (or one or more sub-pictures) may be signaled in a VPS or an SPS. Fig.16A shows an example of syntax elements in a VPS, Fig. 16B An example of a syntax element in an SPS is shown. In this example, vps_sub_picture_dividing_flag is signaled in the VPS. This flag may indicate whether one or more input pictures are divided into multiple sub-regions. When the value of vps_sub_picture_dividing_flag is equal to 0, one or more input pictures in one or more encoded video sequences corresponding to the current VPS may not be divided into multiple sub-regions. In this case, the input picture size may be equal to the encoded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples), which is signaled in the SPS. When the value of vps_sub_picture_dividing_flag is equal to 1, one or more input pictures may be divided into multiple sub-regions. In this case, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled in the VPS. The values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may be equal to the width and height of the input picture, respectively.

[0173] In an embodiment, the values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may not be used for decoding, but may be used for synthesis and display.

[0174] In an embodiment, when the value of vps_sub_picture_dividing_flag is equal to 1, the syntax elements pic_offset_x and pic_offset_y may be signaled in the SPS, corresponding to one or more specific layers. In this case, the coded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to the specific layer. Moreover, the position of the upper left corner of the sub-region (pic_offset_x, pic_offset_y) may be signaled in the SPS.

[0175] In an embodiment, the position information (pic_offset_x, pic_offset_y) of the upper left corner of the sub-region may not be used for decoding, but may be used for synthesis and display.

[0176] In the same or another embodiment, layout information (size and position), inter-layer dependency information of all or a subset of sub-regions of one or more input pictures may be signaled in a parameter set or SEI message. Fig.17 An example of a syntax element for indicating the layout of a sub-region, dependencies between layers, and information about the relationship between a sub-region and one or more layers is shown. In this example, the syntax element num_sub_region indicates the number of (rectangular) sub-regions in the current encoded video sequence. The syntax element num_layers indicates the number of layers in the current encoded video sequence. The value of num_layers may be equal to or greater than the value of num_sub_region. When any sub-region is encoded as a single layer, the value of num_layers may be equal to the value of num_sub_region. When one or more sub-regions are encoded as multiple layers, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] indicates a dependency from the jth layer to the ith layer. num_layers_for_region[i] indicates the number of layers associated with the i-th sub-region. sub_region_layer_id[i][j] indicates the layer_id of the jth layer associated with the i-th sub-region. sub_region_offset_x[i] and sub_region_offset_y[i] indicate the horizontal position and vertical position of the upper left corner of the i-th sub-region, respectively. sub_region_width[i] and sub_region_height[i] indicate the width and height of the i-th sub-region, respectively.

[0177] In an embodiment, one or more syntax elements may be signaled in a high-level syntax structure (e.g., VPS, DPS, SPS, PPS, APS, or SEI message) that specifies an output layer set to indicate that one of the multiple layers is output with or without profile tier level information. Fig.18 , a syntax element num_output_layer_sets may be signaled in the VPS, indicating the number of output layer sets (OLS) in the coded video sequence referencing the VPS. For each output layer set, as many output_layer_flags as the number of output layers may be signaled.

[0178] In an embodiment, output_layer_flag[i] is equal to 1, which may specify that the i-th layer is output. vps_output_layer_flag[i] is equal to 0, which may specify that the i-th layer is not output.

[0179] In an embodiment, one or more syntax elements may be signaled in a high-level syntax structure (e.g., VPS, DPS, SPS, PPS, APS, or SEI message) that specify the profile level information for each output layer set. Fig.18 , a syntax element num_profile_tile_level may be signaled in the VPS, indicating the number of profile layer level information for each OLS in the coded video sequence referring to the VPS. For each output layer set, as many sets of syntax elements for profile layer level information as the number of output layers may be signaled or an index indicating a specific profile layer level information among the profile layer level information entries.

[0180] In an embodiment, profile_tier_level_idx[i][j] may specify the index of a profile_tier_level() syntax structure in a list of profile_tier_level() syntax structures in a VPS that applies to the jth layer of the ith OLS.

[0181] In the embodiments, reference Fig.19 , when the maximum number of layers is greater than 1 (vps_max_layers_minus1>0), the syntax elements num_profile_tile_level and / or num_output_layer_sets may be signaled.

[0182] In the embodiments, reference Fig.19The syntax element vps_output_layers_mode[i], used to indicate the output layer signaling mode of the i-th output layer set, can be present in the VPS.

[0183] In an embodiment, vps_output_layers_mode[i] is equal to 0, which may specify that only the highest layer of the i-th output layer set is output. vps_output_layer_mode[i] is equal to 1, which may specify that all layers of the i-th output layer set are output. vps_output_layer_mode[i] is equal to 2, which may specify that the output layer is the layer in the i-th output layer set with vps_output_layer_flag[i][j] equal to 1. More values ​​may be reserved.

[0184] In an embodiment, output_layer_flag[i][j] may or may not be signaled, depending on the value of vps_output_layers_mode[i] for the i-th output layer set.

[0185] In the embodiments, reference Fig.19 , there may be a flag vps_ptl_signal_flag[i] for the i-th output layer set. Depending on the value of vps_ptl_signal_flag[i], the profile layer level information of the i-th output layer set may or may not be signaled.

[0186] In the embodiments, reference Fig. 20 The number of sub-pictures in the current CVS, max_subpics_minus1, may be signaled in a high-level syntax structure (eg, VPS, DPS, SPS, PPS, APS, or SEI message).

[0187] In the embodiments, reference Fig. 20 , when the number of sub-pictures is greater than 1 (max_subpics_minus1>0), the sub-picture identifier sub_pic_id[i] of the i-th sub-picture may be signaled.

[0188] In an embodiment, one or more syntax elements indicating sub-picture identifiers of each layer belonging to each output layer set may be signaled in the VPS. Fig. 20 , sub_pic_id_layer[i][j][k] indicates the kth sub-picture present in the jth layer of the i-th output layer set. Using this information, the decoder can identify which sub-picture can be decoded and output for each layer of a specific output layer set.

[0189] In an embodiment, a picture header (PH) is a syntax structure containing syntax elements that apply to all slices of a coded picture. A picture unit (PU) is a group of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain exactly one coded picture. A PU may contain a picture header (PH) and one or more VCL (video coding layer) NAL units that make up a coded picture.

[0190] In an embodiment, the SPS (RBSP (Raw Byte Sequence Payload)) may be used by the decoding process before it is referenced, included in at least one AU with TemporalId equal to 0, or provided by an external device.

[0191] In an embodiment, the SPS (RBSP) may be used by the decoding process before it is referenced, included in at least one AU with TemporalId equal to 0 in the CVS, or provided by an external device, the CVS containing one or more PPSs referencing the SPS.

[0192] In an embodiment, the SPS (RBSP) may be used by the decoding process before it is referenced by one or more PPSs, included in at least one PU whose nuh_layer_id is equal to the lowest nuh_layer_id value of the PPS NAL unit referencing the SPS NAL unit in the CVS, or provided by an external device, and the CVS contains one or more PPSs referencing the SPS.

[0193] In an embodiment, the SPS (RBSP) may be used by the decoding process before it is referenced by one or more PPSs, included in at least one PU, or provided by an external device, wherein the at least one PU has a TemporalId equal to 0 and a nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL unit.

[0194] In an embodiment, the SPS (RBSP) may be used by the decoding process before it is referenced by one or more PPSs, included in at least one PU in the CVS, or provided by an external device, wherein the TemporalId of the at least one PU is equal to 0 and the nuh_layer_id is equal to the lowest nuh_layer_id value of the PPS NAL unit that references the SPS NAL unit, and the CVS contains one or more PPSs that reference the SPS.

[0195] In the same or another embodiment, pps_seq_parameter_set_id may specify the value of sps_seq_parameter_set_id of the referenced SPS.The value of pps_seq_parameter_set_id in all PPSs referenced by coded pictures in a layer-by-layer coded video sequence (CLVS) may be the same.

[0196] In the same or another embodiment, all SPS NAL units with a particular value of sps_seq_parameter_set_id in a CVS may have the same content.

[0197] In the same or another embodiment, SPS NAL units may share the same value space of sps_seq_parameter_set_id regardless of the nuh_layer_id value.

[0198] In the same or another embodiment, the nuh_layer_id value of an SPS NAL unit may be equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL unit.

[0199] In an embodiment, when an SPS with nuh_layer_id equal to m is referenced by one or more PPSs with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0200] In an embodiment, the PPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit, or provided by an external device.

[0201] In an embodiment, the PPS (RBSP) may be used by the decoding process before it is referenced, included in at least one AU whose TemporalId is equal to the TemporalId of the PPS NAL unit in the CVS, or provided by an external device, and the CVS contains one or more PHs (or coded slice NAL units) that reference the PPS.

[0202] In an embodiment, the PPS (RBSP) may be used by the decoding process before it is referenced by one or more PHs (or coded slice NAL units), included in at least one PU whose nuh_layer_id is equal to the lowest nuh_layer_id value of the coded slice NAL units referencing the PPS NAL unit in the CVS, or provided by an external device, and the CVS contains one or more PHs (or coded slice NAL units) referencing the PPS.

[0203] In an embodiment, the PPS (RBSP) may be used by the decoding process before it is referenced by one or more PHs (or coded slice NAL units), included in at least one PU in the CVS, or provided by an external device, wherein the TemporalId of the at least one PU is equal to the TemporalId of the PPS NAL unit and the nuh_layer_id is equal to the lowest nuh_layer_id value of the coded slice NAL unit that references the PPS NAL unit, and the CVS contains one or more PHs (or coded slice NAL units) that reference the PPS.

[0204] In the same or another embodiment, ph_pic_parameter_set_id in the PH may specify the value of pps_pic_parameter_set_id for the referenced PPS in use.The value of pps_seq_parameter_set_id in all PPSs referenced by coded pictures in the CLVS may be the same.

[0205] In the same or another embodiment, all PPS NAL units within a PU with a particular value of pps_pic_parameter_set_id shall have the same content.

[0206] In the same or another embodiment, regardless of the nuh_layer_id value, PPS NAL units may share the same value space for pps_pic_parameter_set_id.

[0207] In the same or another embodiment, the nuh_layer_id value of a PPS NAL unit may be equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the NAL unit that references the PPS NAL unit.

[0208] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0209] In an embodiment, the PPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit, or provided by an external device.

[0210] In an embodiment, the PPS (RBSP) may be used by the decoding process before it is referenced, included in at least one AU whose TemporalId is equal to the TemporalId of the PPS NAL unit in the CVS, or provided by an external device, and the CVS contains one or more PHs (or coded slice NAL units) that reference the PPS.

[0211] In an embodiment, the PPS (RBSP) may be used by the decoding process before it is referenced by one or more PHs (or coded slice NAL units), included in at least one PU whose nuh_layer_id is equal to the lowest nuh_layer_id value of the coded slice NAL units referencing the PPS NAL unit in the CVS, or provided by an external device, and the CVS contains one or more PHs (or coded slice NAL units) referencing the PPS.

[0212] In an embodiment, the PPS (RBSP) may be used by the decoding process before it is referenced by one or more PHs (or coded slice NAL units), included in at least one PU in the CVS, or provided by an external device, wherein the TemporalId of the at least one PU is equal to the TemporalId of the PPS NAL unit and the nuh_layer_id is equal to the lowest nuh_layer_id value of the coded slice NAL unit that references the PPS NAL unit, and the CVS contains one or more PHs (or coded slice NAL units) that reference the PPS.

[0213] In the same or another embodiment, ph_pic_parameter_set_id in the PH may specify the value of pps_pic_parameter_set_id in use for the referenced PPS.The value of pps_seq_parameter_set_id in all PPSs referenced by coded pictures in the CLVS may be the same.

[0214] In the same or another embodiment, all PPS NAL units within a PU with a particular value of pps_pic_parameter_set_id may have the same content.

[0215] In the same or another embodiment, regardless of the nuh_layer_id value, PPS NAL units may share the same value space for pps_pic_parameter_set_id.

[0216] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the NAL unit that references the PPS NAL unit.

[0217] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n or the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0218] When a picture is encoded into a codestream that includes one or more layers with different resolutions, each coded video sequence of a layer may use reference picture resampling when the reference picture may have a different resolution than the current picture. It is desirable to specify syntax elements that indicate reference picture resampling for spatial scalability or adaptive picture resolution change (ARC).

[0219] refer to Fig.21 , output layer may indicate a layer in an output layer set that is output. An output layer set (OLS) may indicate a set of layers including a specified set of layers, wherein one or more layers in the set of layers are designated as output layers. An output layer set (OLS) layer index is an index of a layer in the OLS to a list of layers in the OLS.

[0220] Sublayer may indicate a temporal scalable layer of a temporal scalable codestream, including VCL NAL units and associated non-VCL NAL units with a specific value of the TemporalId variable.Sublayer representation may indicate a subset of a codestream including NAL units of a specific sublayer and lower sublayers.

[0221] The VPS RBSP may be available to the decoding process before it is referenced, included in at least one AU with TemporalId equal to 0, or provided by an external device. All VPS NAL units in the CVS with a specific value of vps_video_parameter_set_id may have the same content.

[0222] like Fig. 22 As shown in , in an embodiment, vps_video_parameter_set_id provides an identifier of the VPS for reference by other syntax elements. The value of vps_video_parameter_set_id may be greater than 0.

[0223] In an embodiment, vps_max_layers_minus1 plus 1 may specify the maximum allowed number of layers in each CVS of the reference VPS.

[0224] vps_max_sublayer_minus1 plus 1 may specify the maximum number of temporal sublayers that may exist in a layer in each CVS of a reference VPS. The value of vps_max_sublayer_minus1 may be in the range of 0 to 6, inclusive.

[0225] In an embodiment, vps_all_layers_same_num_sublayers_flag equal to 1 may specify that the number of temporal sublayers for all layers in each CVS of the referenced VPS is the same. vps_all_layers_same_num_sublayers_flag equal to 0 may specify that the layers in each CVS of the referenced VPS may or may not have the same number of temporal sublayers. When vps_all_layers_same_num_sublayers_flag is not present, its value may be inferred to be equal to 1.

[0226] In an embodiment, vps_all_independent_layers_flag equal to 1 may specify that all layers in the CVS are independently coded without using inter-layer prediction.

[0227] In an embodiment, vps_all_independent_layers_flag equal to 0 may specify that one or more layers in the CVS may use inter-layer prediction. When vps_all_independent_layers_flag is not present, its value may be inferred to be equal to 1.

[0228] In an embodiment, vps_layer_id[i] may specify the nuh_layer_id value of the i-th layer. For any two non-negative integer values ​​of m and n, when m is less than n, the value of vps_layer_id[m] may be less than vps_layer_id[n].

[0229] In an embodiment, vps_independent_layer_flag[i] equal to 1 may specify that the layer with index i does not use inter-layer prediction. vps_independent_layer_flag[i] equal to 0 may specify that the layer with index i may use inter-layer prediction, and the syntax element vps_direct_ref_layer_flag[i][j] (j is in the range of 0 to i-1 (inclusive)) is present in the VPS. When vps_independent_layer_flag[i] is not present, its value may be inferred to be equal to 1.

[0230] In an embodiment, vps_direct_ref_layer_flag[i][j] equal to 0 may specify that the layer with index j is not a direct reference layer of the layer with index i. vps_direct_ref_layer_flag[i][j] equal to 1 may specify that the layer with index j is a direct reference layer of the layer with index i. When vps_direct_ref_layer_flag[i][j] (i and j are in the range of 0 to vps_max_layers_minus1 (inclusive)) is not present, it may be inferred to be equal to 0. When vps_independent_layer_flag[i] is equal to 0, there may be at least one value of j in the range of 0 to i-1 (inclusive) such that the value of vps_direct_ref_layer_flag[i][j] is equal to 1.

[0231] The variables NumDirectRefLayers[i], DirectRefLayerIdx[i][d], NumRefLayers[i], RefLayerIdx[i][r], and LayerUsedAsRefLayerFlag[j] can be derived as follows:

[0232] for(i=0;i<=vps_max_layers_minus1;i++){

[0233] for(j=0;j<=vps_max_layers_minus1;j++){

[0234] dependencyFlag[i][j]=vps_direct_ref_layer_flag[i][j]

[0235] for(k=0;k <i;k++)

[0236] if(vps_direct_ref_layer_flag[i][k]&&dependencyFlag[k][j])

[0237] dependencyFlag[i][j]=1

[0238] }

[0239] LayerUsedAsRefLayerFlag[i]=0

[0240] }

[0241] for(i=0;i<=vps_max_layers_minus1;i++){

[0242] for(j=0,d=0,r=0;j<=vps_max_layers_minus1;j++)

[0243] if(vps_direct_ref_layer_flag[i][j]){

[0244] DirectRefLayerIdx[i][d++]=j

[0245] LayerUsedAsRefLayerFlag[j]=1

[0246] }

[0247] if(dependencyFlag[i][j])

[0248] RefLayerIdx[i][r++]=j

[0249] }

[0250] NumDirectRefLayers[i] = d

[0251] NumRefLayers[i] = r

[0252] }

[0253] The variable GeneralLayerIdx[i] that specifies the layer index of the layer whose nuh_layer_id is equal to vps_layer_id[i] can be derived as follows:

[0254] for(i=0;i<=vps_max_layers_minus1;i++)

[0255] GeneralLayerIdx[vps_layer_id[i]]=i

[0256] For any two different values ​​of i and j (i, j are both in the range of 0 to vps_max_layers_minus1 (including the end values)), when dependencyFlag[i][j] is equal to 1, the requirement for code stream consistency can be that the values ​​of chroma_format_idc and bit_depth_minus8 applied to the i-th layer can be equal to the values ​​of chroma_format_idc and bit_depth_minus8 applied to the j-th layer, respectively.

[0257] max_tid_ref_present_flag[i] equal to 1 may specify that the syntax element max_tid_il_ref_pics_plus1[i] is present. max_tid_ref_present_flag[i] equal to 0 may specify that the syntax element max_tid_il_ref_pics_plus1[i] is not present.

[0258] max_tid_il_ref_pics_plus1[i] equal to 0 may specify that non-IRAP pictures of layer i do not use inter-layer prediction. max_tid_il_ref_pics_plus1[i] greater than 0 may specify that for decoding of pictures of layer i, pictures with TemporalId greater than max_tid_il_ref_pics_plus1[i]-1 are not used as ILRPs. When max_tid_il_ref_pics_plus1[i] is not present, its value may be inferred to be 7.

[0259] each_layer_is_an_ols_flag equal to 1 may specify that each OLS contains only one layer, and that each layer in the CVS of the referenced VPS is itself an OLS, where the single included layer is the only output layer. each_layer_is_an_ols_flag equal to 0 may indicate that an OLS may contain more than one layer. If vps_max_layers_minus1 is equal to 0, the value of each_layer_is_an_ols_flag may be inferred to be equal to 1. Otherwise, when vps_all_independent_layers_flag is equal to 0, the value of each_layer_is_an_ols_flag may be inferred to be equal to 0.

[0260] ols_mode_idc equal to 0 may specify that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (inclusive), and for each OLS only the highest layer in the OLS is output.

[0261] ols_mode_idc equal to 1 may specify that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (inclusive), and all layers in the OLS are output for each OLS.

[0262] ols_mode_idc equal to 2 may specify that the total number of OLSs specified by the VPS is explicitly signaled, and for each OLS, its output layer is explicitly signaled, and the other layers are layers that are direct reference layers or indirect reference layers of the output layer of the OLS.

[0263] The value of ols_mode_idc may be in the range of 0 to 2, inclusive. The value 3 of ols_mode_idc is reserved for future use by ITU-T | ISO / IEC.

[0264] When vps_all_independent_layers_flag is equal to 1 and each_layer_is_an_ols_flag is equal to 0, the value of ols_mode_idc can be inferred to be equal to 2.

[0265] num_output_layer_sets_minus1 plus 1 can specify the total number of OLSs specified by the VPS when ols_mode_idc is equal to 2.

[0266] The variable TotalNumOlss specifying the total number of OLSs specified by VPS can be derived as follows:

[0267] if(vps_max_layers_minus1==0)

[0268] TotalNumOlss=1

[0269] else if(each_layer_is_an_ols_flag||ols_mode_idc==0||ols_mode_idc==1)

[0271] TotalNumOlss=vps_max_layers_minus1+1

[0272] else if (ols_mode_idc == 2)

[0273] TotalNumOlss=num_output_layer_sets_minus1+1

[0274] When ols_mode_idc is equal to 2, ols_output_layer_flag[i][j] is equal to 1 to specify that the layer with nuh_layer_id equal to vps_layer_id[j] is the output layer of the i-th OLS. When ols_mode_idc is equal to 2, ols_output_layer_flag[i][j] is equal to 0 to specify that the layer with nuh_layer_id equal to vps_layer_id[j] is not the output layer of the i-th OLS.

[0275] The variable NumOutputLayersInOls[i] specifying the number of output layers in the i-th OLS, the variable NumSubLayersInLayerInOLS[i][j] specifying the number of sublayers in the j-th layer in the i-th OLS, the variable OutputLayerIdInOls[i][j] specifying the nuh_layer_id value of the j-th output layer in the i-th OLS, and the variable LayerUsedAsOutputLayerFlag[k] specifying whether the k-th layer is used as an output layer in at least one OLS can be derived as follows:

[0276]

[0277]

[0278] For each value of i in the range of 0 to vps_max_layers_minus1 (inclusive), the values ​​of LayerUsedAsRefLayerFlag[i] and LayerUsedAsOutputLayerFlag[i] may not both be equal to 0. In other words, there may be no layer that is neither an output layer of at least one OLS nor a direct reference layer of any other layer.

[0279] For each OLS, there can be at least one layer that is an output layer. In other words, for any value of i in the range of 0 to TotalNumOlss-1 (inclusive), the value of NumOutputLayersInOls[i] can be greater than or equal to 1.

[0280] The variable NumLayersInOls[i] specifying the number of layers in the i-th OLS and the variable LayerIdInOls[i][j] specifying the nuh_layer_id value of the j-th layer in the i-th OLS can be derived as follows:

[0281]

[0282]

[0283] The variable OlsLayerIdx[i][j] that specifies the OLS layer index of the layer whose nuh_layer_id is equal to LayerIdInOls[i][j] is derived as follows:

[0284] The lowest layer in each OLS may be an independent layer. In other words, the value of vps_independent_layer_flag[GeneralLayerIdx[LayerIdInOls[i][0]]] may be equal to 1 for each i in the range of 0 to TotalNumOlss-1 (inclusive).

[0285] Each layer may be included in at least one OLS specified by the VPS. In other words, for each layer for which a particular value of nuh_layer_id, nuhLayerId, is equal to one of vps_layer_id[k], where k is in the range of 0 to vps_max_layers_minus1, inclusive, there may be at least one pair of values ​​i and j, where i is in the range of 0 to TotalNumOlss-1, inclusive, and j is in the range of NumLayersInOls[i]-1, inclusive, such that the value of LayerIdInOls[i][j] is equal to nuhLayerId.

[0286] In an embodiment, for the current picture CurrPic, the decoding process operates as follows:

[0287] - You can set the PictureOutputFlag as follows:

[0288] -PictureOutputFlag can be set equal to 0 if one of the following conditions is true:

[0289] -The current image is a RASL image and the associated IRAP image

[0290] NoOutputBeforeRecoveryFlag is equal to 1.

[0291] -gdr_enabled_flag is equal to 1 and the current picture has NoOutputBeforeRecoveryFlag

[0292] GDR image with value equal to 1.

[0293] -gdr_enabled_flag is equal to 1, the current picture is associated with a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, and the PicOrderCntVal of the current picture is less than the associated GDR

[0294] RpPicOrderCntVal of the picture.

[0295] -sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0 and the current AU contains a picture picA that satisfies all of the following conditions:

[0296] - PicA has PictureOutputFlag equal to 1.

[0297] - PicA has a nuh_layer_id nuhLid greater than nuh_layer_id nuhLid of the current picture.

[0298] - PicA belongs to the output layer of OLS (i.e., OutputLayerIdInOls[TargetOlsIdx][0]

[0299] equal to nuhLid).

[0300] -sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 2, and

[0301] ols_output_layer_flag[TargetOlsIdx][GeneralLayerIdx[nuh_layer_id]] is equal to 0.

[0302] - Otherwise, PictureOutputFlag can be set equal to pic_output_flag.

[0303] After all slices of the current picture are decoded, the current decoded picture is marked as "used for short-term reference", and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as "used for short-term reference".

[0304] In an embodiment, when each layer is an output layer set, PictureOutputFlag may be set equal to pic_output_flag regardless of the value of ols_mode_idc.

[0305] In an embodiment, PictureOutputFlag may be set equal to 0 when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is equal to 0, ols_mode_idc is equal to 0, and the current AU contains a picture picA that satisfies all of the following conditions: PicA has a PictureOutputFlag equal to 1, PicA has a nuh_layer_id nuhLid greater than the nuh_layer_id nuhLid of the current picture, and PicA belongs to the output layer of the OLS (i.e., OutputLayerIdInOls[TargetOlsIdx][0] is equal to nuhLid).

[0306] In an embodiment, when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is equal to 0, ols_mode_idc is equal to 2, and ols_output_layer_flag[TargetOlsIdx][GeneralLayerIdx[nuh_layer_id]] is equal to 0, PictureOutputFlag may be set equal to 0.

[0307] Reference picture resampling can enable adaptive resolution changes within a coded (layered) video sequence, as well as spatial scalability across layers with dependencies between layers belonging to the same output layer set.

[0308] In an embodiment, Fig.23As shown, sps_ref_pic_resampling_enabled_flag is signaled in a parameter set (e.g., a sequence parameter set). The flag sps_ref_pic_resampling_enabled_flag may indicate whether reference picture resampling is used for adaptive resolution change within a coded video sequence that references the SPS, or for spatial scalability across layers. sps_ref_pic_resampling_enabled_flag equal to 1 may specify that reference picture resampling is enabled, and that one or more slices of a picture in the CLVS reference a reference picture with a different spatial resolution in an active entry in a reference picture list. sps_ref_pic_resampling_enabled_flag equal to 0 may specify that reference picture resampling is disabled, and that no slice of a picture in the CLVS references a reference picture with a different spatial resolution in an active entry in a reference picture list.

[0309] In an embodiment, when sps_ref_pic_resampling_enabled_flag is equal to 1, for a current picture, a reference picture having a different spatial resolution therefrom belongs to the same layer or a different layer as the layer containing the current picture.

[0310] In another embodiment, sps_ref_pic_resampling_enabled_flag equal to 1 may specify that reference picture resampling is enabled, and one or more slice reference picture lists of a picture in the CLVS reference a reference picture with a different spatial resolution or a different scaling window than the active entry. sps_ref_pic_resampling_enabled_flag equal to 0 may specify that reference picture resampling is disabled, and no slice reference picture lists of a picture in the CLVS reference a reference picture with a different spatial resolution or a different scaling window than the active entry.

[0311] In an embodiment, when sps_ref_pic_resampling_enabled_flag is equal to 1, for a current picture, a reference picture having a different spatial resolution or a different scaling window from the current picture belongs to the same layer or a different layer as the layer containing the current picture.

[0312] In an embodiment, sps_res_change_in_clvs_allowed_flag may indicate whether the picture resolution changes within a CLVS or CVS. sps_res_change_in_clvs_allowed_flag equal to 1 may specify that the picture spatial resolution may change within a CLVS referencing an SPS. sps_res_change_in_clvs_allowed_flag equal to 0 may specify that the picture spatial resolution does not change within any CLVS referencing an SPS. When sps_res_change_in_clvs_allowed_flag is not present, its value may be inferred to be equal to 0.

[0313] In an embodiment, when sps_ref_pic_resampling_enabled_flag is equal to 1 and sps_res_change_in_clvs_allowed_flag is equal to 0, reference picture resampling may be used only for spatial scalability within CLVS but not for adaptive resolution change.

[0314] In an embodiment, when sps_ref_pic_resampling_enabled_flag is equal to 1 and sps_res_change_in_clvs_allowed_flag is equal to 1, reference picture resampling may be used for both spatial scalability and adaptive resolution change within CLVS.

[0315] When sps_ref_pic_resampling_enabled_flag is equal to 1, sps_res_change_in_clvs_allowed_flag is equal to 0, and sps_video_parameter_set_id is equal to 0, pps_scaling_window_explicit_signalling_flag may be equal to 1. This means that when the picture resolution is constant within CLVS or CVS and reference picture resampling is used, the scaling window parameters need to be explicitly signaled instead of inferring their values ​​from the consistent window parameters.

[0316] In an embodiment, sps_virtual_boundaries_present_flag is signaled in the SPS, such as Fig.23 The flag sps_virtual_boundaries_present_flag may indicate whether virtual boundary information is signaled in the SPS.

[0317] In an embodiment, sps_virtual_boundaries_present_flag is conditionally signaled only when sps_res_change_in_clvs_allowed_flag is equal to 0, because when reference picture resampling is used, virtual boundary information may not be signaled in the SPS.

[0318] In an embodiment, sps_virtual_boundaries_present_flag equal to 1 can specify that information about virtual boundaries is signaled in the SPS. sps_virtual_boundaries_present_flag equal to 0 can specify that information about virtual boundaries is not signaled in the SPS. When one or more virtual boundaries are signaled in the SPS, in-loop filtering operations are disabled for virtual boundaries in pictures that reference the SPS. In-loop filtering operations include deblocking filter operations, sample adaptive offset filter operations, and adaptive in-loop filter operations. When sps_virtual_boundaries_present_flag is not present, it can be inferred that its value is equal to 0.

[0319] In an embodiment, sps_subpic_info_present_flag is signaled in the SPS, such as Fig.23 The flag sps_subpic_info_present_flag may indicate whether sub-picture partition information is signaled in the SPS.

[0320] In an embodiment, sps_subpic_info_present_flag is conditionally signaled only when sps_res_change_in_clvs_allowed_flag is equal to 0, because sub-picture partitioning information may not be signaled in the SPS when reference picture resampling is used.

[0321] In an embodiment, sps_subpic_info_present_flag equal to 1 may specify that for CLVS, sub-picture information is present, and one or more sub-pictures may be present in each picture of the CLVS. sps_subpic_info_present_flag equal to 0 may specify that for CLVS, sub-picture information is not present, and only one sub-picture may be present in each picture of the CLVS. When sps_subpic_info_present_flag is not present, its value may be inferred to be equal to 0.

[0322] In an embodiment, pps_res_change_in_clvs_allowed_flag may be signaled in the PPS, such as Fig.24 As shown in . The value of pps_res_change_in_clvs_allowed_flag in the PPS may be equal to the value of sps_res_change_in_clvs_allowed_flag in the SPS referenced by the PPS.

[0323] In the same embodiment, picture width and height information may be signaled in the PPS only when the value of pps_res_change_in_clvs_allowed_flag is equal to 1. When pps_res_change_in_clvs_allowed_flag is equal to 0, the picture width and height values ​​may be inferred to be equal to the picture maximum width and picture maximum height values ​​signaled in the SPS.

[0324] In an embodiment, pps_pic_width_in_luma_samples may specify the width in luma samples of each decoded picture that references the PPS. pps_pic_width_in_luma_samples shall not be equal to 0, shall be an integer multiple of Max(8,MinCbSizeY), and shall be less than or equal to sps_pic_width_max_in_luma_samples. When pps_pic_width_in_luma_samples is not present, its value may be inferred to be equal to sps_pic_width_max_in_luma_samples. When sps_ref_wraparound_enabled_flag is equal to 1, the value of (CtbSizeY / MinCbSizeY+1) shall be less than or equal to the value of (pps_pic_width_in_luma_samples / MinCbSizeY-1). pps_pic_height_in_luma_samples may specify the height in luma samples of each decoded picture that references the PPS. pps_pic_height_in_luma_samples shall not be equal to 0 and shall be an integer multiple of Max(8,MinCbSizeY) and shall be less than or equal to sps_pic_height_max_in_luma_samples. When pps_pic_height_in_luma_samples is not present, its value may be inferred to be equal to sps_pic_height_max_in_luma_samples.

[0325] FIG. 25A to FIG. 25C 2500A, 2500B, and 2500C are flowcharts illustrating example methods for decoding an encoded video stream. In some embodiments, FIG. 25A to FIG. 25C One or more method blocks of may be performed by the decoder 210. In some implementations, FIG. 25A to FIG. 25C One or more method blocks of may be performed by another device or a group of devices (such as encoder 203) separate from or including the decoder 210.

[0326] like Fig.25A As shown in , method 2500A may include obtaining a first flag from a coded video bitstream, the first flag indicating that reference picture resampling is enabled for a coded video sequence (CVS) (block 2511). In an embodiment, the first flag may correspond to sps_ref_pic_resampling_enabled_flag.

[0327] like Fig.25A As further shown in FIG. 25, method 2500A may include determining a value of a second flag indicating whether the picture resolution is changed in the CVS (block 2512). In an embodiment, the second flag may correspond to sps_res_change_in_clvs_allowed_flag.

[0328] like Fig.25A As further shown in FIG. 25, method 2500A may include determining whether the value of the second flag indicates whether the picture resolution has changed in the CVS (block 2513).

[0329] like Fig.25A As further shown in , method 2500A may include: based on the value of the second flag indicating that the picture resolution is changed in the CVS ("yes" at box 2513), decoding the CVS by resampling the reference picture for spatial scalability and adaptive resolution change (ARC) (box 2514).

[0330] like Fig.25A As further shown in, method 2500A may include: based on the value of a second flag indicating that the picture resolution does not change in the CVS ("No" at box 2513), decoding the CVS (box 2515) by resampling the reference picture for spatial scalability but not resampling the reference picture for ARC.

[0331] like Fig.25B As shown in , method 2500B may include determining whether the value of the second flag indicates whether the picture resolution has changed in the CVS (block 2523). In an embodiment, block 2523 of method 2500B may correspond to block 2513 of method 2500A.

[0332] like Fig.25B As further shown in FIG. 25, method 2500B may include: based on the value of the second flag indicating that the picture resolution is not changed in the CVS (“No” at block 2523), obtaining a third flag indicating whether virtual boundary information is signaled (block 2525). In an embodiment, the third flag may correspond to sps_virtual_boundaries_present_flag.

[0333] like Fig.25B As further shown in , method 2500B may include: based on the value of the second flag indicating that the picture resolution is changed in the CVS ("yes" at block 2523), the third flag may not be signaled (block 2524).

[0334] like Fig.25C As shown in , method 2500C may include determining whether the value of the second flag indicates whether the picture resolution has changed in the CVS (block 2533). In an embodiment, block 2533 of method 2500C may correspond to block 2513 of method 2500A.

[0335] like Fig.25C As further shown in FIG. 25, method 2500C may include: based on the value of the second flag indicating that the picture resolution is not changed in the CVS (“No” at block 2533), obtaining a fourth flag indicating whether sub-picture partition information is signaled (block 2535). In an embodiment, the fourth flag may correspond to sps_subpic_info_present_flag.

[0336] like Fig.25C As further shown in , method 2500C may include: based on the value of the second flag indicating that the picture resolution is changed in the CVS ("yes" at block 2533), the fourth flag may not be signaled (block 2534).

[0337] In an embodiment, one or more blocks of methods 2500A, 2500B, and 2500C may be combined in any order. FIG. 25A to FIG. 25C Example blocks of methods 2500A, 2500B, and 2500C are shown, but in some embodiments, methods 2500A, 2500B, and 2500C may include FIG. 25A to FIG. 25C In some embodiments, the method 2500A, 2500B, and 2500C may be executed in parallel in addition to, fewer than, different from, or arranged differently from those depicted in the method 2500A, 2500B, and 2500C. Additionally or alternatively, two or more of the blocks in the method 2500A, 2500B, and 2500C may be executed in parallel.

[0338] In addition, the proposed method can be implemented by a processing circuit (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable storage medium to perform one or more of the proposed methods.

[0339] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable storage media. Fig.26 A computer system 2600 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0340] The computer software may be encoded using any suitable machine code or computer language, which may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., either directly or through interpretation, microcode execution, etc.

[0341] The instructions may be executed on various types of computers or computer components, including, for example, personal computers, tablets, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0342] Fig.26 The components for computer system 2600 shown in the example are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present application. Nor should the configuration of components be interpreted as having any dependency or requirement on any one or combination of components shown in the exemplary embodiment of computer system 2600.

[0343] Computer system 2600 may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, taps), visual input (e.g., gestures), olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that may not be directly related to a person's conscious input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0344] Input human interface devices may include one or more of the following (only one of each is depicted): keyboard 2601 , mouse 2602 , track pad 2603 , touch screen 2610 and associated graphics adapter 2650 , data gloves, joystick 2605 , microphone 2606 , scanner 2607 , camera 2608 .

[0345] The computer system 2600 may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen 2610, a data glove, or a joystick 2605, although there may also be tactile feedback devices that do not serve as input devices), audio output devices (e.g., speakers 2609, headphones (not depicted)), visual output devices (e.g., screens 2610, including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light emitting diode (OLED) screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which are capable of outputting two-dimensional visual output or output greater than three dimensions through, for example, stereographic output; virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)), and printers (not depicted).

[0346] The computer system 2600 may also include human-accessible storage devices and associated media for the storage devices, such as optical media including CD / DVD ROM / RW 2620 with CD / DVD etc. media 2621, thumb drive 2622, removable hard drive or solid state drive 2623, legacy magnetic media such as tapes and floppy disks (not depicted), ROM / application specific integrated circuit (ASIC) / programmable logic device (PLD) based special devices such as security protection devices (not depicted), and the like.

[0347] Those skilled in the art will also appreciate that the term "computer-readable storage medium" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transient signals.

[0348] The computer system 2600 may also include an interface to one or more communication networks (2655). The network may be, for example, wireless, wired, optical. The network may also be local, wide area, metropolitan, vehicle-mounted and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including Global System for Mobile Communications (GSM), third generation (3G), fourth generation (4G), fifth generation (5G), long term evolution (LTE), etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicle-mounted networks and industrial networks including controller area network bus (CANBus), and the like. Some networks typically require an external network interface adapter attached to some general data port or peripheral bus (2649) (e.g., a universal serial bus (USB) port of computer system 2600); other networks are typically integrated into the core of computer system 2600 by attaching to a system bus as described below (e.g., integrated into a PC computer system through an Ethernet interface, or integrated into a smart phone computer system through a cellular network interface). As an example, network 2655 can be connected to peripheral bus 2649 using network interface 2654. Using any of these networks, computer system 2600 can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast TV), one-way send only (e.g., CANBus connected to some CANBus devices), or bidirectional, for example, using a local area digital network or a wide area digital network to connect to other computer systems. Certain protocols and protocol stacks can be used on each of those networks and network interfaces (2654) as described above.

[0349] The above-mentioned human-machine interface device, human-accessible storage device, and network interface may be attached to the core 2640 of the computer system 2600 .

[0350] The core 2640 may include one or more central processing units (CPUs) 2641, graphics processing units (GPUs) 2642, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 2643, hardware accelerators for certain tasks 2644, and the like. These devices, along with read-only memory (ROM) 2645, random access memory 2646, and internal mass storage devices 2647 such as internal non-user accessible hard disk drives, solid-state drives (SSDs), etc., may be connected via a system bus 2648. In some computer systems, the system bus 2648 may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be attached to the core's system bus 2648 directly or via a peripheral bus 2649. Architectures for peripheral buses include peripheral component interconnect (PCI), USB, and the like.

[0351] The CPU 2641, GPU 2642, FPGA 2643, and accelerator 2644 may execute certain instructions, which in combination may constitute the above-mentioned computer code. The computer code may be stored in ROM 2645 or RAM 2646. Transient data may also be stored in RAM 2646, while permanent data may be stored, for example, in an internal mass storage device 2647. Fast storage and retrieval of any memory device may be achieved by using a cache memory, which may be closely associated with one or more CPUs 2641, GPUs 2642, mass storage devices 2647, ROM 2645, RAM 2646, etc.

[0352] The computer readable storage medium may have computer codes for performing various computer-implemented operations. The media and computer codes may be media and computer codes designed and constructed specifically for the purpose of this application, or may be of a type well known and available to technicians in the computer software field.

[0353] By way of example and not limitation, a computer system having architecture 2600 and in particular core 2640 may provide functionality resulting from the execution of software embodied in one or more tangible computer-readable storage media by a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such computer-readable storage media may be media associated with the user-accessible mass storage device introduced above and certain storage devices of the non-temporary nature of core 2640 (e.g., core internal mass storage device 2647 or ROM 2645). Software implementing various embodiments of the present application may be stored in such devices and executed by core 2640. Depending on specific needs, the computer-readable storage medium may include one or more memory devices or chips. The software may enable core 2640 and, in particular, the processor therein (including CPU, GPU, FPGA, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM 2646 and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality produced by logic hardwired or otherwise embodied in circuits (e.g., accelerator 2644), which may operate in place of or in conjunction with software to perform specific processes or specific portions of specific processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable storage media may encompass circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits embodying logic for execution, or both, where appropriate. The present application encompasses any suitable combination of hardware and software.

[0354] Although the present application describes several exemplary embodiments, various modifications, permutations and combinations, and various alternative equivalents are possible within the scope of the present application. Therefore, it should be understood that within the spirit and scope of the application, those skilled in the art can design various systems and methods that are not explicitly shown or described herein but can embody the principles of the present application.

Claims

1. A method for decoding a video code stream, characterized in that: The method comprises: Acquire a first flag from the video bitstream, where the first flag indicates enabling reference picture resampling for a coded video sequence CVS; In response to the first flag indicating that reference picture resampling is enabled for the CVS, determining, based on a second flag, whether picture resolution change is allowed in the CVS; When the second flag indicates that picture resolution change is allowed in the CVS, decoding the CVS by using reference picture resampling for spatial scalability and adaptive resolution change ARC; and When the value of the second flag indicates that the picture resolution does not change in the CVS, the CVS is decoded by resampling the reference picture for spatial scalability but not resampling the reference picture for ARC, and a third flag is obtained, which indicates whether sub-picture information of the CVS will appear in the video code stream.

2. The method according to claim 1, characterized in that The CVS is a layer-by-layer coded video sequence CLVS.

3. The method according to claim 1, characterized in that The second flag is not signaled in the video stream, and a value of the second flag is inferred to indicate that picture resolution does not change in the CVS.

4. The method according to any one of claims 1 to 3, characterized in that: Also includes: When the value of the second flag indicates that the picture resolution does not change in the CVS, a fourth flag indicating whether virtual boundary information is signaled is acquired.

5. The method according to any one of claims 1 to 3, characterized in that: When the value of the second flag indicates that the picture resolution does not change in the CVS, a scaling window parameter is signaled.

6. The method according to any one of claims 1 to 3, characterized in that: The first flag and the second flag are signaled in a sequence parameter set SPS.

7. The method according to claim 6, characterized in that When the value of the second flag indicates that the picture resolution is changed in the CVS, a fifth flag included in the picture parameter set PPS referring to the SPS indicates that the picture resolution is changed in the CVS, and When the fifth flag indicates that a picture resolution is changed in the CVS, at least one of a width and a height of a picture corresponding to the PPS is explicitly signaled.

8. The method according to claim 6, characterized in that When the value of the second flag indicates that picture resolution is not changed in the CVS, a fifth flag included in the PPS referring to the SPS indicates that picture resolution is not changed in the CVS, When the fifth flag indicates that the picture resolution does not change in the CVS, at least one of the width and height of the picture corresponding to the PPS is not explicitly signaled, At least one of the width and the height is inferred to be equal to at least one of the maximum width and the maximum height signaled in the SPS.

9. A method for encoding a video code stream, characterized in that: The method comprises: signaling a first flag indicating that reference picture resampling is enabled for a coded video sequence CVS; In response to the first flag indicating that reference picture resampling is enabled for the CVS, signaling a second flag indicating whether picture resolution changes are allowed in the CVS; When the second flag indicates that picture resolution change is allowed in the CVS, encoding the CVS by using reference picture resampling for spatial scalability and adaptive resolution change ARC; and When the value of the second flag indicates that the picture resolution does not change in the CVS, the CVS is encoded by resampling the reference picture for spatial scalability but not resampling the reference picture for ARC, and a third flag is signaled, which indicates whether sub-picture information of the CVS will appear in the video code stream.

10. A video code stream decoding device, characterized in that: The method comprises a processing circuit configured to execute the method according to any one of claims 1 to 8.

11. A video code stream encoding device, characterized in that: The method comprises a processing circuit configured to execute the method according to any one of claims 1 to 9.

12. A method for processing a video code stream, characterized in that: The video code stream is decoded according to the decoding method according to any one of claims 1 to 8, or is generated based on the encoding method according to claim 9.

13. A non-volatile computer-readable storage medium, characterized in that: A video code stream is stored thereon, and the video code stream is decoded according to the decoding method according to any one of claims 1 to 8, or is generated based on the encoding method according to claim 9.

14. An electronic device, characterized in that: The invention comprises a memory for storing computer-readable instructions; a processor for reading the computer-readable instructions and executing the method according to any one of claims 1 to 9 and 12 as instructed by the computer-readable instructions.

Images