Tile and sub-picture partitioning

Abstract

Systems and methods for decoding are provided, wherein the method is performed by at least one processor to decode a sub-bitstream of a coded video stream, the coded video stream comprises a coded version of a first sub-picture and a second sub-picture of a picture, and the method comprises the step of decoding the first sub-picture of a picture independently of the second sub-picture using sub-picture and tile partitioning, wherein (i) the first sub-picture comprises a first rectangular area of ​​the picture and the second sub-picture comprises a second rectangular area of ​​the picture, and the second rectangular area is different from the first rectangular area, (ii) each of the first sub-picture and the second sub-picture comprises at least one tile, and (iii) the first sub-picture and the second sub-picture do not share a common tile.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims priority from U.S. Provisional Application No. 62 / 816,846 filed on March 11, 2019, the entire disclosure of which is incorporated herein by reference.

[0003] Technology field

[0004] The present disclosure relates to a set of advanced video coding techniques. More specifically, it relates to tile and sub-picture partitioning designs. Background Technology

[0005] A video coding standard unofficially known as VVC (Versatile Video Coding) is under development.

[0006] Some embodiments of the present disclosure address problems related to VVC and other problems.

[0007] In some embodiments, a method is provided. The method may be performed by at least one processor to decode a sub-bitstream of a coded video stream, the coded video stream comprising a coded version of a first sub-picture and a second sub-picture of a picture. The method comprises: receiving a sub-bitstream; and decoding the first sub-picture of the picture independently of the second sub-picture using sub-picture and tile partitioning, wherein (i) the first sub-picture comprises a first rectangular area of ​​the picture and the second sub-picture comprises a second rectangular area of ​​the picture, the second rectangular area is different from the first rectangular area, (ii) each of the first sub-picture and the second sub-picture comprises at least one tile, and (iii) the first sub-picture and the second sub-picture do not share a common tile.

[0008] In one embodiment, the method further comprises the step of decoding a second sub-picture of a picture independently of the first sub-picture using sub-picture and tile partitioning, wherein at least one tile of the first sub-picture is a first plurality of tiles and at least one tile of the second sub-picture is a second plurality of tiles. In one embodiment, the decoding of the first sub-picture is performed in a tile scan order different from the decoding of the second sub-picture. In one embodiment, the decoding of the first sub-picture and the decoding of the second sub-picture are performed using sub-picture and tile partitioning, the first plurality of tiles of the first sub-picture are grouped into at least one first tile group, and the second plurality of tiles of the second sub-picture are grouped into at least one second tile group, and no tile of the at least one first tile group is located in the at least one second tile group. In one embodiment, at least one of the first tile groups is a non-rectangular tile group. In one embodiment, the first sub-picture is decoded according to a decoding technique in which tile-group-level loop filtering control at the boundary between two tile groups of at least one first tile group is allowed only when each of the two tile groups is rectangular.

[0009] In one embodiment, the step of receiving a sub-bitstream includes the step of receiving a coded video stream, and the coded video stream includes a sequence parameter set (SPS) containing information on a method for partitioning sub-pictures of a picture including a first sub-picture and a second sub-picture. In one embodiment, the received coded video stream includes a picture parameter set (PPS) containing information on a method for partitioning tiles of a picture including at least one tile of the first sub-picture and at least one tile of the second sub-picture. In one embodiment, the received coded video stream includes an active parameter set (APS) for signaling adaptive loop filter (ALF) coefficients of the first sub-picture.

[0010] In some embodiments, a decoder is provided. The decoder may be for decoding a sub-bitstream of a coded video stream, the coded video stream comprising a coded version of a first sub-picture and a second sub-picture of a picture. The decoder may include: a memory configured to store computer program code; and at least one processor configured to receive the sub-bitstream, access the computer program code, and operate as instructed by the computer program code. The computer program code may include a decoding code configured to allow at least one processor to decode a first sub-picture of a picture independently of a second sub-picture using sub-picture and tile partitioning, wherein (i) the first sub-picture includes a first rectangular area of ​​the picture and the second sub-picture includes a second rectangular area of ​​the picture, and the second rectangular area is different from the first rectangular area, (ii) each of the first sub-picture and the second sub-picture includes at least one tile, and (iii) the first sub-picture and the second sub-picture do not share a common tile.

[0011] In one embodiment, the decoding code is further configured to allow at least one processor to decode a second sub-picture of a picture independently of the first sub-picture, using sub-picture and tile partitioning, wherein at least one tile of the first sub-picture is a first plurality of tiles and at least one tile of the second sub-picture is a second plurality of tiles. In one embodiment, the decoding code is configured to allow at least one processor to decode the first sub-picture with a tile scan order different from the tile scan order used to decode the second sub-picture. In one embodiment, the decoding code is configured to allow at least one processor to decode a first sub-picture and a second sub-picture using sub-picture and tile partitioning, wherein a first plurality of tiles of the first sub-picture are grouped into at least one first tile group, and a second plurality of tiles of the second sub-picture are grouped into at least one second tile group, and none of the tiles of the at least one first tile group are located in the at least one second tile group. In one embodiment, one of the at least one first tile group is a non-rectangular tile group. In one embodiment, the decoding code is configured to allow at least one processor to decode according to a decoding technique in which tile-group-level loop filtering control at the boundary between two tile groups of the at least one first tile group is allowed only when each of the two tile groups is rectangular.

[0012] In one embodiment, a decoder is configured to receive a coded video stream including a sub-bitstream, the coded video stream includes a sequence parameter set (SPS) including first information on a method for partitioning sub-pictures of a picture including a first sub-picture and a second sub-picture, and a decoding code is configured to cause at least one processor to partition the sub-pictures of the picture according to the first information. In one embodiment, the coded video stream includes a picture parameter set (PPS) including second information on a method for partitioning tiles of a picture including at least one tile of a first sub-picture and at least one tile of a second sub-picture, and a decoding code is configured to cause at least one processor to partition the tiles of the picture according to the second information. In one embodiment, the coded video stream includes an active parameter set (APS) that signals the adaptive loop filter (ALF) coefficients of a first sub-picture, and the decoding code is configured to allow at least one processor to use the APS to decode the first sub-picture.

[0013] In some embodiments, a non-transient computer-readable medium is provided in which computer instructions are stored. The computer instructions, when executed by at least one processor, cause at least one processor to: decode a first sub-picture of a picture of a coded video stream independently of a second sub-picture of a picture of a coded video stream using sub-picture and tile partitioning, wherein (i) the first sub-picture includes a first rectangular area of ​​the picture and the second sub-picture includes a second rectangular area of ​​the picture, and the second rectangular area is different from the first rectangular area; (ii) each of the first sub-picture and the second sub-picture includes at least one tile; and (iii) the first sub-picture and the second sub-picture do not share a common tile.

[0014] In one embodiment, when computer instructions are executed by at least one processor, at least one processor additionally causes a second sub-picture of a picture to be decoded using sub-picture and tile partitioning independently of the first sub-picture, and at least one tile of the first sub-picture is a first plurality of tiles, and at least one tile of the second sub-picture is a second plurality of tiles. Brief explanation of the drawing

[0015] Additional features, nature, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. FIG. 2 is a schematic diagram of a simplified block diagram of a streaming system according to one embodiment. FIG. 3 is a schematic diagram of a simplified block diagram of a video decoder and a display according to one embodiment. FIG. 4 is a schematic diagram of a simplified block diagram of a video encoder and a video source according to one embodiment. FIG. 5a is a drawing of a picture of an embodiment illustrating sub-pictures and tiles. FIG. 5b is a drawing of a picture of an embodiment illustrating sub-pictures and tile groups. FIG. 5c is a drawing of a picture of an embodiment illustrating sub-pictures, tile groups, and tiles. Figure 6 is a schematic diagram of a system of one embodiment. Figure 7 is a schematic diagram of a composite video of one embodiment. Figure 8 is a schematic diagram of adaptive loop filter processing across the tile boundaries of tile groups. FIG. 9 is a drawing illustrating a method of one embodiment. FIG. 10 is a drawing of a computer system suitable for implementing embodiments. Specific details for implementing the invention

[0016] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to one embodiment of the present disclosure. The system (100) may include at least two terminals (110-120) interconnected via a network (150). For unidirectional transmission of data, the first terminal (110) may encode video data at a local location to transmit to another terminal (120) via the network (150). The second terminal (120) may receive the coded video data from the other terminal from the network (150), decode the coded data, and display the recovered video data. Unidirectional data transmission may be common in media serving applications, etc.

[0017] FIG. 1 illustrates a second pair of terminals (130, 140) provided to support bidirectional transmission of coded video that may occur, for example, during a video conference. For bidirectional transmission of data, each terminal (130, 140) may code video data captured at a local location to transmit to another terminal via a network (150). Each terminal (130, 140) may also receive coded video data transmitted by another terminal, decode the coded data, and display the recovered video data on a local display device.

[0018] In FIG. 1, the terminals (110-140) may be, for example, servers, personal computers, and smartphones, and / or any other type of terminal. For example, the terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (150) represents any number of networks that transmit coded video data between the terminals (110-140), including, for example, wired and / or wireless communication networks. The communication network (150) may exchange data on circuit-switched and / or packet-switched channels. Representative networks include communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (150) may not be important to the operation of the disclosure unless described herein below.

[0019] FIG. 2 illustrates the arrangement of a video encoder and a decoder in a streaming environment as an example of an application for the disclosed subject. The disclosed subject may be used with other video-enabled applications, such as, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0020] As illustrated in FIG. 2, the streaming system (200) may include a capture subsystem (213) comprising a video source (201) and an encoder (203). The streaming system (200) may further include at least one streaming server (205) and / or at least one streaming client (206).

[0021] A video source (201) may, for example, generate an uncompressed video sample stream (202). The video source (201) may, for example, be a digital camera. The sample stream (202), depicted with a thick line to emphasize the large data volume compared to encoded video bitstreams, may be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement the embodiments of the disclosed subject matter as described in more detail below. The encoder (203) may also generate an encoded video bitstream (204). The encoded video bitstream (204), depicted with a thin line to emphasize the smaller data volume compared to the uncompressed video sample stream (202), may be stored on a streaming server (205) for future use. One or more streaming clients (206) can access the streaming server (205) to retrieve video bitstreams (209) which may be copies of the encoded video bitstream (204).

[0022] Streaming clients (206) may include a video decoder (210) and a display (212). The video decoder (210) may, for example, decode a video bitstream (209), which is an incoming copy of an encoded video bitstream (204), and generate an outgoing video sample stream (211) that can be rendered on a display (212) or another rendering device (not described). In some streaming systems, the video bitstreams (204, 209) may be encoded according to specific video coding / compression standards. Examples of such standards include, but are not limited to, ITU-T Recommendation H.265. A video coding standard informally known as Versatile Video Coding (VVC) is under development. Embodiments of this disclosure may be used in the context of VVC.

[0023] FIG. 3 illustrates an exemplary functional block diagram of a video decoder (210) attached to a display (212) according to an embodiment of the present disclosure.

[0024] The video decoder (210) may include a channel (312), a receiver (310), a buffer memory (315), an entropy decoder / parser (320), a scaler / inverse conversion unit (351), an intra prediction unit (352), a motion compensation prediction unit (353), an aggregator (355), a loop filter unit (356), a reference picture memory (357), and a current picture memory (358). In at least one embodiment, the video decoder (210) may include an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The video decoder (210) may also be partially or wholly implemented as software running on one or more CPUs having associated memories.

[0025] In this embodiment and other embodiments, the receiver (310) may receive one or more coded video sequences for which the decoder (210) decodes one coded video sequence at a time, wherein the decoding of each coded video sequence is independent of other coded video sequences. The coded video sequences may be received from a channel (312), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (310) may receive the encoded video data along with other data, e.g., encoded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not described). The receiver (310) may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory (315) may be combined between the receiver (310) and the entropy decoder / parser (320) (hereinafter "parser"). When the receiver (310) receives data from a storage / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer (315) may not be used or may be small. For use on best effort packet networks such as the Internet, the buffer (315) may be required, may be relatively large, and may be of an adaptive size.

[0026] The video decoder (210) may include a parser (320) to reconstruct symbols (321) from an entropy-coded video sequence. The categories of the symbols include, for example, information used to manage the operation of the decoder (210), and potentially information for controlling a rendering device, such as a display (212) that can be coupled to the decoder as illustrated in FIG. 2. The control information for the rendering device(s) may be in the form, for example, Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not depicted). The parser (320) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (320) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to a group. The subgroups may include Group of Pictures (GOPs), pictures, tiles, slices, macroblocks, Coding Units (CUs), blocks, Transform Units (TUs), Prediction Units (PUs), etc. The parser (320) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0027] The parser (320) can generate symbols (321) by performing an entropy decoding / parsing operation on a video sequence received from the buffer (315).

[0028] The reconstruction of the symbols (321) may involve a number of different units depending on the type of the coded video picture or its parts (e.g., inter- and intra-picture, inter- and intra-block), and other factors. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser (320). The flow of this subgroup control information between the parser (320) and the number of units described below is not described for clarity.

[0029] In addition to the functional blocks already mentioned, the decoder (210) may be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with one another and be at least partially integrated with one another. However, for the purpose of illustrating the disclosed subject, the conceptual subdivision into the functional units below is appropriate.

[0030] One unit may be a scaler / inverse transform unit (351). The scaler / inverse transform unit (351) may receive quantized transform coefficients as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, etc., as symbol(s) (321) from the parser (320). The scaler / inverse transform unit (351) may output blocks containing sample values ​​that can be input to the aggregator (355).

[0031] In some cases, the output samples of the scaler / inverse transform (351) may relate to an intra-coded block; that is, a block that uses prediction information from previously reconstructed pictures, rather than prediction information from previously reconstructed parts of the current picture. This prediction information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) uses surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory (358) to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator (355) adds the prediction information generated by the intra-prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351) on a per-sample basis.

[0032] In other cases, the output samples of the scaler / inverse unit (351) may be intercoded and potentially associated with a motion-compensated block. In such cases, the motion compensation prediction unit (353) may access the reference picture memory (357) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (321) associated with the block, these samples may be added by the aggregator (355) to the output of the scaler / inverse unit (351) (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (357) from which the motion compensation prediction unit (353) fetches prediction samples may be controlled by motion vectors. Motion vectors may be available to the motion compensation prediction unit (353) in the form of symbols (321) that may have, for example, x, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (357) when subsample accurate motion vectors are in use, motion vector prediction mechanisms, etc.

[0033] Various loop filtering techniques within the loop filter unit (356) may be performed on the output samples of the aggregator (355). Video compression techniques may include in-loop filter techniques that are made available to the loop filter unit (356) as symbols (321) from the parser (320) and are controlled by parameters included in the coded video bitstream, but which respond not only to meta-information acquired during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, but also to previously reconstructed and loop-filtered sample values.

[0034] The output of the loop filter unit (356) may be a sample stream that is not only output to a rendering device such as a display (212) but may also be stored in a reference picture memory (357) for use in future inter-picture predictions.

[0035] Certain coded pictures, when fully reconstructed, can be used as reference pictures for future predictions. When a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (320)), the current reference picture stored in the current picture memory (358) can become part of the reference picture memory (357), and a new (fresh) current picture memory can be reallocated before starting the reconstruction of the next coded picture.

[0036] The video decoder (210) can perform decoding operations according to a predetermined video compression technique that may be documented in a standard such as ITU-T Rec. H.265. The coded video sequence may comply with the syntax specified by the video compression technique or standard in use, in that it adheres to the syntax of the video compression technique or standard as specified in the video compression technique document or standard and specifically in the profile document within it. Additionally, to comply with some video compression techniques or standards, the complexity of the coded video sequence may be within boundaries defined by the levels of the video compression technique or standard. In some cases, the levels limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the levels may, in some cases, be further limited through HRD (Hypothetical Reference Decoder) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0037] In one embodiment, the receiver (310) may receive additional (redundant) data along with the encoded video. This additional data may be included as part of the encoded video sequence(s). This additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] FIG. 4 illustrates an exemplary functional block diagram of a video encoder (203) associated with a video source (201) according to an embodiment of the present disclosure.

[0039] The video encoder (203) may include, for example, an encoder which is a source coder (430), a coding engine (432), a (local) decoder (433), a reference picture memory (434), a predictor (435), a transmitter (440), an entropy coder (445), a controller (450), and a channel (460).

[0040] The encoder (203) can receive video samples from a video source (201) (not part of the encoder) capable of capturing video image(s) to be coded by the encoder (203).

[0041] A video source (201) may provide a source video sequence to be coded by an encoder (203) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., x bits, 10 bits, 12 bits, …), any color space (e.g., BT.601 Y CrCB, RGB, …), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (201) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (203) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. A person skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on the samples.

[0042] According to one embodiment, the encoder (203) can code and compress pictures of a source video sequence into a coded video sequence (443) in real time or under any other time constraints required by the application. Implementing an appropriate coding rate may be one function of the controller (450). The controller (450) may also control other functional units as described below and may be functionally coupled to these units. The coupling is not described for clarity. Parameters set by the controller (450) may include rate control related parameters (picture skip, quantizer, lambda values ​​of rate-distortion optimization techniques, …), picture size, picture group (GOP) layout, maximum motion vector search range, etc. A person skilled in the art can easily identify other functions of the controller (450) because they may be related to the video encoder (203) optimized for a specific system design.

[0043] Some video encoders operate in what a person skilled in the art would easily recognize as a "coding loop." For simplified description, the coding loop may consist of an encoding portion of a source coder (430) (responsible for generating symbols based on the input picture to be coded and reference picture(s)), and a (local) decoder (433) embedded in an encoder (203) that reconstructs the symbols to generate sample data that a (remote) decoder would also generate when the compression between the symbols and the coded video bitstream is lossless in certain video compression techniques. The reconstructed sample stream may be input into a reference picture memory (434). Because the decoding of the symbol stream yields bit-exact results independently of the decoder location (local or remote), the content of the reference picture memory is also bit exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" as reference picture samples exactly the same sample values ​​that the decoder "sees" when using the prediction during decoding. This basic principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, for example, due to channel errors) is known to those skilled in the art.

[0044] The operation of the “local” decoder (433) may be substantially the same as the operation of the “remote” decoder (210) already described in detail above in relation to FIG. 3. However, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (445) and the parser (320) may be lossless, the entropy decoding parts of the decoder (210), including the channel (312), receiver (310), buffer (315), and parser (320), may not be fully implemented in the local decoder (433).

[0045] An observation that can be made at this point is that any decoder technique, excluding parsing / entropy decoding present in the decoder, may need to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject focuses on decoder operations. Since encoder techniques may be the inverse of the comprehensively described decoder techniques, their descriptions may be condensed. Further details are required only in specific areas and are provided below.

[0046] As part of the operation, the source coder (430) may perform motion-compensated predictive coding, which predictively codes an input frame by referencing one or more previously coded frames from a video sequence designated as "reference frames." In this way, the coding engine (432) codes the differences between pixel blocks of the input frame and pixel blocks of the reference frame(s) that can be selected as predictive reference(s) for the input frame.

[0047] The local video decoder (433) can decode the coded video data of frames that may be designated as reference frames based on the symbols generated by the source coder (430). The operations of the coding engine (432) may advantageously be lossy processes. When the coded video data is decoded in the video decoder (not shown in FIG. 4), the reconstructed video sequence may be a replica of the source video sequence, typically having some errors. The local video decoder (433) can replicate the decoding processes that may be performed by the video decoder on the reference frames and allow the reconstructed reference frames to be stored in the reference picture memory (434). In this way, the encoder (203) can locally store copies of the reconstructed reference frames having common content as the reconstructed reference frames to be acquired by the far-end video decoder (without transmission errors).

[0048] The predictor (435) can perform predictive searches for the coding engine (432). That is, for a new frame to be coded, the predictor (435) can search the reference picture memory (434) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which can serve as appropriate predictive references for the new pictures. The predictor (435) can operate on a sample block-by-pixel block basis to find appropriate predictive references. In some cases, as determined by the search results obtained by the predictor (435), the input picture may have predictive references drawn from a number of reference pictures stored in the reference picture memory (434).

[0049] The controller (450) can manage the coding operation of the video coder (430), including, for example, the settings of parameters and subgroup parameters used to encode video data.

[0050] The outputs of all the aforementioned functional units may undergo entropy coding in an entropy coder (445). The entropy coder converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to a person skilled in the art, such as, for example, Huffman coding, variable-length coding, and arithmetic coding.

[0051] The transmitter (440) may buffer the coded video sequence(s) generated by the entropy coder (445) to prepare for transmission over a communication channel (460), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter (440) may merge the coded video data from the video coder (430) with other data to be transmitted, e.g., coded audio data and / or auxiliary data streams (sources not shown).

[0052] The controller (450) can manage the operation of the encoder (203). During coding, the controller (450) can assign a specific coded picture type to each coded picture, which can influence the coding techniques that can be applied to each picture. For example, pictures may often be assigned as an Intra Picture (I picture), a Predictive Picture (P picture), or a Bi-directionally Predictive Picture (B picture).

[0053] An Intra Picture (I Picture) may be one that can be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow different types of Intra Pictures, including, for example, Independent Decoder Refresh (IDR) Pictures. A person skilled in the art recognizes the corresponding variations of I Pictures and their respective applications and features.

[0054] The Predictive Picture (P Picture) may be coded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict sample values ​​of each block.

[0055] A bidirectionally predictive picture (B picture) may be coded and decoded using intra-prediction or inter-prediction, using at most two motion vectors and reference indices to predict sample values ​​of each block. Similarly, multi-predictive pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0056] Source pictures are generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and can be coded on a block-by-block basis. Blocks can be predictively coded by referencing other (already coded) blocks determined by the coding assignment applied to each of the blocks' pictures. For example, blocks of I pictures can be non-predictively coded, or they can be predictively coded by referencing already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of P pictures can be non-predictively coded through spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of B pictures can be non-predictively coded through spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0057] The video coder (203) can perform coding operations according to a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In the operation, the video coder (203) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancies in the input video sequence. Thus, the coded video data can comply with the syntax specified by the video coding technology or standard in use.

[0058] In one embodiment, the transmitter (440) may transmit additional data along with the encoded video. The video coder (430) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0059] The encoders and decoders of the present disclosure may encode and decode video streams according to a tile and sub-picture partitioning design. Embodiments including methods using a tile and sub-picture partitioning design may be used individually or combined in any order. Additionally, each of the methods, encoders, and decoders of the embodiments may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In the embodiments, one or more processors execute a program stored on a non-transient computer-readable medium to perform encoding and decoding of a video stream according to a tile and sub-picture partitioning design of one or more embodiments. Some aspects of the tile and sub-picture partitioning design of the present disclosure are described below.

[0060] It is useful that VVC (Versatile Video Coding) and some other video compression standards include MCTS (motion constrained tile set)-like functionality having the following features: (1) a sub-bitstream including a video coding layer (VCL) network abstraction layer (NAL) and non-VCL NAL units necessary for decoding a subset of tiles that can be extracted from a single VVC bitstream covering all tiles constituting the entire picture; and (2) an extracted sub-bitstream that can be decoded independently in a VVC decoder without referencing the NAL units.

[0061] Such a partitioning mechanism having the above characteristics may be called an MCTS (motion constrained tile set) or a sub-picture. To avoid confusion between the terms "tile group" and "tile set," the term "sub-picture" is used in this disclosure to describe the above partitioning mechanism.

[0062] A non-limiting exemplary structure of a tile and sub-picture partitioning design of one embodiment is described below with reference to FIGS. 5a through 5c. In one embodiment, a video stream comprises a plurality of pictures (500). Each picture (500) may comprise one or more sub-pictures (510), and each sub-picture (510) may comprise one or more tiles (530), for example, as illustrated in FIGS. 5a and 5b. The size and shape of the sub-pictures (510) and tiles (530) are not limited by FIGS. 5a and 5b and may be of any shape or size. The tiles (510) of each sub-picture (510) may be divided into one or more tile groups (520), for example, as illustrated in FIGS. 5b and 5c. The size and shape of the tile groups (520) are not limited by FIG. 5b and FIG. 5c and may be any shape or size. In one embodiment, one or more tiles (530) may not be provided in any tile group (520). In one embodiment, tile groups (520) within the same sub-picture (510) may share one or more tiles (530). Embodiments of the present disclosure may decode and encode video streams according to a tile and sub-picture partitioning design, wherein sub-pictures (510), tile groups (520), and tiles (530) are defined and used.

[0063] Embodiments of the sub-picture design of the present disclosure may include aspects of the sub-picture design from JVET-M0261.

[0064] In addition, the following features of embodiments of the sub-picture design of the present disclosure enhance the utility of the sub-picture design of the present disclosure for immersive media use cases:

[0065] (1) Each sub-picture (510) may have different random access periods and different inter-prediction structures. Using these features, non-uniform random access periods may be used for view-port dependent 360 streaming. In view-port dependent 360 streaming, only central areas with dynamic content may receive the attention of viewers, while other background areas may change slowly. Having different GOP structures and different random access periods may help provide faster access to specific areas with better overall visual quality.

[0066] (2) Sub-pictures (510) may have different resampling ratios. Due to this feature, the quality of background areas (e.g., top and bottom in 360, non-dynamic objects in PCC) can be efficiently sacrificed for overall bit efficiency.

[0067] (3) A plurality of sub-pictures (510) constituting a picture (500) may be encoded by a single encoder and decoded by a single decoder, or they may not. For example, a sub-picture (510) may be encoded independently by an encoder, and other sub-pictures (510) may be encoded by different encoders. Subsequently, sub-bitstreams corresponding to such sub-pictures (510) may be merged into a bitstream that can be decoded by a decoder. This feature may be used, for example, with e-sports content. For example, in one embodiment, there may be a plurality of players (Player 1 to Player N) participating in a game (e.g., a video game), and game and camera views of each of these players may be captured and transmitted. Based on the selected players of each viewer, the media processor may group the related game views (e.g., of the following players) and convert the group into video.

[0068] In one embodiment, there may be cameras that capture images of each player playing the game. For example, as illustrated in FIG. 6, the embodiment may include a camera among the cameras (610) that capture a video image (611) of Player 1, a camera among the cameras (610) that capture a video image (612) of Player 2, etc. Additionally, one or more processors having memory may capture video images including observer views (620) and player views (630). The observer views (620) may each be a spectator view of the game that, for example, the players of the game may not see while actively playing the game. For example, the spectator views may be from a perspective of the video game world distinct from the viewpoints seen by the players, and / or may include information that helps the spectator watch the game, which the players cannot see while actively playing. The observer views (620) may include one or more observer views, including the first observer view (621). The player views (630) may be views seen by each player who is playing the game. For example, the first player view (631) may be video game images seen by Player 1 while playing the game, the second player view (632) may be video game images seen by Player 2 while playing the game, and so on.

[0069] Video images from cameras (610), observer views (620), and player views (630) can be received by a compositor (640). In embodiments, any number of these video images may be individually encoded by their respective encoders and / or one or more of the video images may be encoded together by a single encoder. The compositor (640) may receive video images after they have been encoded by one or more encoders, before they have been encoded, or after they have been decoded. The compositor (640) may also function as an encoder and / or decoder. Based on inputs such as layout and stream selection (660), the compositer (640) may provide a specific composite of two or more video images from cameras (610), observer views (620), and player views (630) to a transcoder (650) as a composite video (690) illustrated in FIG. 7. The transcoder (650) may transcode the composite video (690) and output the composite video (690) to a media sink (680), the media sink may include one or more displays for displaying the composite video (690). The transcoder (650) may be monitored by a network monitor (670).

[0070] As illustrated in FIG. 7, the transcoded composite video (690) may include, for example, a composite of a video image (611) of Player 1, a video image (612) of Player 2, a first player view (631), a second player view (632), and a first observer view (621). However, any combination of any number of any video images provided to the synthesizer (640) may be combined together as the composite video (690).

[0071] In one embodiment, the sub-picture design includes the following features and technical details: (1) The sub-picture (510) is a rectangular area. (2) The sub-picture (510) may or may not be partitioned into a plurality of tiles (530). (3) If partitioned into a plurality of tiles (530), each sub-picture (510) has its own tile scan order. (4) The tiles (530) within the sub-picture (510) may be combined into rectangular or non-rectangular tile groups (520), but tiles (530) belonging to different sub-pictures (510) cannot be grouped together.

[0072] At least the following (1) to (7) aspects of the sub-picture design of the embodiments of the present disclosure are distinguished from JVET-M0261.

[0073] (1) In one embodiment, each sub-picture (510) may refer to its own SPS, PPS and APS, but each SPS may include all sub-picture partitioning information.

[0074] (a) In one embodiment, sub-picture partitioning and layout (informative) information may be signaled in the SPS. For example, the decoded sub-picture and the output size of the sub-picture (510) may be signaled. For example, reference picture list (RPL) information of the sub-picture (510) may be signaled. In one embodiment, each sub-picture (510) may or may not have the same RPL information within the same picture. (b) In one embodiment, tile partitioning information of the sub-picture (510) may be signaled in the PPS. (c) In one embodiment, ALF coefficients of the sub-picture (510) may be signaled in the APS. (d) In one embodiment, any parameter set or SEI message may be referenced by a plurality of sub-pictures (510).

[0075] (2) In one embodiment, the sub-picture ID can be signaled in the NAL unit header.

[0076] (3) In one embodiment, any decoding process across sub-picture boundaries (e.g., in-loop filtering, motion compensation) may not be allowed.

[0077] (4) In one embodiment, the boundary of the sub-picture (510) may be expanded and padded for motion compensation. In one embodiment, a flag indicating whether the boundary is expanded may be signaled in the SPS.

[0078] (5) In one embodiment, the decoded sub-picture (510) may or may not be resampled for output. In one embodiment, the spatial ratio between the output sub-picture size signaled in the SPS and the decoded sub-picture size may be used to calculate the resampling ratio.

[0079] (6) In one embodiment, in relation to extracting the sub-bitstream, VCL NAL units corresponding to the sub-picture ID are extracted, and others are removed. The parameter sets referenced by the VCL NAL units having the sub-picture ID are extracted, and others are removed.

[0080] (7) In one embodiment, regarding the assembly of sub-bitstreams, all VCL NAL units having the same POC value may be interleaved with the same access unit (AU). Partitioning information of the sub-pictures (510) in the SPS is rewritten if necessary. Sub-picture IDs and any parameter set IDs are rewritten if necessary.

[0081] The sequence parameter set RBSP syntax of the embodiments of the present disclosure is provided in Table 1 below.

[0082]

[0083] "num_sub_pictures_in_pic" specifies the number of sub-pictures (510) within each picture (500) that references the SPS.

[0084] "signalled_sub_pic_id_length_minus1" of 1 specifies the number of bits used to represent the syntax element "sub_pic_id[ i ]" and the syntax element "tile_group_sub_pic_id[ i ]" of the tile group headers when present. The value of "signalled_sub_pic_id_length_minus1" may be in the range of 0 to 15 (inclusive).

[0085] "dec_sub_pic_width_in_luma_samples[ i ]" specifies the width of the i-th decoded sub-picture (510) in units of luma samples in the coded video sequence. "dec_sub_pic_width_in_luma_samples[ i ]" may not be 0 and may be an integer multiple of "MinCbSizeY".

[0086] "dec_sub_pic_height_in_luma_samples[ i ]" specifies the height of the i-th decoded sub-picture (510) in units of luma samples in the coded video sequence. "dec_sub_pic_height_in_luma_samples[ i ]" may not be 0 and may be an integer multiple of "MinCbSizeY".

[0087] "output_sub_pic_width_in_luma_samples[ i ]" specifies the width of the i-th output sub-picture (510) in units of luma samples. "output_sub_pic_width_in_luma_samples" may not be 0.

[0088] "output_sub_pic_height_in_luma_samples[ i ]" specifies the height of the i-th output sub-picture (510) in units of luma samples. "output_sub_pic_height_in_luma_samples" may not be 0.

[0089] "sub_pic_id[ i ]" specifies the sub-picture identifier of the i-th sub-picture (510). The length of the "sub_pic_id[ i ]" syntax element is "sub_pic_id_length_minus1" + 1 bit. When it does not exist, the value of "sub_pic_id[ i ]" is set to 0.

[0090] "left_top_pos_x_in_luma_samples[ i ]" specifies the column position of the first pixel of the i-th sub-picture (510).

[0091] "left_top_pos_y_in_luma_samples[ i ]" specifies the row position of the first pixel of the i-th sub-picture (510).

[0092] In one embodiment, each sub-picture (510) is resampled to its corresponding output sub-picture size after decoding. In one embodiment, a sub-picture (510) cannot overlap with an area of ​​another sub-picture (510A). In one embodiment, the width and height of the picture size composited by all sub-picture output sizes and locations may be the same as "pic_width_in_luma_samples" and "pic_height_in_luma_samples", but a partial picture area composited by a subset of sub-pictures may be decoded.

[0093] The tile group header syntax of an embodiment of the present disclosure is provided in Table 2 below.

[0094]

[0095] "tile_group_sub_pic_id" specifies the sub-picture identifier of the sub-picture to which the current tile group belongs.

[0096] Tiles in HEVC were designed to support the following two main use cases: (1) parallel decoding processes, and (2) partial forwarding and decoding. The first use case was essentially made possible by using the original tiles in HEVC, but still had some dependencies on inter-prediction operations. The second use case was made possible by employing additional SEI messages called MCTS (motion constrained tile set), but this was an optional approach. In VVC, the same tile design was inherited from HEVC, but a new approach, so-called tile groups, was adopted to support multiple use cases.

[0097] The present disclosure provides embodiments comprising a sub-picture design that can be used for view-port-based delivery of VR360 or other immersive content supporting 3 / 6 degrees of freedom. This functionality will be useful in VVCs that will be widely used in future immersive content services. It is also desirable to have full capability to enable complete parallel decoding without any dependencies across tile boundaries.

[0098] Two syntax elements ("loop_filter_across_tiles_enabled_flag" and "loop_filter_across_tile_groups_enabled_flag") may be used to provide better parallel processing capabilities. These syntax elements indicate that in-loop filtering operations are not performed across tile boundaries or tile group boundaries, respectively. In embodiments of the present disclosure, the above two syntax elements may be included together with two additional syntax elements. The two additional syntax elements may indicate that inter-predicting operations are performed or are not performed across tile or tile group boundaries, respectively. In this semantic, inter-predicting operations include, for example, temporal motion compensation, current picture reference, temporal motion vector prediction, and any parameter prediction operations between pictures. Thus, in embodiments of the present disclosure, motion-constrained tiles may be used in coding / decoding standards such as HEVC, in contrast to motion-constrained tile sets. This feature is compatible with sub-picture or MCTS methods.

[0099] The above syntax elements are useful for at least two use cases: (1) a full parallel decoding process without any dependencies across tile / tile group boundaries, and (2) reconstructing tile-group layouts without transcoding VCL NAL units.

[0100] In relation to the first use case, even if the picture (500) is partitioned into two tile groups (520) and delivered, multiple decoders can decode the tile groups (520) in parallel when the tile groups (520) contain multiple tiles (530). Then, the overall parallel processing capability will be helpful.

[0101] A second use case relates to a use case of VR360 view-port dependent processing. When a targeted view-port is shifted to a boundary between two tile groups, the decoder may be required to receive and decode two tile groups to display the targeted view-port. However, in an embodiment of the disclosure having the described syntax elements, the partitioning information of the tile groups (520) in the PPS may be updated, for example, by a server or cloud processor, so that only one tile group (520) contains the entire targeted view-port, and one tile group (520) may be delivered to the decoder. When all tiles are encoded as motion-constrained tiles, this instant repartitioning is possible without VLC-level modifications.

[0102] Examples of syntax elements of the embodiments of the present disclosure are provided below.

[0103] The picture parameter set RBSP syntax of the embodiment of the present disclosure is provided in Table 3 below.

[0104]

[0105] The "full_parallel_decoding_enabled_flag" specifies that arbitrary processing and prediction across tile boundaries are not allowed.

[0106] "loop_filter_across_tiles_enabled_flag" of 1 specifies that in-loop filtering operations are performed across tile boundaries in pictures referencing the PPS. "loop_filter_across_tiles_enabled_flag" of 0 specifies that in-loop filtering operations are not performed across tile boundaries in pictures referencing the PPS. In-loop filtering operations include, for example, deblocking filters, sample adaptive offset filters, and adaptive loop filter operations. When not present, the value of loop_filter_across_tiles_enabled_flag can be inferred to be 0.

[0107] "loop_filter_across_tile_groups_enabled_flag" being 1 specifies that in-loop filtering operations are performed across tile group boundaries in pictures referencing the PPS. "loop_filter_across_tile_group_enabled_flag" being 0 specifies that in-loop filtering operations are not performed across tile group boundaries in pictures referencing the PPS. In-loop filtering operations include, for example, deblocking filters, sample adaptive offset filters, and adaptive loop filter operations. When not present, the value of "loop_filter_across_tile_groups_enabled_flag" can be inferred to be 0.

[0108] "inter_prediction_across_tiles_enabled_flag" being 1 specifies that inter-prediction operations are performed across tile boundaries in pictures referencing the PPS. "inter_prediction_across_tiles_enabled_flag" being 0 specifies that inter-prediction operations are not performed across tile boundaries in pictures referencing the PPS. Inter-prediction operations include, for example, temporal motion compensation, current picture reference, temporal motion vector prediction, and arbitrary parameter prediction operations between pictures. When not present, the value of "inter_prediction_across_tiles_enabled_flag" can be inferred to be 0.

[0109] "inter_prediction_across_tile_groups_enabled_flag" being 1 specifies that inter-prediction operations are performed across tile group boundaries in pictures referencing the PPS. "inter_prediction_across_tile_groups_enabled_flag" being 0 specifies that inter-prediction operations are not performed across tile group boundaries in pictures referencing the PPS. Inter-prediction operations include, for example, temporal motion compensation, current picture reference, temporal motion vector prediction, and arbitrary parameter prediction operations between pictures. When not present, the value of "inter_prediction_across_tile_groups_enabled_flag" can be inferred to be 0.

[0110] In VVC, the tile design is similar to the tile design in HEVC, but a new method, also referred to as a "tile group," has been adopted to support a number of embodiments including but not limited to (1) partial delivery and decoding, (2) parallel decoding processes, and (3) an MTU size that matches the tile granularity.

[0111] For parallel processing, two syntax elements ("loop_filter_across_tiles_enabled_flag" and "loop_filter_across_tile_groups_enabled_flag") can be used to optionally allow an in-loop filtering process across tile group boundaries. However, when tile groups are non-rectangular, enabling filtering across tile boundaries while disabling in-loop filtering across tile group boundaries by these syntax elements can complicate the filtering process. As illustrated in FIG. 8, the adaptive loop filter (ALF) is processed as a 7x7 diamond filter (840) across the tile group boundary between two tile groups (820). When tile groups (820) are non-rectangular, the filtering process is confusing. For each pixel and each filter coefficient, a boundary checking process is essential to identify which pixel belongs to the current tile group. The cases of temporal motion compensation and current picture reference are not different. When boundary expansion is applied at tile group boundaries for motion compensation, its padding process is also complex.

[0112] According to an embodiment of the present disclosure, tile-group-level loop filtering (or any other operation) control at the boundaries of tile groups will be allowed only for rectangular tile groups. Thus, confusion in the filtering and padding processes can be better avoided.

[0113] Additionally, the flag "loop_filter_across_tile_groups_enabled_flag" may not need to be signaled when single_tile_per_tile_group_flag is 1. Accordingly, embodiments of the present disclosure may include the following picture parameter set RBSP syntax, for example, as shown in Table 4.

[0114]

[0115] "loop_filter_across_tile_groups_enabled_flag" being 1 specifies that in-loop filtering operations are performed across tile group boundaries in pictures referencing the PPS. "loop_filter_across_tile_group_enabled_flag" being 0 specifies that in-loop filtering operations are not performed across tile group boundaries in pictures referencing the PPS. In-loop filtering operations include, for example, deblocking filters, sample adaptive offset filters, and adaptive loop filter operations. When absent, the value of "loop_filter_across_tile_groups_enabled_flag" can be inferred to be the same as the value of loop_filter_across_tiles_enabled_flag.

[0116] In one embodiment, the method may be performed by at least one processor to decode a sub-bitstream of a coded video stream. The coded video stream may include coded versions of a plurality of sub-pictures (510) for one or more pictures (500).

[0117] As illustrated in FIG. 9, the method may include the step (850) of at least one processor receiving a sub-bitstream. In one embodiment, the sub-bitstream may contain information about one or more sub-pictures (510) of one or more pictures (500) and may not contain information about other sub-pictures (510) of one or more pictures (500). After receiving the sub-bitstream, at least one processor may decode the sub-pictures (510) contained in the sub-bitstream. For example, assuming the sub-bitstream contains information about a first sub-picture and a second sub-picture of a single picture, at least one processor may decode the first sub-picture of the picture independently of the second sub-picture according to a decoding technique using the tile and sub-picture partitioning design of the present disclosure (860). Decoding of the first sub-picture may include decoding the tiles of the first sub-picture. At least one processor may also decode a second sub-picture of the picture according to a decoding technique using the tile and sub-picture partitioning design of the present disclosure (870). Decoding of the second sub-picture may include decoding the tiles of the second sub-picture.

[0118] In embodiments, at least one processor can encode sub-pictures (510) of one or more pictures (500) according to the tile and sub-picture partitioning design of the present disclosure, and transmit one or more sub-bitstreams of a video bitstream containing one or more encoded sub-pictures (510) to one or more decoders for decoding according to the tile and sub-picture partitioning design of the present disclosure.

[0119] In embodiments, decoders of the present disclosure (e.g., video decoder (210)) may perform the decoding methods of the present disclosure by accessing computer program code stored in memory and operating as commanded by said computer program code. For example, in one embodiment, the computer program code may include decoding code configured to cause a decoder to decode a first sub-picture of a picture independently of a second sub-picture according to a decoding technique using the tile and sub-picture partitioning design of the present disclosure, and also to cause a decoder to decode a second sub-picture of a picture independently of the first sub-picture according to a decoding technique using the tile and sub-picture partitioning design of the present disclosure.

[0120] In embodiments, encoders of the present disclosure (e.g., encoder (203)) may perform the encoding methods of the present disclosure by accessing computer program code stored in memory and operating as commanded by said computer program code. For example, in one embodiment, the computer program code may include encoding code configured to cause the encoder to encode sub-pictures (510) of one or more pictures (500) according to the tile and sub-picture partitioning design of the present disclosure. The computer program code may also include transmission code configured to cause one or more sub-bitstreams of a video bitstream containing one or more encoded sub-pictures (510) to be transmitted to one or more decoders for decoding according to the tile and sub-picture partitioning design of the present disclosure.

[0121] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 10 illustrates a computer system (900) suitable for implementing specific embodiments of the present disclosure.

[0122] Computer software may be coded using any suitable machine code or computer language in which assembly, compilation, linking, or similar mechanisms can be performed to generate code containing instructions that can be executed directly or through interpretation, micro-code execution, etc., by a computer central processing unit (CPU), graphics processing unit (GPU), etc.

[0123] The commands can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0124] The components of the computer system (900) illustrated in FIG. 10 are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependency or requirement in relation to any one or a combination thereof of the components illustrated in the non-limiting embodiments of the computer system (900).

[0125] The computer system (900) may include specific human interface input devices. Such human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove motions), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not described). Human interface devices may also be used to capture specific media that are not necessarily directly related to conscious input by humans, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., 2D video, 3D video including stereoscopic video).

[0126] Input human interface devices may include one or more of: a keyboard (901), a mouse (902), a trackpad (903), a touch screen (910), a data-glove, a joystick (905), a microphone (906), a scanner (907), and a camera (908) (each only one is depicted).

[0127] The computer system (900) may also include certain human interface output devices. Such human interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touch-screen (910), data glove, or joystick (905), but there may also be tactile feedback devices that do not function as input devices). For example, these devices may include audio output devices (e.g.: speakers (909), headphones (not described)), visual output devices (e.g., screens (910) including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touch-screen input capability and each with or without haptic feedback capability - some of which may output two-dimensional visual output or output beyond three dimensions through means such as stereographic output -; virtual reality glasses (not described), holographic displays and smoke tanks (not described)), and printers (not described).

[0128] The computer system (900) may also include human-accessible storage devices and their associated media, such as optical media including a CD / DVD ROM / RW (920) having a media (921) such as a CD / DVD, a thumb-drive (922), a removable hard drive or solid-state drive (923), legacy magnetic media such as tape and floppy disk (not depicted), specialized ROM / ASIC / PLD-based devices such as a security dongle (not depicted).

[0129] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carriers, or other transient signals.

[0130] The computer system (900) may also include an interface to one or more communication networks. The networks may be, for example, wireless, wired, or optical. The networks may additionally be local, wide-area, metropolitan, automotive and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks, such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, and automotive and industrial networks including CANBus. Certain networks generally require external network interface adapters attached to specific general-purpose data ports or peripheral buses (949) (e.g., USB ports of the computer system (900), etc.); others are generally integrated into the core of the computer system (900) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (900) may communicate with other entities. Such communication may be unidirectional receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus to specific CANbus devices), or bidirectional with other computer systems using, for example, local area or wide area digital networks. Such communication may include communication to a cloud computing environment (955). Specific protocols and protocol stacks may be used for each of the networks and network interfaces described above.

[0131] The aforementioned human interface devices, human-accessible storage devices, and network interfaces (954) can be attached to the core (940) of the computer system (900).

[0132] The core (940) may include one or more central processing units (CPUs) (941), specialized programmable processing units in the form of graphics processing units (GPUs) (942), field programmable gate areas (FPGAs) (943), hardware accelerators (944) for specific tasks, etc. These devices may be connected via a system bus (948), along with internal mass storage (947), such as read-only memory (ROM) (945), random access memory (946), internal non-user accessible hard drives, SSDs, etc. In some computer systems, the system bus (948) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (948) or via a peripheral bus (949). Architectures for peripheral buses include PCI, USB, etc. A graphics adapter (950) may be included in the core (940).

[0133] CPUs (941), GPUs (942), FPGAs (943), and accelerators (944) can be combined to execute specific instructions that can constitute the aforementioned computer code. The computer code may be stored in ROM (945) or RAM (946). Transient data may also be stored in RAM (946), while persistent data may be stored, for example, in internal mass storage (947). High-speed storage and retrieval of any of the memory devices may be made possible through the use of cache memory, which may be closely associated with one or more CPUs (941), GPUs (942), mass storage (947), ROM (945), RAM (946), etc.

[0134] A computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code may be those specifically designed and configured for the purposes of this disclosure, or they may be of a kind well known and available to those skilled in the field of computer software technology.

[0135] As an example rather than a limitation, a computer system (900) having an architecture, and specifically a core (940), may provide functionality as a result of processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented on one or more types of tangible computer-readable media. Such computer-readable media may be media associated with specific storage of the core (940) that is of a non-transient nature, such as core-internal mass storage (947) or ROM (945), as well as user-accessible mass storage as described above. Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (940). The computer-readable media may include one or more memory devices or chips as needed. Software may enable the core (940) and, specifically, the processors within it (including a CPU, GPU, FPGA, etc.) to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (946) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise implemented in a circuit (e.g., accelerator (944)) that can operate instead of or with the software to execute specific processes or specific parts of specific processes described herein. Reference to software may include logic, where appropriate, and vice versa. Reference to a computer-readable medium may include, where appropriate, a circuit storing software for execution (e.g., an integrated circuit (IC)), or a circuit implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0136] Although the present disclosure describes various non-limiting embodiments, modifications, substitutions, and various alternative equivalents exist within the scope of the present disclosure. Accordingly, it will be acknowledged that those skilled in the art will be able to devise a number of systems and methods that implement the principles of the present disclosure and are therefore within the true meaning and scope thereof, even though they are not expressly illustrated or described in this specification.

Claims

Claim 1 A method performed by at least one processor, comprising the step of encoding a first sub-picture of a picture using a sub-picture, a tile group, and tile partitioning, independently of a second sub-picture of the picture; The method comprises the step of transmitting at least one coded sub-bitstream of a coded video stream including the first sub-picture and the second sub-picture to at least one decoder, wherein (i) the first sub-picture is a first rectangular area of ​​the picture and the second sub-picture is a second rectangular area of ​​the picture, and the second rectangular area is different from the first rectangular area; (ii) the first sub-picture includes a first plurality of tiles and the second sub-picture includes at least one tile; (iii) the first sub-picture and the second sub-picture do not share a common tile; (iv) the first plurality of tiles of the first sub-picture are grouped into at least two first tile groups; and (v) the first sub-picture is encoded according to a coding technique in which loop filtering control at the boundary between the at least two first tile groups is allowed only when each of the at least two first tile groups is rectangular. method. Claim 2 A method according to claim 1, further comprising the step of encoding the second sub-picture of the picture independently of the first sub-picture using the sub-picture, tile group, and tile partitioning, wherein the at least one tile of the second sub-picture is a second plurality of tiles. Claim 3 A method according to paragraph 2, wherein the encoding of the first sub-picture is performed in a tile scan order different from the encoding of the second sub-picture. Claim 4 A method according to claim 2, wherein the encoding of the first sub-picture and the encoding of the second sub-picture are performed using the sub-picture, tile group, and tile partitioning, the second plurality of tiles of the second sub-picture are grouped into at least one second tile group, and none of the tiles of the at least two first tile groups are located in the at least one second tile group. Claim 5 A method according to paragraph 4, wherein one of the at least two first tile groups is a non-rectangular tile group. Claim 6 The method of claim 1, wherein the step of transmitting at least one coded sub-bitstream comprises the step of transmitting a flag indicating whether in-loop filtering operations are performed across tile group boundaries. Claim 7 The method according to claim 1, wherein the step of transmitting at least one coded sub-bitstream includes the step of transmitting the coded video stream, and the coded video stream includes a sequence parameter set (SPS) comprising information for a method of partitioning sub-pictures of the picture, including the first sub-picture and the second sub-picture, each having an identifier. Claim 8 The method according to claim 1, wherein the step of transmitting at least one coded sub-bitstream includes the step of transmitting the coded video stream, and the coded video stream includes a picture parameter set (PPS) comprising information for a method of partitioning, arranging, or rearranging tiles of the picture, including at least one tile of the first sub-picture and at least one tile of the second sub-picture. Claim 9 A method according to claim 1, wherein the step of transmitting at least one coded sub-bitstream includes the step of transmitting the coded video stream, and the coded video stream includes an active parameter set (APS) that signals adaptive loop filter (ALF) coefficients of the first sub-picture. Claim 10 An encoder for encoding a video stream, wherein the video stream comprises coded versions of a first sub-picture and a second sub-picture of a picture, and the encoder comprises: a memory configured to store computer program code; and at least one processor configured to access the computer program code and operate as commanded by the computer program code, and the computer program code comprises: an encoding code configured to cause the at least one processor to encode the first sub-picture of the picture independently of the second sub-picture, using sub-pictures, tile groups, and tile partitioning; and includes a transmission code configured to cause the at least one processor to transmit at least one coded sub-bitstream of a coded video stream comprising the first sub-picture and the second sub-picture to at least one decoder, wherein (i) the first sub-picture is a first rectangular area of ​​the picture and the second sub-picture is a second rectangular area of ​​the picture, and the second rectangular area is different from the first rectangular area; (ii) the first sub-picture includes a first plurality of tiles and each of the second sub-pictures includes at least one tile; (iii) the first sub-picture and the second sub-picture do not share a common tile; (iv) the first plurality of tiles of the first sub-picture are grouped into at least two first tile groups; and (v) the first sub-picture is according to a coding technique in which loop filtering control at the boundary between the at least two first tile groups is allowed only when each of the at least two first tile groups is rectangular. Encoding, encoder. Claim 11 In claim 10, the encoding code is further configured to allow the at least one processor to encode the second sub-picture of the picture independently of the first sub-picture, using the sub-picture, tile group, and tile partitioning, and the at least one tile of the second sub-picture is a second plurality of tiles, an encoder. Claim 12 In claim 11, the encoding code is configured to cause the at least one processor to encode the first sub-picture in a tile scan order different from the tile scan order used to encode the second sub-picture, an encoder. Claim 13 In claim 11, the encoding code is configured to allow the at least one processor to encode the first sub-picture and the second sub-picture using the sub-picture, tile group, and tile partitioning, wherein the second plurality of tiles of the second sub-picture are grouped into at least one second tile group, and none of the tiles of the at least two first tile groups are located in the at least one second tile group, an encoder. Claim 14 In paragraph 13, an encoder in which one of the at least two first tile groups is a non-rectangular tile group. Claim 15 In paragraph 10, the encoder is further configured such that the transmission code causes the at least one processor to transmit a flag indicating whether in-loop filtering operations are performed across tile group boundaries. Claim 16 In claim 10, the encoder is configured to transmit the coded video stream comprising the at least one coded sub-bitstream, and the coded video stream comprises a sequence parameter set (SPS) comprising first information for a method of partitioning sub-pictures of the picture, including the first sub-picture and the second sub-picture, each having an identifier. Claim 17 In claim 10, the encoder is configured to transmit the coded video stream comprising the at least one coded sub-bitstream, and the coded video stream comprises a picture parameter set (PPS) comprising second information for a method of partitioning, arranging, or rearranging the tiles of the picture, including at least one tile of the first sub-picture and at least one tile of the second sub-picture. Claim 18 In claim 10, the encoder is configured to transmit the coded video stream including the at least one coded sub-bitstream, and the transmission code is configured to transmit the coded video stream including an active parameter set (APS) that signals the adaptive loop filter (ALF) coefficients of the first sub-picture. Claim 19 A non-transient computer-readable medium for storing computer instructions, wherein the computer instructions, when executed by at least one processor, cause the at least one processor to execute the method of any one of claims 1 to 9.

Images