Tile and sub-image division

Abstract

To provide a system and a method for decoding.SOLUTION: A method is executed by at least one processor to decode sub-bitstreams of an encoded video stream, the encoded video stream includes encoded versions of the first sub-image and the second sub-image of an image, the method includes a step of decoding a first sub-image of the image independently of a second sub-image using sub-images and tiling, (i) the first sub-image includes a first rectangular region of the image, the second sub-image includes a second rectangular region of the image, the second rectangular region is different from the first rectangular region, (ii) the first sub-image and the second sub-image each include at least one tile, and (iii) the first sub-image and the second sub-image do not share common tiles.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 816,846, filed March 11, 2019, the disclosure of which is incorporated herein by reference in its entirety.

[0002] This disclosure relates to a set of advanced video coding techniques, and more particularly to tile and sub-image partitioning designs. [Background technology]

[0003] A video coding standard, informally known as Versatile Video Coding (VVC), is under development. Summary of the Invention [Problem to be solved by the invention]

[0004] Some embodiments of the present disclosure address issues related to VVC, as well as other issues. [Means for solving the problem]

[0005] In some embodiments, a method is provided that may be executed by at least one processor for decoding a sub-bitstream of an encoded video stream, the encoded video stream including encoded versions of a first sub-image and a second sub-image of an image, the method including receiving the sub-bitstream and decoding the first sub-image of the image independently of the second sub-image using the sub-image and tile partitioning, where (i) the first sub-image includes a first rectangular region of the image and the second sub-image includes a second rectangular region of the image, the second rectangular region being different from the first rectangular region, (ii) the first sub-image and the second sub-image each include at least one tile, and (iii) the first sub-image and the second sub-image do not share a common tile.

[0006] In one embodiment, the method further includes decoding a second sub-image of the image, independent of the first sub-image, using sub-image and tile partitioning, where at least one tile of the first sub-image is a first plurality of tiles and at least one tile of the second sub-image is a second plurality of tiles. In one embodiment, the step of decoding the first sub-image is performed in a different tile scan order than the step of decoding the second sub-image. In one embodiment, the steps of decoding the first sub-image and decoding the second sub-image are performed using sub-image and tile partitioning, where the first plurality of tiles of the first sub-image are grouped into at least one first tile group, the second plurality of tiles of the second sub-image are grouped into at least one second tile group, and none of the tiles of the at least one first tile group are placed in the at least one second tile group. In one embodiment, one of the at least one first tile group is a non-rectangular tile group. In one embodiment, a first sub-image is decoded according to a decoding technique, and tile group level loop filtering control at the boundary between two tile groups of at least one first tile group is only possible if each of the two tile groups is rectangular.

[0007] In one embodiment, the step of receiving the sub-bitstream includes receiving an encoded video stream, the encoded video stream including a sequence parameter set (SPS) having information about how to divide sub-images of the image, including a first sub-image and a second sub-image. In one embodiment, the received encoded video stream includes a picture parameter set (PPS) having information about how to divide tiles of the image, including at least one tile of the first sub-image and at least one tile of the second sub-image. In one embodiment, the received encoded video stream includes an active parameter set (APS) signaling adaptive loop filter (ALF) coefficients for the first sub-image.

[0008] In some embodiments, a decoder is provided for decoding a sub-bitstream of an encoded video stream, the encoded video stream including encoded versions of a first sub-image and a second sub-image of an image. The decoder may comprise a memory configured to store computer program code, and at least one processor configured to receive the sub-bitstream, access the computer program code, and operate as instructed by the computer program code. The computer program code may include decoding code configured to cause the at least one processor to decode a first sub-image of the image independently of the second sub-image using the sub-image and tile division, where (i) the first sub-image includes a first rectangular region of the image, the second sub-image includes a second rectangular region of the image, the second rectangular region being different from the first rectangular region, (ii) the first sub-image and the second sub-image each include at least one tile, and (iii) the first sub-image and the second sub-image do not share a common tile.

[0009] In one embodiment, the decoding code is further configured to cause the at least one processor to decode a second sub-image of the image independently of the first sub-image using the sub-image and tile division; At least one tile of the first sub-image is a first plurality of tiles and at least one tile of the second sub-image is a second plurality of tiles. In one embodiment, the decoding code is configured to cause the at least one processor to decode the first sub-image in a tile scan order that is different from a tile scan order used to decode the second sub-image. In one embodiment, the decoding code is configured to cause the at least one processor to decode the first sub-image and the second sub-image using a sub-image and tile partitioning, where the first plurality of tiles of the first sub-image are grouped into at least one first tile group and the second plurality of tiles of the second sub-image are grouped into at least one second tile group, and none of the tiles of the at least one first tile group are placed in the at least one second tile group. In one embodiment, one of the at least one first tile group is a non-rectangular tile group. In one embodiment, the decoding code is configured to cause at least one processor to decode according to a decoding technique, whereby tile group level loop filtering control at a boundary between two tile groups of at least one first tile group is only possible if each of the two tile groups is rectangular.

[0010] In one embodiment, a decoder is configured to receive an encoded video stream including a sub-bitstream, the encoded video stream including a sequence parameter set (SPS) having first information about how to divide a sub-image of the image including a first sub-image and a second sub-image, and the decoding code is configured to cause at least one processor to divide the sub-image of the image according to the first information. In one embodiment, the encoded video stream includes a picture parameter set (PPS) having second information about how to divide tiles of the image including at least one tile of the first sub-image and at least one tile of the second sub-image, and the decoding code is configured to cause at least one processor to divide tiles of the image according to the second information. In one embodiment, the encoded video stream includes an active parameter set (APS) signaling adaptive loop filter (ALF) coefficients of the first sub-image, and the decoding code is configured to cause at least one processor to use the APS for decoding the first sub-image.

[0011] In some embodiments, a non-transitory computer-readable medium is provided that stores computer instructions that, when executed by at least one processor, cause the at least one processor to decode a first sub-image of an image of the encoded video stream independently of a second sub-image of the image of the encoded video stream using the sub-image and tile division, where (i) the first sub-image includes a first rectangular region of the image and the second sub-image includes a second rectangular region of the image, the second rectangular region being different from the first rectangular region, (ii) the first sub-image and the second sub-image each include at least one tile, and (iii) the first sub-image and the second sub-image do not share a common tile.

[0012] In one embodiment, the computer instructions, when executed by the at least one processor, further cause the at least one processor to decode a second sub-image of the image independently of the first sub-image using the sub-image and tile division, wherein at least one tile of the first sub-image is a first plurality of tiles and at least one tile of the second sub-image is a second plurality of tiles.

[0013] Further features, nature and various advantages of the subject matter of the present disclosure will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system, according to an embodiment. [Diagram 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system, according to an embodiment. [Diagram 3] FIG. 2 is a simplified block diagram schematic of a video decoder and display device according to an embodiment; [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder and a video source according to an embodiment; [Figure 5A] FIG. 2 is an image of an embodiment showing sub-images and tiles. [Figure 5B] FIG. 1 is an image of an embodiment showing sub-images and tile groups. [Figure 5C] FIG. 2 is a diagram of an image of an embodiment showing sub-images, tile groups, and tiles. [Figure 6] FIG. 1 is a schematic diagram of an embodiment of a system. [Figure 7] FIG. 2 is a schematic diagram of a composite image of an embodiment. [Figure 8] 1 is a schematic diagram of an adaptive loop filter that processes across tile boundaries in a tile group. [Figure 9] FIG. 2 illustrates a method of an embodiment. [Figure 10]FIG. 1 illustrates a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] FIG. 1 illustrates a simplified block diagram of a communication system 100 according to an embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of a one-way transmission of data, a first terminal 110 may locally encode video data for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the other terminal's encoded video data from the network 150, decode the encoded data, and display the recovered video data. One-way data transmission may be common in media distribution applications, etc.

[0016] 1 shows a second pair of terminals 130, 140 provided to support bidirectional transmission of encoded video, such as might occur during a video conference. For bidirectional transmission of data, each terminal 130, 140 may encode video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 may receive the encoded video data transmitted by the other terminal, may decode the encoded data, and may display the recovered video data on a local display device.

[0017] In FIG. 1, terminals 110-140 may be, for example, servers, personal computers, and smartphones, and / or any other type of terminal. For example, terminals (110-140) may be notebook computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks, including wired and / or wireless communication networks, that communicate encoded video data between terminals 110-140. Communication network 150 may exchange data over circuit-switched channels and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 150 may be irrelevant to the operation of the present disclosure, unless described below.

[0018] 2 shows an arrangement of video encoders and decoders in a streaming environment as an example application of the disclosed subject matter. The disclosed subject matter can be used in other video applications including, for example, video conferencing, digital television, and storage of compressed video on digital media such as CDs, DVDs, memory sticks, etc.

[0019] 2, the streaming system 200 may include a capture subsystem (213) that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0020] The video source 201 may generate, for example, an uncompressed video sample stream 202. The video source 201 may be, for example, a digital camera. The sample stream 202, shown in bold to emphasize its large amount of data compared to an encoded video bitstream, may be processed by an encoder 203 coupled to the camera 201. As described in more detail below, the encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter. The encoder 203 may further generate an encoded video bitstream 204. The encoded video bitstream 204, shown in thin to emphasize its small amount of data compared to the uncompressed video sample stream 202, may be stored on a streaming server 205 for later use. One or more streaming clients 206 may access the streaming server 205 to retrieve a video bitstream 209, which may be a copy of the encoded video bitstream 204.

[0021] The streaming client 206 may include a video decoder 210 and a display device 212. The video decoder 210 decodes a video bitstream 209, which may be, for example, an incoming copy of the encoded video bitstream 204, to generate an outgoing video sample stream 211, which may be displayed on the display device 212 or another display device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to some video encoding / compression standard. Examples of such standards include, but are not limited to, ITU-T Recommendation H.265. A video encoding standard, informally known as Versatile Video Coding (VVC), is under development. Embodiments of the present disclosure may be used in conjunction with VVC.

[0022] FIG. 3 is an exemplary functional block diagram of a video decoder 210 attached to a display device 212, according to an embodiment of the present disclosure.

[0023] The video decoder 210 may comprise a channel 312, a receiver 310, a buffer memory 315, an entropy decoder / parser 320, a scaler / inverse transform unit 351, an intra prediction unit 352, a motion compensated prediction unit 353, an aggregator 355, a loop filter unit 356, a reference picture memory 357, and a current picture memory 358. In at least one embodiment, the video decoder 210 may comprise an integrated circuit, a series of integrated circuits, and / or other electronic circuitry. The video decoder 210 may also be embodied, in part or in whole, as software executing on one or more CPUs with associated memory.

[0024] In this and other embodiments, the receiver 310 may receive one or more coded video sequences to be decoded by the decoder 210, or may receive one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel 312, which may be hardware / software coupled with a storage device that stores the coded video data. The receiver 310 may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to a respective entity (not shown). The receiver 310 may separate the coded video sequences from other data. To combat network jitter, a buffer memory 315 may be coupled between the receiver 310 and the entropy decoder / parser 320 (hereafter referred to as the "parser"). When the receiver 310 is receiving data from a storage / transmission device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer 315 may not be used or may be small. Buffer 315 may be required for use in a best effort packet network such as the Internet, and may be relatively large and adaptively sized.

[0025] The video decoder 210 may include a parser 320 to reconstruct symbols 321 from the entropy coded video sequence. Such symbol classification includes, for example, information used to manage the operation of the decoder 210, as well as potential information for controlling a display device, such as display device 212 as shown in FIG. 2, that may be coupled to the decoder. The control information for the display device may be in the form of, for example, a Supplementary Enhancement Information (SEI) message, or a Video Usability Information (VUI) parameter set fragment (not shown). The parser 320 may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be in accordance with a video coding technique or standard, and may be in accordance with principles well known to those skilled in the art, including variable length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. The parser 320 may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group from the coded video sequence. The subgroups may include groups of pictures (GOPs), images, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser 320 may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0026] Parser 320 may perform an entropy decoding / parsing operation on the video sequence received from buffer 315 to generate symbols 321 .

[0027] The reconstruction of symbols 321 may include a number of different units depending on the type of coded video or portions thereof (e.g., inter and intra pictures, inter and intra blocks), as well as other factors. Which units are included and how they are included may be controlled by subgroup control information parsed from the coded video sequence by parser 320. The flow of such subgroup control information between parser 320 and the units described below is not shown for clarity.

[0028] In addition to the functional blocks already mentioned, the decoder 210 may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of such units may closely interact with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0029] One unit may be a scalar / inverse transform unit 351. The scalar / inverse transform unit 351 may receive quantized transform coefficients as well as control information, including the transform to use, block size, quantization factor, quantization scaling matrix, etc., as symbol(s) 321 from the parser 320. The scalar / inverse transform unit 351 may output a block containing sample values, which may be input to an aggregator 355.

[0030] In some cases, the output samples of the scalar / inverse transform unit 351 may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image may use prediction information from a previously reconstructed part of the current image. Such prediction information may be provided by an intra-image prediction unit 352. In some cases, the intra-image prediction unit 352 uses surrounding already reconstructed information taken from the current (partially reconstructed) image in the current image memory 358 to generate a block of the same size and shape as the block being reconstructed. The aggregation device 355 may add the prediction information generated by the intra-prediction unit 352 to the output sample information provided by the scalar / inverse transform unit 351 on a sample-by-sample basis.

[0031] In other cases, the output samples of the scalar / inverse transform unit 351 may relate to an inter-coded and potentially motion-compensated block. In such cases, the motion compensated prediction unit 353 may access the reference image memory 357 to retrieve samples to use for prediction. After motion compensating the retrieved samples according to the symbols 321 associated with the block, these samples may be added to the output of the scalar / inverse transform unit 351 (in this case called residual samples or residual signals) by the aggregation device 355 to generate output sample information. The addresses in the reference image memory 357 from which the motion compensated prediction unit 353 retrieves the prediction samples may be controlled by a motion vector. The motion vector is available to the motion compensated prediction unit 353 in the form of the symbols 321 and may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference image memory 357 when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0032] The output samples of aggregator 355 may be subjected to various loop filtering techniques in loop filter unit 356. The video compression techniques may include in-loop filter techniques controlled by parameters contained in the encoded video bitstream and made available to loop filter unit 356 as symbols 321 from parser 320, but may also be responsive to meta-information obtained during decoding of previous (in decoding order) portions of the encoded image or encoded video sequence, as well as to previously reconstructed loop filtered sample values.

[0033] The output of the loop filter unit 356 may be a sample stream that can be output to a display device, such as display device 212, and stored in a reference picture memory 357 for use in future inter-picture prediction.

[0034] Some coded pictures, once fully reconstructed, can be used as reference pictures for future predictions. Once a coded picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser 320), the current reference picture stored in the current picture memory 358 can become part of the reference picture memory 357, and a new current picture memory can be reallocated before starting the reconstruction of the subsequent coded picture.

[0035] The video decoder 210 may perform decoding operations according to a given video compression technique, which may be described in a standard such as ITU-T Rec. H.265. An encoded video sequence is said to conform to the syntax specified by the video compression technique or standard used in the sense that it conforms to the syntax of the video compression technique or standard, as specified in the video compression technique document or standard, and specifically in the profiles described therein. To conform to some video compression techniques or standards, the complexity of the encoded video sequence may also be within a range prescribed by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by a Hypothetical Reference Decoder (HRD) specification and HRD buffer management metadata signaled in the encoded video sequence.

[0036] In one embodiment, receiver 310 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by video decoder 210 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0037] FIG. 4 is an exemplary functional block diagram of a video encoder 203 associated with a video source 201, according to an embodiment of the present disclosure.

[0038] The video encoder 203 may comprise, for example, an encoder, such as a source encoder 430 , a coding engine 432 , a (local) decoder 433 , a reference image memory 434 , a predictor 435 , a transmitter 440 , an entropy encoder 445 , a controller 450 , and a channel 460 .

[0039] The encoder 203 may receive video samples from a video source 201 (not part of the encoder) that may capture the video to be encoded by the encoder 203 .

[0040] The video source 201 may provide a source video sequence to be encoded by the encoder 203 in the form of a digital video sample stream that may be of any suitable bit depth (x-bit, 10-bit, 12-bit, etc.), any color space (BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (Y CrCb 4:2:0, Y CrCb 4:4:4, etc.). In a media delivery system, the video source 201 may be a storage device storing previously prepared video. In a video conferencing system, the video source 203 may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual images that convey motion when viewed in sequence. The images themselves may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art will readily appreciate the relationship between pixels and samples. The following description focuses on samples.

[0041] According to one embodiment, the encoder 203 may encode and compress images of a source video sequence into an encoded video sequence 443 in real time or under other time constraints required by the application. Providing an appropriate encoding rate may be one function of the controller 450. The controller 450 may further control and be operatively coupled to other functional units as described below. For clarity, coupling is not shown. Parameters set by the controller 450 may include rate control related parameters (e.g., picture skip, quantization, lambda value for rate-distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller 450 as they may be relevant to a video encoder 203 optimized for some system designs.

[0042] Some video encoders operate in what is easily recognized by those skilled in the art as a "coding loop." Briefly, in some video compression techniques, when the compression between symbols and the encoded video bitstream is lossless, the encoding loop may consist of an encoding part of a source encoder 430 (responsible for generating symbols based on the input image to be encoded and the reference image) and a (local) decoder 433 embedded in the encoder 203, which reconstructs the symbols to generate sample data that is also generated by the (remote) decoder. The reconstructed sample stream may be input to a reference image memory 434. If the decoding of the symbol stream results in bit-exact, regardless of the location of the decoder (local or remote), the contents of the reference image memory will also be bit-perfect between the local and remote encoders. In other words, the predictor part of the encoder "sees" the reference image samples as exactly the same sample values ​​that the decoder "sees" when using prediction during decoding. This basic principle of reference image synchronicity (and the resulting drift if synchronicity cannot be maintained, for example due to channel errors) is known to those skilled in the art.

[0043] The operation of the "local" decoder 433 may be substantially the same as the "remote" decoder 210, which has already been described in detail above in relation to Figure 3. However, the entropy decoding portion of the decoder 210, including the channel 312, receiver 310, buffer 315, and parser 320, need not be entirely implemented in the local decoder 433, since symbols are available and can be losslessly encoded / decoded into an encoded video sequence by the entropy coder 445 and parser 320.

[0044] It is currently contemplated that any decoder techniques, other than parsing / entropy decoding, present in the decoder will need to be present in the corresponding encoder in approximately the same functional form. For this reason, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques may be omitted, as they may be the inverse of the decoder techniques, which are described generically. Only in a few areas are more detailed descriptions required and are described below.

[0045] The source encoder 430 may perform motion compensated predictive encoding as part of its operation, predictively encoding an input frame with respect to one or more previously encoded frames from the video sequence designated as “reference frames.” In this method, the encoding engine 432 encodes the differences between pixel blocks of the input frame and pixel blocks of reference frame(s) that may be selected as predictive reference(s) for the input frame.

[0046] The local video decoder 433 may decode the encoded video data of frames that may be designated as reference frames based on the symbols generated by the source encoder 430. The operation of the encoding engine 432 may preferably be a lossy process. When the encoded video data is decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence may be a copy of the source video sequence, usually with some errors. The local video decoder 433 replicates the decoding process that may be performed on the reference frames by the video decoder and may cause the reconstructed reference frames to be stored in the reference image memory 434. In this manner, the encoder 203 may locally store copies of the reconstructed reference frames that have common content with the reconstructed reference frames obtained by the far-end video decoder (free of transmission errors).

[0047] The predictor 435 may perform a prediction search for the coding engine 432. That is, the predictor 435 may search for sample data (as candidate reference pixel blocks) from the reference image memory 434 for a new frame to be coded, or some metadata such as reference image motion vectors, block shapes, etc., which serve as suitable prediction references for the new image. The predictor 435 may operate on a sample by sample basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor 435, the input image may have prediction references drawn from multiple reference images stored in the reference image memory 434.

[0048] Controller 450 may manage the encoding operations of video encoder 430, including, for example, setting parameters and subgroup parameters used to encode the video data.

[0049] The output of all the functional units mentioned above may be entropy coded in the entropy coder 445, which converts the symbols produced by the various functional units into an encoded video sequence by losslessly compressing the symbols with techniques known to those skilled in the art as Huffman coding, variable length coding, arithmetic coding, etc.

[0050] The transmitter 440 may buffer the encoded video sequence as it is generated by the entropy encoder 445 in preparation for transmission over the communication channel 460, which may be a hardware / software association with a storage device that stores the encoded video data. The transmitter 440 may merge the encoded video data of the video encoder 430 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0051] A controller 450 may manage the operation of the encoder 203. During encoding, the controller 450 may assign a number of encoding image types to each of the encoded images, which may affect the encoding technique that may be applied to each image. For example, images may often be assigned as intra-images (I-images), predicted images (P-images), or bidirectionally predicted images (B-images).

[0052] An intra picture (I-picture) is one that can be coded and decoded without using other frames in a sequence as a source of prediction. Some video codecs allow different kinds of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of such variations of I-pictures, as well as their respective uses and characteristics.

[0053] A predicted image (P picture) can be encoded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0054] Bidirectionally predicted images (B-pictures) are those that can be coded and decoded using intra- or inter-prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predictive images can use more than two reference images and associated metadata to reconstruct a block.

[0055] A source image may be subdivided, usually spatially, into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by a code assignment applied to each image of the block. For example, blocks of an I-picture may be non-predictively coded or may be predictively coded with reference to already coded blocks of the same image (spatial or intra prediction). Pixel blocks of a P-picture may be non-predictively coded by spatial prediction with reference to one previously coded reference image or by temporal prediction. Blocks of a B-picture may be non-predictively coded by spatial prediction with reference to one or two previously coded reference images or by temporal prediction.

[0056] Video encoder 203 may perform encoding operations in accordance with a given video encoding technique or standard, such as ITU-T Rec. H.265. In its operations, video encoder 203 may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore conform to a syntax specified by the video encoding technique or standard used.

[0057] In one embodiment, the transmitter 440 may transmit additional data along with the encoded video. The video encoder 430 may include such data as part of the encoded video sequence. The additional data may include other types of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0058] The encoders and decoders of the present disclosure may encode and decode the video stream according to the tile and sub-image partitioning design. The embodiments, including the methods, using the tile and sub-image partitioning design may be used individually or in any order in combination. Also, the methods, encoders, and decoders of the embodiments may each be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In an embodiment, one or more processors execute a program stored on a non-transitory computer-readable medium to perform the encoding and decoding of the video stream according to the tile and sub-image partitioning design of one or more embodiments. Some aspects of the tile and sub-image partitioning design of the present disclosure are described below.

[0059] In Versatile Video Coding (VVC) and some other video compression standards, it is useful to include a Motion Constrained Tile Set (MCTS)-like feature that has the following characteristics: (1) A sub-bitstream contains the Video Coding Layer (VCL), Network Abstraction Layer (NAL), and non-VCL NAL units required to decode a subset of tiles, and can be extracted from a single VVC bitstream that covers all tiles that make up the entire image, and (2) the extracted sub-bitstream can be independently decodable by a VVC decoder without reference to the NAL units.

[0060] Such a partitioning scheme with the above mentioned characteristics may be called MCTS (Motion Constrained Tile Set) or sub-image. To avoid confusion with the terms "tile group" and "tile set", in this disclosure the term "sub-image" is used to describe the above mentioned partitioning scheme.

[0061] A non-limiting exemplary structure of the tile and sub-image division design in the embodiment is described below with reference to FIGS. 5A-5C. In one embodiment, a video stream includes a plurality of images 500. Each image 500 may include one or more sub-images 510, and each sub-image 510 includes one or more tiles 530, for example, as shown in FIGS. 5A-5B. The size and shape of the sub-images 510 and tiles 530 are not limited by FIGS. 5A-5B and can be any shape or size. The tiles 510 of each sub-image 510 may be divided into one or more tile groups 520, for example, as shown in FIGS. 5B-5C. The size and shape of the tile groups 520 are not limited by FIGS. 5B-5C and can be any shape or size. In one embodiment, none of the tile groups 520 may be provided with one or more tiles 530. In one embodiment, the tile groups 520 in the same sub-image 510 may share one or more tiles 530. An embodiment of the present disclosure may decode and encode a video stream according to a tile and sub-image division design, in which sub-images 510, tile groups 520, and tiles 530 are defined and used.

[0062] Embodiments of the sub-image design of the present disclosure may include aspects of the sub-image design according to JVET-M0261.

[0063] Additionally, the sub-image design embodiments of the present disclosure have the following features that enhance the usefulness of the sub-image designs of the present disclosure when used for immersive media.

[0064] (1) Each sub-image 510 may have a different random access period and a different inter-prediction structure. Such a feature can be used to have unequal random access periods for viewport-dependent 360-degree streaming, where only focused areas containing dynamic content may be noticed by the viewer, while other background areas may change slowly. Having different GOP structures and different random access periods can help to access specific areas quickly while improving the overall visual quality.

[0065] (2) The sub-images 510 may have different resampling rates from each other, such that the quality of background regions (e.g., top and bottom in 360 degrees, non-dynamic objects in PCC) can be effectively sacrificed for overall bit-efficiency.

[0066] (3) The multiple sub-images 510 that make up the image 500 may or may not be encoded by a single encoder and decoded by a single decoder. For example, a sub-image 510 can be encoded independently by an encoder, while another sub-image 510 is encoded by a different encoder. The sub-bitstreams corresponding to these sub-images 510 can then be combined into one bitstream that can be decoded by a decoder. Such a feature may be used, for example, in e-sports content. For example, in one embodiment, there may be several players (player 1 to player N) participating in a game (such as a video game), and the game view and camera view of each of these players may be captured and transmitted. Depending on the player selected by each viewer, the media processor may group the relevant game views (e.g., of the next player) and convert the group into a video.

[0067] In an embodiment, there may be a camera that captures an image of each player playing the game. For example, as shown in FIG. 6, an embodiment may include a camera 610 that captures a video image 611 of player 1, a camera 610 that captures a video image 612 of player 2, and so on. One or more processors having memory may also capture video images including an observer view 620 and a player view 630. Each observer view 620 may be a spectator view of the game, or the like, where the players of the game may not be seen while actively playing the game. For example, the spectator view may be a whole view of the video game world that is different from the perspective that the players see, and / or may include information that assists a spectator watching the game, where the players who are actively playing are not seen. The observer view 620 may include one or more observer views, including a first observer view 621. Each player view 630 may be a view seen by each player playing the game. For example, the first player view 631 may be an image of the video game that player 1 sees while playing the game, and the second player view 632 may be an image of the video game that player 2 sees while playing the game.

[0068] The video images of the camera 610, the observer view 620, and the competitor view 630 may be received by a compositor 640. In an embodiment, any number of such video images may be encoded by each encoder individually, and / or one or more video images may be encoded together in a single encoder. The compositor 640 may receive the video images after the video images have been encoded by one or more encoders, before the video images are encoded, or after the video images have been decoded. The compositor 640 may also function as an encoder and / or a decoder. Based on inputs such as the layout & stream selection 660, the compositor 640 may provide a particular composite of two or more video images from the camera 610, the observer view 620, and the competitor view 630 to the transcoder 650 as a composite video 690 shown in FIG. 7. Transcoder 650 transcodes composite video 690 and outputs composite video 690 to media sink 680, which may include one or more display devices that display composite video 690. Transcoder 650 may be monitored by network monitor 670.

[0069] 7, the transcoded composite video 690 may include, for example, a composite of player 1 video image 611, player 2 video image 612, first player view 631, second player view 632, and first observer view 621. However, any number and combination of any of the video images provided to the compositor 640 may be composited together as the composite video 690.

[0070] In one embodiment, the sub-image design includes the following features and technical details: (1) a sub-image 510 is a rectangular region. (2) a sub-image 510 may or may not be divided into multiple tiles 530. (3) if divided into multiple tiles 530, each sub-image 510 has its own tile scan order. (4) tiles 530 within a sub-image 510 can be combined into rectangular or non-rectangular tile groups 520, but tiles 530 belonging to different sub-images 510 cannot be grouped together.

[0071] At least the following aspects (1) to (7) of the sub-image design in the embodiments of the present disclosure are distinguished from JVET-M0261.

[0072] (1) In one embodiment, each sub-image 510 may reference its own SPS, PPS, and APS, although each SPS may contain all the sub-image segmentation information.

[0073] (a) In one embodiment, sub-image partitioning and layout (reporting) information may be signaled in SPS. For example, the decoded sub-image and output size of sub-image 510 may be signaled. For example, reference picture list (RPL) information of sub-image 510 may be signaled. In one embodiment, each sub-image 510 may or may not have the same RPL information in the same image. (b) In one embodiment, tile partitioning information of sub-image 510 may be signaled in PPS. (c) In one embodiment, ALF coefficients of sub-image 510 may be signaled in APS. (d) In one embodiment, multiple sub-images 510 can reference any parameter set or SEI message.

[0074] (2) In one embodiment, the sub-picture ID may be signaled in the NAL unit header.

[0075] (3) In one embodiment, all decoding processes (eg, in-loop filtering, motion compensation) across sub-image boundaries may be rejected.

[0076] (4) In one embodiment, the borders of the sub-image 510 may be extended and padded for motion compensation. In one embodiment, a flag indicating whether the borders are extended may be signaled in the SPS.

[0077] (5) In one embodiment, the decoded sub-image 510 may or may not be resampled for output. In one embodiment, the spatial ratio between the decoded sub-image size and the output sub-image size, as signaled in the SPS, may be used to calculate the resampling ratio.

[0078] (6) In one embodiment, for extracting the sub-bitstream, the VCL NAL unit corresponding to the sub-picture ID is extracted and the rest are removed. The parameter set referenced in the VCL NAL unit having the sub-picture ID is extracted and the rest are removed.

[0079] (7) In one embodiment, for sub-bitstream assembly, all VCL NAL units with the same POC value may be interleaved with the same access unit (AU). The partitioning information of the sub-picture 510 in the SPS is rewritten as needed. The sub-picture ID and any parameter set ID are rewritten as needed.

[0080] The sequence parameter set RBSP syntax of an embodiment of the present disclosure is shown in Table 1 below.

[0081] [Table 1]

[0082] "num_sub_pictures_in_pic" specifies the number of sub-images 510 in each image 500 that references the SPS.

[0083] "signalled_sub_pic_id_length_minus1", if equal to 1, specifies the number of bits used to represent the syntax element "sub_pic_id[i]", if present, and the syntax element "tile_group_sub_pic_id[i]" in the tile group header. The value of "signalled_sub_pic_id_length_minus1" may be in the range from 0 to 15.

[0084] "dec_sub_pic_width_in_luma_samples[i]" specifies the width of the i-th decoded sub-image 510 in units of luma samples in the coded video sequence. "dec_sub_pic_width_in_luma_samples[i]" may be non-zero and may be an integer multiple of "MinCbSizeY".

[0085] "dec_sub_pic_height_in_luma_samples[i]" specifies the height of the i-th decoded sub-image 510 in units of luma samples in the coded video sequence. "dec_sub_pic_height_in_luma_samples[i]" need not be 0 and may be an integer multiple of "MinCbSizeY".

[0086] "output_sub_pic_width_in_luma_samples[i]" specifies the width, in units of luma samples, of the i-th output sub-image 510. "output_sub_pic_width_in_luma_samples" does not have to be zero.

[0087] "output_sub_pic_height_in_luma_samples[i]" specifies the height, in units of luma samples, of the i-th output sub-image 510. "output_sub_pic_height_in_luma_samples" does not have to be zero.

[0088] "sub_pic_id[i]" specifies the sub-picture identifier of the i-th sub-picture 510. The length of the "sub_pic_id[i]" syntax element is "sub_pic_id_length_minus1" + 1 bits. When not present, the value of "sub_pic_id[i]" is set to 0.

[0089] “left_top_pos_x_in_luma_samples[i]” specifies the column position of the first pixel in the i-th subimage 510 .

[0090] “left_top_pos_y_in_luma_samples[i]” specifies the row position of the first pixel in the i-th sub-image 510 .

[0091] In one embodiment, each sub-image 510 is resampled to its corresponding output sub-image size after decoding. In one embodiment, a sub-image 510 cannot overlap an area of ​​another sub-image 510A. In one embodiment, the width and height of the image size, comprised by the output size and position of all sub-images, may be equal to "pic_width_in_luma_samples" and "pic_height_in_luma_samples", but partial image areas comprised of a subset of sub-images can be decoded.

[0092] The syntax of the tile group header in an embodiment of the present disclosure is shown in Table 2 below.

[0093] [Table 2]

[0094] "tile_group_sub_pic_id" specifies the sub-image identifier of the sub-image to which the current tile group belongs.

[0095] Tiles in HEVC were designed to support two main use cases: (1) parallel decoding processes and (2) partial transmission and decoding. The first use case was achieved essentially by using the original tiles in HEVC, but still with some dependency on inter prediction operations. The second use case was achieved, albeit in an optional way, by using an additional SEI message called Motion Constrained Tile Set (MCTS). In VVC, the same tile design was inherited from HEVC, but a new scheme, so-called tile groups, was applied to support multiple use cases.

[0096] This disclosure provides embodiments including sub-picture designs that can be used for viewport-based transmission in VR360 and other immersive content that supports 3 / 6 degrees of freedom. Such capabilities will be useful for VVC to be widely used for future immersive content services. It is also desirable to have complete functionality to enable fully parallel decoding with no dependencies across tile boundaries.

[0097] In order to provide better parallel processing capabilities, two syntax elements ("loop_filter_across_tiles_enabled_flag" and "loop_filter_across_tile_groups_enabled_flag") may be used. These syntax elements indicate that in-loop filtering operations are not performed across tile boundaries or tile group boundaries, respectively. In an embodiment of the present disclosure, the above two syntax elements may be included with two additional syntax elements. The two additional syntax elements may indicate that inter prediction operations are or are not performed across tile boundaries or tile group boundaries, respectively. In these semantics, inter prediction operations include, for example, temporal motion compensation, current image reference, temporal motion vector prediction, and any parameter prediction operations between images. Therefore, in an embodiment of the present disclosure, in an encoding / decoding standard such as HEVC, motion constrained tiles may be used instead of motion constrained tile sets. This function may be cross-compatible with sub-image or MCTS schemes.

[0098] The above syntax elements are useful for at least two use cases: (1) a fully parallel decoding process with no dependencies across tile / tile group boundaries, and (2) reconstructing the layout of a tile group without transcoding any VCL NAL units.

[0099] Regarding the first use case, even if an image 500 is split into two tile groups 520 for transmission, multiple decoders can decode the tile groups 520 in parallel if the tile groups 520 contain multiple tiles 530. In this way, the full parallel processing capability is made available.

[0100] The second use case concerns the VR360 viewport dependent processing use case. When a target viewport is transitioned to the boundary between two tile groups, the decoder may need to receive and decode two tile groups to display the target viewport. However, in the embodiment of the present disclosure with the described syntax elements, the partitioning information of the tile groups 520 in the PPS can be updated, for example, by a server or cloud processor, and one tile group 520 can be transmitted to the decoder, so that only one tile group 520 contains the entire target viewport. This instantaneous repartitioning is possible without any VLC level changes when all tiles are coded as motion constrained tiles.

[0101] Examples of syntax elements according to embodiments of the present disclosure are shown below:

[0102] The image parameter set RBSP syntax of an embodiment of the present disclosure is shown in Table 3 below.

[0103] [Table 3]

[0104] "full_parallel_decoding_enabled_flag" if set to 1 specifies that all processing and prediction across tile boundaries is rejected.

[0105] "loop_filter_across_tiles_enabled_flag" when set to 1 specifies that in-loop filtering operations are performed across tile boundaries in images that reference the PPS. "loop_filter_across_tiles_enabled_flag" when set to 0 specifies that in-loop filtering operations are not performed across tile boundaries in images that reference the PPS. In-loop filtering operations include, for example, a deblocking filter operation, a sample adaptive offset filter operation, and an adaptive loop filter operation. If not present, the value of loop_filter_across_tiles_enabled_flag may be inferred to be 0.

[0106] "loop_filter_across_tile_groups_enabled_flag" when set to 1 specifies that in-loop filtering operations are performed across tile group boundaries in images that reference the PPS. "loop_filter_across_tile_group_enabled_flag" when set to 0 specifies that in-loop filtering operations are not performed across tile group boundaries in images that reference the PPS. In-loop filtering operations include, for example, a deblocking filter operation, a sample adaptive offset filter operation, and an adaptive loop filter operation. If not present, the value of "loop_filter_across_tile_groups_enabled_flag" may be inferred to be 0.

[0107] "inter_prediction_across_tiles_enabled_flag" when set to 1 specifies that inter prediction operations are performed across tile boundaries in images that reference the PPS. "inter_prediction_across_tiles_enabled_flag" when set to 0 specifies that inter prediction operations are not performed across tile boundaries in images that reference the PPS. Inter prediction operations include, for example, temporal motion compensation, current picture reference, temporal motion vector prediction, and any parametric prediction operations between pictures. If not present, the value of "inter_prediction_across_tiles_enabled_flag" may be inferred to be 0.

[0108] "inter_prediction_across_tile_groups_enabled_flag" is set to 1 to specify that inter prediction operations are performed across tile group boundaries in images that reference the PPS. "inter_prediction_across_tile_groups_enabled_flag" is set to 0 to specify that inter prediction operations are not performed across tile group boundaries in images that reference the PPS. Inter prediction operations include, for example, temporal motion compensation, current picture reference, temporal motion vector prediction, and any parametric prediction operations between pictures. If not present, the value of "inter_prediction_across_tile_groups_enabled_flag" may be inferred to be 0.

[0109] In VVC, the tile design is similar to that in HEVC, but to support multiple embodiments, a new scheme, also called "tile groups", has been applied, including but not limited to: (1) partial transmission and decoding, (2) parallel decoding processes, and (3) matching MTU size with tile granularity.

[0110] For parallel processing, two syntax elements ("loop_filter_across_tiles_enabled_flag" and "loop_filter_across_tile_groups_enabled_flag") may be used to selectively enable in-loop filtering process across tile group boundaries. However, such syntax elements may complicate the filtering process when the tile groups are non-rectangular by disabling in-loop filtering across tile group boundaries while enabling filtering across tile boundaries. As shown in FIG. 8, the adaptive loop filter (ALF) operates with a 7x7 diamond filter 840 across the tile group boundaries of two tile groups 820. If the tile group 820 is non-rectangular, the filtering process is confused. A boundary checking process is mandatory for each pixel, each filter coefficient, to identify which pixel belongs to the current tile group. There is no difference between the temporal motion compensation case and the current picture reference case. If a boundary extension is applied at the tile group boundary for motion compensation, the padding process is also complicated.

[0111] According to an embodiment of the present disclosure, tile group level loop filtering (or any other operation) control at the tile group boundary is only allowed for rectangular tile groups, thus successfully avoiding confusion in the filtering and padding process.

[0112] Additionally, the flag "loop_filter_across_tile_groups_enabled_flag" may not be signaled when single_tile_per_tile_group_flag is 1. Thus, an embodiment of the present disclosure may include the following picture parameter set RBSP syntax, for example, as shown in Table 4:

[0113] [Table 4]

[0114] "loop_filter_across_tile_groups_enabled_flag" when set to 1 specifies that in-loop filtering operations are performed across tile group boundaries in images that reference the PPS. "loop_filter_across_tile_group_enabled_flag" when set to 0 specifies that in-loop filtering operations are not performed across tile group boundaries in images that reference the PPS. In-loop filtering operations include, for example, a deblocking filter operation, a sample adaptive offset filter operation, and an adaptive loop filter operation. If not present, the value of "loop_filter_across_tile_groups_enabled_flag" may be inferred to be equal to the value of loop_filter_across_tiles_enabled_flag.

[0115] In one embodiment, a method may be performed by at least one processor to decode sub-bitstreams of an encoded video stream, which may include encoded versions of multiple sub-images 510 for one or more images 500.

[0116] As shown in FIG. 9, the method may include at least one processor receiving a sub-bitstream (850). In one embodiment, the sub-bitstream may include information about one or more sub-images 510 of one or more images 500, and may not include information about other sub-images 510 of one or more images 500. After receiving the sub-bitstream, the at least one processor may decode the sub-images 510 included in the sub-bitstream. For example, assuming that the sub-bitstream includes information about a first sub-image and a second sub-image of an image, the at least one processor may decode the first sub-image of the image independently of the second sub-image according to the decoding techniques of the present disclosure using a tile and sub-image partitioning design (860). The step of decoding the first sub-image may include decoding tiles of the first sub-image. The at least one processor may further decode the second sub-image of the image according to the decoding techniques of the present disclosure using a tile and sub-image partitioning design (870). The step of decoding the second sub-image may include decoding tiles of the second sub-image.

[0117] In an embodiment, at least one processor may encode sub-images 510 of one or more images 500 in accordance with the tile and sub-image partitioning design of the present disclosure, and transmit one or more sub-bitstreams of a video bitstream including the one or more encoded sub-images 510 to one or more decoders for decoding in accordance with the tile and sub-image partitioning design of the present disclosure.

[0118] In an embodiment, a decoder of this disclosure (e.g., video decoder 210) may perform the decoding methods of this disclosure by accessing computer program code stored in memory and operating as instructed by the computer program code. For example, in one embodiment, the computer program code may include decoding code configured to cause the decoder to decode a first sub-image of an image independent of a second sub-image according to a decoding technique using the tile and sub-image partitioning design of this disclosure, and to cause the decoder to decode a second sub-image of the image independent of the first sub-image according to a decoding technique using the tile and sub-image partitioning design of this disclosure.

[0119] In an embodiment, an encoder of this disclosure (e.g., encoder 203) may perform the encoding method of this disclosure by accessing computer program code stored in a memory and operating as instructed by the computer program code. For example, in one embodiment, the computer program code may include encoding code configured to cause the encoder to encode one or more sub-images 510 of an image 500 according to the tile and sub-image partitioning design of this disclosure. The computer program code may further include transmission code configured to transmit one or more sub-bitstreams of a video bitstream, including the one or more encoded sub-images 510, to one or more decoders for decoding according to the tile and sub-image partitioning design of this disclosure.

[0120] The techniques described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 10 illustrates a computer system 900 suitable for implementing some embodiments of the present disclosure.

[0121] Computer software can be encoded using any suitable machine code or computer language, which may follow mechanisms such as assembly, compilation, linking, etc. to produce code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or via interpretation, microcode execution, etc.

[0122] The instructions may be executed by various types of computers or components thereof, such as personal computers, tablet computers, servers, smartphones, gaming consoles, and Internet of Things (IoT) devices.

[0123] 10 are exemplary in nature and are not intended to suggest any limitation to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the non-limiting embodiment of computer system 900.

[0124] The computer system 900 may include several human interface input devices. Such human interface input devices may be responsive to input by one or more users, for example, by tactile input (pressing a key, swiping, moving a data glove, etc.), audio input (voice, clapping hands, etc.), visual input (gestures, etc.), olfactory input (not shown). The human interface devices may further be used to capture several media that do not necessarily involve direct conscious input by a person, such as audio (speech, music, ambient sounds, etc.), images (scanned images, photographic images captured by a still camera, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0125] The input human interface devices may include one or more of a keyboard 901, a mouse 902, a trackpad 903, a touch screen 910, a data glove, a joystick 905, a microphone 906, a scanner 907, a camera 908 (only one of each is shown).

[0126] The computer system 900 may include several human interface output devices. Such human interface output devices may stimulate one or more of the user's senses, such as haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 910, data gloves, or joystick 905, although there may also be haptic feedback devices that do not function as input devices). For example, such devices may be audio output devices (speakers 909, headphones (not shown)), visual output devices (such as screens 910 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting output in more than three dimensions by means of two-dimensional video output, or stereoscopic output, etc., VR glasses (not shown), holographic displays, smoke generating devices (smoke tanks (not shown)), and printers (not shown)).

[0127] The computer system 900 may further include human-accessible storage devices and associated media therewith, such as CD / DVD ROM / RW 920, including media 921 such as CDs / DVDs, thumb-drives 922, removable hard drives or solid-state drives 923, legacy magnetic media such as tapes and floppy disks (not shown), optical media including specialized ROM / ASIC / PLD based devices (not shown) such as security dongles.

[0128] Those skilled in the art should further appreciate that the term "computer-readable medium" as used with respect to the subject matter disclosed herein does not include transmission media, carrier waves or other transitory signals.

[0129] The computer system 900 may further comprise an interface to one or more communication networks. The network may be, for example, wireless, wired, optical. The network may further be local, wide area, metropolitan, in-vehicle and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, mobile communication networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide area digital networks including cable, satellite, and terrestrial television, in-vehicle and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter attached to some general-purpose data port or peripheral bus (949) (e.g., USB port of the computer system 900), while other networks are typically integrated into the core of the computer system 900 by attaching to the system bus (e.g., an Ethernet interface to a PC computer system, or a mobile communication network interface to a smartphone computer system) as described below. Using any such network, the computer system 900 can communicate with other entities. Such communications may be one-way, receive-only (e.g., television broadcast), one-way transmit-only (e.g., device transmitting from CANbus to CANbus), or bidirectional to other computer systems, for example, using local or wide area digital networks. Such communications may include communications with cloud computing environment 955. As previously mentioned, several protocols and protocol stacks may be used for each such network and each network interface.

[0130] The human interface devices, human accessible storage, and network interface 954 described above may be attached to the core 940 of the computer system 900 .

[0131] The core 940 may include one or more central processing units (CPU) 941, graphic processing units (GPU) 942, specialized programmable processing units in the form of FPGAs (Field Programmable Gate Areas) 943, hardware accelerators for some tasks 944, etc. Such devices may be connected via a system bus 948, along with read only memory (ROM) 945, random access memory 946, internal mass storage such as internal non-user accessible hard drives, SSDs, etc. 947. In some computer systems, the system bus 948 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 948 or via a peripheral bus 949. Peripheral bus architectures include PCI, USB, etc. A graphics adapter 950 may be included in the core 940.

[0132] The CPU 941, GPU 942, FPGA 943, and accelerator 944 may execute a combination of instructions that may make up the computer code described above. The computer code may be stored in ROM 945 or RAM 946. Transient data may also be stored in RAM 946, whereas persistent data may be stored, for example, in internal mass storage device 947. A cache memory may be used to allow quick storage and retrieval of any memory device, which may be closely associated with one or more of the CPU 941, GPU 942, mass storage device 947, ROM 945, RAM 946, etc.

[0133] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0134] By way of example and not of limitation, a computer system having the architecture 900, and in particular the core 940, can provide functionality as a result of processor(s) (CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage devices as introduced above, as well as media associated with some storage devices of the core 940 that are non-transitory in nature, such as the core internal mass storage device 947 or ROM 945. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core 940. The computer-readable media can include one or more memory devices or chips according to particular needs. The software can cause the core 940, and in particular the processors therein (including the CPU, GPU, FPGA, etc.) to perform certain operations, or certain portions of certain operations, described herein, including the definition of data structures stored in the RAM 946 and the modification of such data structures according to the software-defined operations. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired into or embodied in circuitry (e.g., accelerator 944), which may operate in place of or in conjunction with software to perform certain operations, or portions of certain operations, described herein. References to software may encompass logic, where appropriate, and vice versa. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, where appropriate, circuitry embodying logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0135] While this disclosure has described several non-limiting embodiments, there are alterations, substitutions, and various substitute equivalents, which are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods that embody the principles of the present disclosure and are therefore within its scope, even if not explicitly shown or described herein. [Explanation of symbols]

[0136] 100 Communication Systems 110 First Terminal 120 Second Terminal 130 Second pair of terminals 140 Second pair of terminals 150 Communication Network 200 Streaming System 201 Video Source 202 Uncompressed Video Sample Stream 203 Video Encoder 204 Video Bitstream 205 Streaming Server 206 Streaming Client 209 Video Bitstream 210 Video Decoder 211 Video Sample Stream 212 Display device 213 Acquisition Subsystem 310 Receiver 312 Channels 315 Buffer Memory 320 Parser 321 Symbols 351 Scaler / Descaler Unit 352 Intra Image Prediction Unit 353 Motion Compensation Prediction Unit 355 Aggregation Device 356 Loop Filter Unit 357 Reference Image Memory 358 Current Image Memory 430 Source Encoder 432 encoding engine 433 Local Decoder 434 Reference Image Memory 435 Predictors 440 Transmitter 443 source video sequence 445 Entropy Encoder 450 Controller 460 Communication Channels 500 images 510 Sub-Images 520 Tile Group 530 tiles 610 Camera 611 Video Images 612 Video Images 620 observer view 621 First Observer View 630 Athlete Views 631 1st Athlete View 632 Second Athlete View 640 Compositor 650 Transcoder 660 Layout & Stream Selection 670 Network Monitor 680 Media Sink 690 Composite Video 820 Tile Group 840 Diamond Filter 900 Computer Systems 901 Keyboard 902 Mouse 903 Trackpad 905 Joystick 906 Mike 907 Scanner 908 Camera 909 Speaker 910 Touchscreen 920 Optical media 921 Medium 922 USB memory 923 Solid State Drive 940 cores 941 Central Processing Unit (CPU) 942 Graphics Processing Unit (GPU) 943 FPGA 944 Accelerator 945 Read-Only Memory (ROM) 946 Random Access Memory 947 Internal mass storage 948 System Bus 949 Surrounding Bus 950 Graphics Adapter 954 Network Interface 955 Cloud Computing Environment

Claims

1. A method, executed by at least one processor, for decoding a coded video stream, the coded video stream including coded versions of sub-images including tiles, the method comprising: receiving the coded video stream; decoding the coded video stream; Including, the coded video stream further includes a first flag indicating whether any processing and prediction across tile boundaries is rejected; The method, wherein the decoding step further includes a step of signaling a second flag indicating whether in-loop filtering operations that may be performed across the tile boundary are allowed based on a value of the first flag indicating that the any processing and the prediction across the tile boundary are not rejected.

2. The method of claim 1, wherein the first flag is full_parallel_decoding_enabled_flag.

3. The method of claim 1, wherein the coded video stream further includes a third flag indicating whether inter prediction operations may be performed across the tile boundaries.

4. The method described in claim 3, wherein the decoding step further includes a step of signaling the third flag based on the value of the first flag indicating that any processing across the tile boundary and the prediction are not rejected.

5. The method described in claim 4, wherein the third flag is inter_prediction_across_tiles_enabled_flag.

6. The method described in claim 4, wherein the decoding step further includes a step of signaling a fourth flag indicating whether an inter prediction operation may be performed across a tile group boundary based on a value of the third flag indicating that an inter prediction operation may be performed across the tile boundary.

7. The method described in claim 6, wherein the fourth flag is inter_prediction_across_tile_groups_enabled_flag.

8. The method of claim 1, wherein the decoding step further includes a step of signaling a third flag indicating whether in-loop filtering operations may be performed across tile group boundaries based on the value of the first flag indicating that any processing and prediction across the tile boundaries are not rejected.

9. The method of claim 8, wherein the third flag is loop_filter_across_tile_groups_enabled_flag.

10. A decoder configured to perform a method according to any one of claims 1 to 9.

11. A computer program product for causing at least one processor to carry out a method according to any one of claims 1 to 9.

Images