Method and apparatus for encoding and decoding a video stream with sub-pictures - Patents.com

Abstract

The present invention relates to an encoding method that includes encoding information that allows a decoder to infer splits of CTBs located to the right or below of sub-pictures whose width or height is not a multiple of the size of the CTB, if the sub-picture is not located to the right or below of the image. A corresponding decoding method for the generated bitstream is also proposed.

Description

[Technical field]

[0001] This disclosure relates to methods and devices for encoding and decoding video bitstreams that facilitate sub-picture replacement, and more particularly, to encoding and decoding video bitstreams that result from the merging of sub-pictures coming from different video bitstreams. [Background technology]

[0002] The size of a picture in a video bitstream may not correspond to a multiple of the size of the coding tree block (CTB) used in the encoding process. The CTB may be recursively split in the encoding to find a coding block size that optimizes the encoding process in particular. Thus, the CTB may be split up to the smallest coding block size. If the size of the picture is not a multiple of the size of the CTB, the right or bottom border of the picture intersects with the rightmost or bottommost CTB. In this case, an estimated split of the CTB is provided. Coding blocks that are outside the picture are typically not encoded.

[0003] When considering the division of an image into sub-pictures, the right-most or bottom-most sub-picture may contain an incomplete CTB that is subject to an inferred split, including some uncoded coding blocks.

[0004] The decoder is provided in the bitstream with the size of the image in pixels and the size of the CTBs, so it can determine the exact location of the right and bottom boundaries of the image and perform an inferential split of the rightmost and bottommost CTBs in the image.

[0005] Because the size of the subpictures is provided as an integer number of CTBs, given the displacement of a subpicture located at the right or bottom border of the image elsewhere in the image, the decoder cannot infer the split of the rightmost or bottommost CTB and the identity of the missing coding block, and decoding fails. Summary of the Invention

[0006] The present invention has been devised to address one or more of the above problems. An encoding method is proposed that includes encoding information that allows a decoder to infer splitting of a CTB located to the right and respectively below a subpicture whose width and height, respectively, are not multiples of the size of the CTB when the subpicture is not located to the right and respectively below the image. A corresponding decoding method for the generated bitstream is also proposed.

[0007] According to one aspect of the invention, there is provided a method for encoding video data comprising a picture into a bitstream, the method comprising the steps of: dividing the picture into sub-pictures; and, for at least one of the sub-pictures, encoding, in the bitstream, information indicating that a picture has a size that is a multiple of a size of a coding tree block and that a sub-picture is independently decodable; and encoding the coding tree blocks that constitute the sub-picture into a bitstream.

[0008] In one embodiment, the information pertains to all sub-pictures.

[0009] In one embodiment, the information is defined in a sequence parameter set, which is a syntax structure that contains syntax elements that apply to pictures in a bitstream.

[0010] In one embodiment, the sub-picture is further identified using a sub-picture identifier.

[0011] In one embodiment, defining sub-picture identifiers in a picture header is prohibited.

[0012] In one embodiment, the sub-picture identifiers defined in a picture parameter set must be identical in all picture parameter sets.

[0013] In one embodiment, the information is associated with a particular profile.

[0014] According to another aspect of the invention, there is provided a method for encoding video data comprising a picture into a bitstream, the picture being divided into sub-pictures, and for at least one sub-picture, encoding, in the bitstream, information indicative of a conformance window of the sub-picture; and encoding the coding tree blocks that constitute the sub-picture into a bitstream.

[0015] In one embodiment, the information indicating the compatibility window is defined in the SEI message.

[0016] In one embodiment, the information defines a left offset, a right offset, a top offset, and a bottom offset for the subpicture.

[0017] According to another aspect of the invention, there is provided a method for encoding video data comprising a picture into a bitstream, the picture being divided into sub-pictures, and for at least one sub-picture, encoding, in the bitstream, first information indicating that the sub-picture has a size that is a multiple of a size of a coding tree block and that the sub-picture is independently decodable; encoding, in the bitstream, second information indicative of a conformance window of the sub-picture; and encoding the coding tree blocks that constitute the sub-picture into a bitstream.

[0018] According to another aspect of the invention, a computer program product for a programmable device is proposed, the computer program product comprising a series of instructions for implementing the method according to the invention when loaded into and executed by the programmable device.

[0019] According to another aspect of the invention there is provided a computer readable storage medium storing computer program instructions for carrying out the method according to the invention.

[0020] According to another aspect of the invention there is provided a computer program product which when executed causes the computer to carry out the method of the invention.

[0021] According to another aspect of the invention, there is provided a device for encoding video data including a picture into a bitstream, the picture being divided into sub-pictures, and for at least one sub-picture, encoding, in the bitstream, information indicating that the sub-picture has a size that is a multiple of the size of the coding tree block and that the sub-picture is independently decodable; A device is provided having a processor configured to encode coding tree blocks that constitute a sub-picture into a bitstream.

[0022] According to another aspect of the invention, there is provided a device for encoding video data including a picture into a bitstream, the picture being divided into sub-pictures, and for at least one sub-picture, encoding, in the bitstream, information indicative of a conformance window of the subpicture; A device is provided having a processor configured to encode coding tree blocks that constitute a sub-picture into a bitstream.

[0023] According to another aspect of the invention, there is provided a device for encoding video data including a picture into a bitstream, the picture being divided into sub-pictures, and for at least one sub-picture, encoding, in the bitstream, first information indicating that the sub-picture has a size that is a multiple of a size of a coding tree block and that the sub-picture is independently decodable; encoding, in the bitstream, second information indicative of a conformance window of the sub-picture; A device is provided having a processor configured to encode coding tree blocks that constitute a sub-picture into a bitstream.

[0024] At least some of the methods according to the invention may be computer-implemented. Accordingly, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Furthermore, the invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

[0025] Since the present invention can be implemented in software, the present invention can be implemented as computer readable code for provision to a programmable device on any suitable carrier medium. Tangible non-transitory carrier media can include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices or solid state memory devices. Transient carrier media can include signals such as electrical, electronic, optical, acoustic, magnetic or electromagnetic signals, e.g. microwave or RF signals. [Brief description of the drawings]

[0026] Embodiments of the invention will now be described, by way of example only, with reference to the following drawings, in which: [Figure 1a] FIG. 1a shows two different application examples for region of interest combination. [Figure 1b] FIG. 1b shows two different application examples for region of interest combination. [Figure 2a] FIG. 2a shows some divisions in the coding system. [Figure 2b] FIG. 2b shows an example of a picture division in sub-pictures. [Figure 2c] FIG. 2c shows an example of a division of a picture into tiles and bricks. [Diagram 3] FIG. 3 shows the structure of a bitstream in an exemplary coding system VVC. [Figure 4] FIG. 4 illustrates diagrammatically the quadtree inference mechanism used in VVC for coding tree blocks that cross image boundaries. [Figure 5a] FIG. 5a shows the generation of a bitstream in which a border subpicture is moved to a non-border position. [Figure 5b] FIG. 5b shows the generation of a bitstream in which a border subpicture is moved to a non-border position. [Figure 6] FIG. 6 illustrates a method for encoding pictures of a video into a bitstream according to a first embodiment of the invention. [Figure 7] FIG. 7 illustrates the general decoding process of one embodiment of the present invention. [Figure 8] FIG. 8 shows the decoding process of a CTB encoded in a slice. [Figure 9] FIG. 9 shows an example where a picture is divided into 16 sub-pictures. [Figure 10] FIG. 10 shows an example where a picture is split into 16 sub-pictures with a picture-wide split inference boundary. [Figure 11]FIG. 11 illustrates the concept of a picture-wide boundary. [Figure 12] FIG. 12 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] 1a and 1b show two different application examples for combining regions of interest.

[0028] For example, FIG. 1a shows an example in which a picture (or frame) 100 from a first video bitstream and a picture 101 from a second video bitstream are merged into a picture 102 of a resulting bitstream. Each picture consists of four regions of interest, numbered 1 to 4. Picture 100 has been coded using coding parameters that result in high-quality coding. Picture 101 has been coded using coding parameters that result in low-quality coding. As is well known, a picture coded with low quality is associated with a lower bitrate than a picture coded with high quality. The resulting picture 102 combines regions of interest 1, 2, and 4 from picture 101 coded with low quality with region of interest 3 from picture 100 coded with high quality. The goal of such a combination is generally to obtain a region of interest, here region 3, with high quality while keeping the resulting bitrate reasonable by coding regions 1, 2, and 4 with low quality. Such kind of scenarios may occur especially in the context of omnidirectional content, which allows for higher quality for the content that is actually viewed while the rest has a lower quality.

[0029] Fig. 1b shows a second example where four different videos A, B, C and D are merged to form a resulting video. Picture 103 of video A is composed of attention areas A1, A2, A3 and A4. Picture 104 of video B is composed of attention areas B1, B2, B3 and B4. Picture 105 of video C is composed of attention areas C1, C2, C3 and C4. Picture 106 of video D is composed of attention areas D1, D2, D3 and D4. Picture 107 of the resulting video is composed by areas B4, A3, C3 and D1. In this example, the resulting video is a mosaic video of different attention areas of each original video stream. The attention areas of the original video streams are relocated and merged to new positions in the resulting video stream.

[0030] Video compression relies on block-based video coding in most coding systems such as HEVC, short for High Efficiency Video Coding, or the emerging VVC, short for Versatile Video Coding. In these coding systems, video is composed of a sequence of frames or pictures or images or samples that can be displayed at several different times. In the case of multi-layered video (e.g., scalable, stereo, 3D video), several pictures can be decoded to compose the resulting image and displayed at a single moment. A picture can also be composed of different image components, for example encoding luminance, chrominance or depth information.

[0031] The compression of a video sequence relies on several partitioning techniques for each picture. Figure 2a shows several partitions in a coding system. Pictures 201 and 202 are partitioned into coding tree units (CTUs), shown by dotted lines. CTUs are the basic units of encoding and decoding. For example, a CTU can code an area of ​​128x128 pixels.

[0032] A coding tree unit (CTU) can also be named block, macroblock, coding block. It can simultaneously code different image components or can be limited to only one image component. If an image contains several components, a CTU corresponds to one CTB for each component. In the following, the invention applies both to the CTU or CTB level.

[0033] As shown in Fig. 2a, a picture may be divided according to a grid of tiles indicated by thin solid lines. A tile is a picture portion and thus a rectangular region of pixels that can be defined independently of the CTU division. The boundaries of the tiles and the boundaries of the CTUs may be different. A tile may also correspond to a sequence of CTUs, as in the depicted example, meaning that the boundaries of the tiles and the CTUs coincide.

[0034] The tile definition provides for tile boundaries to break spatial coding dependencies, meaning that the coding of a CTU within a tile is not based on pixel data from another tile in the picture.

[0035] Some coding systems, for example VVC, provide the concept of slices. This mechanism allows the division of a picture into one or several groups of tiles. Each slice is composed by one or several tiles. Two different types of slices are provided as shown by pictures 201 and 202. The first type of slice is restricted to slices that form a rectangular area in the picture. Picture 201 shows the division of the picture into five different rectangular slices. The second type of slice is restricted to tiles that are consecutive in raster scan order. Picture 202 shows the division of the picture into three different slices that are composed of tiles that are consecutive in raster scan order. Rectangular slices are a structure chosen to handle regions of interest in a video. Slices can be coded in the bitstream as one or several NAL units. A NAL unit, short for Network Abstraction Layer unit, is a logical unit of data for the encapsulation of data in the coded bitstream. In the example of the VVC coding system, a slice is coded as a single NAL unit. If a slice is bi-stream coded as several NAL units, each NAL unit of the slice is a slice segment. A slice segment contains a slice segment header that contains the coding parameters of the slice segment. The header of the first segment NAL unit of a slice contains all the coding parameters of the slice. The slice segment headers of subsequent NAL units of the slice may contain fewer parameters than the first NAL unit. In such a case, the first slice segment is an independent slice segment and the subsequent segments are dependent slice segments.

[0036] In OMAF v2 ISO / IEC 23090-2, a subpicture is a part of a picture that represents a spatial subset of the original video content, which has been divided into spatial subsets by the content creator before video encoding. A subpicture can be, for example, one or more slices forming a rectangular area.

[0037] Figure 2b shows an example of a division of a picture in sub-pictures. A sub-picture represents a picture portion covering a rectangular area of ​​the picture. Each sub-picture can have different size and coding parameters. For example, a different tile grid and slice division can be defined for each sub-picture. In Figure 2b, picture 204 is subdivided into 24 sub-pictures, including sub-pictures 205 and 206. These two sub-pictures further describe the division in tile grid and slices similar to pictures 201 and 202 in Figure 2a. In the second example, the tile and brick division is defined at the picture level, instead of being defined per sub-picture. A sub-picture is then defined as one or more slices forming a rectangular area.

[0038] Figure 2c shows an example of partitioning using brick partitioning. Each tile can contain a set of bricks. A brick is a contiguous set of CTUs that form a line within the tile. For example, frame 207 in Figure 2c is partitioned into 25 tiles. Each tile contains exactly one brick, except for the rightmost row of tiles, which contains two bricks per tile. For example, tile 208 contains two bricks 209 and 210. When brick partitioning is used, a slice contains either a brick from one tile, or several bricks from other tiles. In other words, a VCL NAL unit is a set of bricks instead of a set of tiles.

[0039] FIG. 3 shows the structure of a bitstream in an exemplary coding system VVC.

[0040] The bitstream 300 from the VVC coding system consists of an ordered sequence of syntax elements and coded data. The syntax elements and coded data are arranged in NAL units 301-305. There are different NAL unit types. The network abstraction layer provides the ability to encapsulate the bitstream into different protocols such as RTP / IP, which stands for Real Time Protocol / Internet Protocol, ISO Base Media File Format, etc. The network abstraction layer also provides a framework for packet loss resiliency.

[0041] NAL units are divided into VCL NAL units and non-VCL NAL units, where VCL stands for Video Coding Layer. VCL NAL units contain the actual coded video data. Non-VCL NAL units contain additional information, which may be parameters required for decoding the coded video data or supplementary data that may improve the usability of the decoded video data. NAL units 305 correspond to slices and constitute the VCL NAL units of the bitstream. Different NAL units 301-304 correspond to different parameter sets, these NAL units are non-VCL NAL units. VPS NAL unit 301, where VPS stands for Video Parameter Set, contains parameters defined for the entire video, i.e. the entire bitstream. The naming of VPS can vary, for example becoming DPS in VVC. Alternatively, VPS and DPS are different parameter set NAL units. DPS (short for Decoder Parameter Set) NAL units may define more static parameters than those in VPS. In other words, the parameters of the DPS change less frequently than the parameters of the VPS. The SPS NAL unit 302, where SPS stands for Sequence Parameter Set, contains parameters defined for a video sequence. In particular, the SPS NAL unit can define sub-pictures of a video sequence. The syntax of the SPS includes, for example, the following syntax elements: [Table 1] The descriptor sequence gives the encoding of the syntax element, where u(1) means that the syntax element is encoded using 1 bit, and ue(v) means that the syntax element is encoded using a variable length encoding of the unsigned integer zeroth order Exp-Golomb encoded syntax element with left bit first.

[0042] The presence of subpictures in a picture depends on the value of subpics_present_flag. If this flag is equal to 0, it indicates that the picture does not contain subpictures. If it is equal to 1, the set of syntax elements specifies the subpictures in the frame. The syntax element max_subpics_minus1 specifies the maximum number of subpictures in a picture of the video sequence. The SPS then defines the subpictures divided in a grid of subpicture grid elements of size defined by subpic_grid_col_width_minus1 and subpic_grid_row_height_minus1. Each grid element specifies a subpicture index subpic_grid_idx[i][j] (where i and j are the coordinates of the element within the rectangle). The value of subpic_grid_idx[i][j] is the same number as there are subpictures in the picture of the video sequence. The subpic_grid_idx[i][j] syntax element is the identifier of the subpicture. All grid elements that share the same index value form a rectangular region that corresponds to the subpicture whose index (or identifier) ​​is equal to the index value. The subpic_treated_as_pic_flag[i] syntax element indicates whether the subpicture boundary should be treated as a picture boundary other than for loop filter processing. The loop_filter_across_subpic_enabled_flag[i] syntax element indicates whether the loop filter is applied across subpicture boundaries.

[0043] The PPS NAL unit 303, where PPS stands for Picture Parameter Set, contains parameters defined for a picture or a group of pictures. The APS NAL unit 304, where APS stands for Adaptation Parameter Set, contains parameters for a loop filter of an adaptive loop filter (ALF) or a reshaper model (or luma mapping with chroma scaling model) that is typically defined at the slice level. The bitstream may also contain a SEI NAL unit, where APS stands for Supplemental Enhancement Information. The periodicity with which these parameter sets occur within the bitstream is variable. A VPS defined for the entire bitstream needs to occur only once in the bitstream. Conversely, an APS defined for a slice may occur once for each slice in each picture. In practice, different slices may rely on the same APS, and thus, in general, there are fewer APSs than slices in each picture. If a picture is divided into subpictures, a new parameter set or PPS may be defined for each subpicture or group of subpictures.

[0044] The VCL NAL unit 305 contains slices, which can correspond to an entire picture or subpicture, a single tile or multiple tiles, or a single brick or multiple bricks. A slice consists of a slice header 310 and a raw byte sequence payload RBSP 311 that contains the bricks.

[0045] The syntax of the PPS as proposed in the current version of VVC also includes syntax elements that specify the size of the picture and luma samples, and the division of each picture into tiles, bricks, and slices.

[0046] The syntax of the PPS as proposed in the current version of VVC is organized as follows: [Table 2] The descriptor sequence gives the encoding of the syntax element, where u(1) means that the syntax element is encoded using 1 bit, and ue(v) means that the syntax element is encoded using a left-bit-first unsigned integer zeroth order Exp-Golomb encoded syntax element, which is a variable length encoding. The syntax elements pic_width_in_luma_samples and pic_height_in_luma_samples specify the width and height of the picture in luma samples.

[0047] If the number of tiles in a picture is greater than one (single_tile_in_pic_flag is equal to 0), the PPS defines several syntax elements (not shown in the table above) that specify the tile division within a frame as a grid of tiles.

[0048] If bricks are present (brick_splitting_present_flag is equal to 1), the PPS contains a loop for each tile in the tile grid to indicate whether the tile is split into bricks. If the tile contains bricks, the brick configuration of the tile is coded in the PPS.

[0049] Slice division is expressed by the following syntax elements:

[0050] The syntax element single_tile_in_pic_flag indicates whether the picture contains a single tile, in other words, if this flag is true, there is only one tile and one slice in the picture.

[0051] The single_brick_per_slice_flag indicates whether each slice contains a single brick. In other words, if this flag is true, all the bricks of the picture belong to different slices.

[0052] The syntax element rect_slice_flag indicates that the slices of a picture form a rectangular shape as represented in picture 201.

[0053] If present, the syntax element num_slices_in_pic_minus1 is equal to the number of rectangular slices in the picture minus one.

[0054] Then, a syntax element (not represented in the table above) goes through all slices of the picture and encodes the position of each slice relative to the brick partition in a "for loop", where the index of the slice position parameter is the index of the slice.

[0055] Slice identifiers are specified if signalled_slice_id_flag is equal to 1. In this case, the signalled_slice_id_length_minus1 syntax element indicates the number of bits used to encode each slice identifier value. The slice_id[] association table is indexed by slice index and contains identifiers of slices. If signalled_slice_id_flag is equal to 0, slice_id is indexed by slice index and contains slice indexes of slices.

[0056] In summary, the PPS contains syntax elements that allow determining the location of slices within a frame. Since a sub-picture forms a rectangular region within a frame, it is possible to determine the set of slices, tiles and bricks that belong to a sub-picture.

[0057] The slice header contains the slice address with the following syntax in the current VVC version: [Table 3] If the slice is not rectangular, the slice header indicates the number of tiles in the slice NAL unit with the help of the num_tiles_in_slice_minus1 syntax element.

[0058] Each tile 320 may include a tile segment header 330 and tile segment data 331. The tile segment data 331 includes encoded coding blocks 340. In the current version of the VVC standard, there is no tile segment header and the tile segment data includes coding block data 340.

[0059] In a variant, if the video sequence includes sub-pictures, the syntax of the slice header becomes: [Table 4] A slice header contains a slice_subpic_id syntax element that specifies the identifier of the subpicture it belongs to (e.g., corresponding to one of the values ​​of subpic_grid_idx[i][j] defined in the SPS). As a result, all slices that share the same slice_sub_pic_id in a video sequence belong to the same subpicture.

[0060] For illustrative purposes only, FIG. 4 shows a schematic of the quadtree inference mechanism used in VVC for coding tree blocks crossing image boundaries. In VVC, images are not restricted to having a multiple of the width and height of the coding tree block size. Then, the coding tree block at the right edge of the frame can cross the right boundary 401 of the image, and the coding tree block at the bottom edge of the frame can cross the bottom boundary 402 of the image. In these cases, VVC defines a quadtree inference mechanism for coding tree blocks crossing the boundary. This mechanism consists in recursively splitting any coding blocks of coding tree blocks crossing the image boundary until there are no more coding blocks crossing the boundary or until a maximum quadtree depth is reached for these coding tree blocks. For example, coding tree block 403 is not automatically split, while coding tree blocks 404, 405 and 406 are split. There is no signaling of the inferred quadtree, and the decoder must infer the same quadtree on the image boundary. However, the automatically obtained quadtree may be further refined for coding tree blocks that are within the frame, for example by signaling partitioning information for these coding tree blocks (if the maximum quadtree depth is not reached), as in 407.

[0061] When dividing the CTB into coding blocks, there is a smallest coding block into which it cannot be divided. The size of this smallest square coding block is given by MinCBSizeY. In some examples, MinCBSizeY is equal to 4.

[0062] During encoding, coding blocks of the coding tree that are located outside the image are typically not coded in the bitstream. During decoding, the decoder uses the same quadtree inference mechanism and knows that these coding blocks are not coded. The decoder can decode other coding blocks that are in the image that were coded in the bitstream. The resulting coding tree is coded into the bitstream by the encoder. The decoder relies on this coded coding tree information to correctly identify the blocks to decode and reconstruct the image. In particular, the coded coding tree contains a parameter, for example a parameter called split_cu_flag that specifies whether the coding unit is split or not.

[0063] 5a and 5b show the generation of a bitstream in which a border sub-picture is moved to a non-border position.

[0064] In this example, the first bitstream 500 contains four subpictures 1 HQ ~4 HQ This bitstream represents a high quality version of the video. The second bitstream 501 is made up of four subpictures 1 LQ ~4 LQ This bitstream represents a lower quality version of the same video. Bitstream 502 is created by merging and rearranging some subpictures emitted from bitstreams 500 and 501.

[0065] In particular, bitstream 502 contains subpicture 2 HQ , 1 LQ , 4 HQ and 3 LQ where subpicture 2 HQ is moved from its top right position in bitstream 500 to its top left position in bitstream 502, and subpicture 4 HQis moved from its bottom right position in bitstream 500 to its bottom left position in bitstream 502, and subpicture 1 LQ is moved from its top left position in bitstream 501 to its top right position in bitstream 502, and subpicture 3 LQ is moved from a bottom left position in bitstream 501 to a bottom right position in bitstream 502.

[0066] Bitstream 500 is composed of pictures with widths that are not multiples of the coding tree block (CTB) size, as shown in Figure 5b. Thus, the rightmost coding tree blocks are subject to inferred splitting. These rightmost coding blocks of the CTB are not coded in the bitstream. These uncoded coding blocks are represented by hatched area 503. In bitstream 502, subpicture 2 is HQ and 4 HQ An uncoded coding block 504 is placed in the center of the image on the right border of the image.

[0067] The size of the pixels (or luma samples) of the image is coded into the bitstream. This information allows the decoder to know the boundaries of the image and to estimate the estimated division of the rightmost CTB in the image. The decoder then knows which coding blocks are not coded and can decode the coding blocks that make up the image.

[0068] Subpicture 2 HQ and 4 HQA problem occurs when a subpicture such as is moved from the rightmost part of the image to another location, such as in bitstream 502. The size of the subpicture is coded into the bitstream as an integer number of CTBs. Due to this peculiarity in the standard, it is not possible for the decoder to infer the right boundary of the subpicture and the corresponding inferred division of the subpicture's rightmost CTB. The decoder expects all coding blocks of the subpicture's rightmost CTB to be coded into the bitstream. Since some of them are missing, the decoding fails.

[0069] It is necessary to find a way to rearrange the subpictures within an image without re-encoding them.

[0070] This problem can be solved by proposing an encoding method that includes the encoding of information that allows the decoder to infer the division of the CTBs located at the right and bottom of the subpictures whose width and height are not multiples of the size of the CTBs when the subpictures are not located at the right and bottom of the image. A corresponding decoding method for the generated bitstream is also proposed.

[0071] A sub-picture is a rectangular region within a picture represented by one or more slices. Sub-pictures are defined, for example, in SPS or PPS NAL units. The encoder may indicate for each sub-picture that their boundaries are treated as picture boundaries, meaning that the sub-pictures can be decoded independently of each other. Intra- and inter-prediction processes are constrained to only use prediction information from the same sub-picture in the current and reference frames. Filtering of sub-picture boundaries is controlled by a flag defined for each sub-picture. This flag allows applying or not applying a loop filter at the boundaries of the sub-picture. When decoding a CTB from one slice, the decoder determines the index or identifier of the sub-picture to which the CTB belongs.

[0072] In the following, the use of sub-pictures allows to decode regions of a video sequence at different positions, and potentially also to move the sub-picture at the new decoding position. For this reason, in a preferred embodiment, sub-picture boundaries are treated as picture boundaries (to ensure that motion prediction is constrained to allow independent decoding of sub-pictures), and the loop filter is disabled at sub-picture boundaries. In the case of VVC, these two conditions are met when the flags subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] of a sub-picture are equal to 1 and 0, respectively. In the case of HEVC, a sub-picture is, for example, a set of slices that belong to a motion-constrained tile set according to the HEVC specification.

[0073] FIG. 6 illustrates a method for encoding pictures of a video into a bitstream according to a first embodiment of the invention.

[0074] In a first step 601, a picture is divided into sub-pictures.

[0075] In step 602, the size of the sub-pictures is determined. The width and height of each sub-picture is a function of the regions of interest present in the input video sequence. Typically, the size of each sub-picture is such that it contains one region of interest. The size of the sub-pictures is determined in pixels (or luma samples). If the size of the sub-picture is not a multiple of the size of the CTB, it is determined that the sub-picture requires a specific split inference process. In particular, this may occur when the encoder merges two bitstreams as shown in Figure 5a.

[0076] In step 603, the size of the subpicture is coded into the bitstream. This size is information indicating the location of the partition inference boundary of the subpicture if a partition inference process is required. If necessary, information indicating the necessity of the partition inference process for this subpicture is coded into the bitstream. The information provided in the bitstream in this step allows the decoder to perform the partition inference process.

[0077] In step 604, at least one slice constituting the subpicture is coded into the bitstream. If a partition inference process is required, the encoder divides the subpicture into two parts. The first part is the area between the left and top boundaries of the subpicture and the horizontal and vertical partition inference boundaries. The second part is the area between the partition inference boundary and the bottom and right boundaries of the subpicture. For example, the white area in the 2HQ subpicture of Fig. 5b corresponds to the first part, and the hatched area corresponds to the second part. In this document, the first part of the subpicture is called the useful part of the subpicture. Coding units that are outside the useful part of the subpicture according to this partition inference process are not coded.

[0078] In an alternative embodiment, coding blocks that are outside the useful portion of the sub-picture are encoded with padding data.

[0079] In another alternative embodiment, a flag is inserted into the bitstream to indicate whether coding blocks outside the useful portion of the subpicture are coded with padding data or not. Typically, an encoder specifies in a picture parameter set NAL unit that a subpicture includes coded coding blocks of padding data for coding units that are outside the useful portion of the subpicture. For example, a flag is associated with each subpicture identifier to specify whether to provide padded coding data for coding units outside the useful portion of the subpicture.

[0080] In some embodiments, the use of the split inference process for each tile can be known from the bitstream context, and in these embodiments, no flag needs to be coded to signal the use of the sub-picture split inference process.

[0081] It is proposed to introduce a new syntax element in one parameter set NAL unit, e.g., SPS, which allows to obtain the same split inference for CTBs at the right and / or bottom border of one subpicture when moved to different positions in the merged bitstream.

[0082] These syntax elements allow the decoder to determine the size of the skipped coding block in the last CTB row and / or the last CTB column of a sub-picture when the sub-picture boundary is treated as a picture boundary.

[0083] As a result, when moving a subpicture from its rightmost position in a picture to another position, the merging or encoding operation consists, for example, of determining the size of the skipped coding block in the last CTB row and column, and then specifying it in the SPS associated with the subpicture. The decoding of the merged bitstream uses these values ​​to determine the usable size of the subpicture.

[0084] For example, the syntax of an SPS includes the following syntax elements: [Table 5] According to the proposed embodiment, some new syntax elements are introduced, marked in bold in the table. The semantics of these new syntax elements are as follows:

[0085] subpic_split_inference_flag equal to 1 indicates that the last CTB row or the last CTB column of the subpicture is not complete. The inference process for CTB splitting of the last CTB row and column of the subpicture takes into account the available size of the subpicture to determine the value of split_cu_flag.

[0086] subpic_split_inference_flag equal to 0 indicates that all CTBs of the subpicture are complete. Split inference processing is not required for the CTBs on the last CTB row and column of the subpicture.

[0087] subpic_split_inference_ctb_width[i], if present, specifies the actual width of the CTB in the rightmost column of the CTB of the ith subpicture (i.e., the width in pixels of the CTB in the useful part of the ith subpicture). The value of subpic_split_inference_ctb_width[i] is specified in units of MinCbSizeY-wide coding blocks and can be in the range [0, CtbSizeY / MinCbSizeY-1]. If not present, the value of subpic_split_inference_ctb_width[i] is inferred to be equal to CtbSizeY / MinCbSizeY, which corresponds to the width of the CTB. This coding syntax element is coded, for example, using a fixed-length code equal to 7 bits or log2(CtbSizeY / MinCbSizeY-1,). It is also possible to use Exp-Golomb coding.

[0088] subpic_split_inference_ctb_height[i], if present, specifies the actual height of the CTB in the last row of the CTB of the ith subpicture. The value of subpic_split_inference_ctb_height[i] is specified in coding block units of MinCbSizeY height and can be in the range [0, CtbSizeY / MinCbSizeY-1]. If not present, the value of subpic_split_inference_ctb_height[i] is inferred to be equal to CtbSizeY / MinCbSizeY, which corresponds to the height of the CTB. This coding syntax element is coded using a fixed length code, for example, equal to 7 bits or log2(CtbSizeY / MinCbSizeY-1,). It is also possible to use Exp-Golomb coding.

[0089] The subpic_split_inference_flag allows the encoder to indicate to the decoder that a specific process is required to process the CTBs of the last CTB rows and columns of some subpictures. The rightmost and / or bottommost CTB rows are meant to contain coding blocks that the encoder has not coded. The division of these CTBs into blocks is handled by a coding tree partitioning process that can infer the division of each block as a function of the values ​​of the subpic_split_inference_ctb_width[i] and subpic_split_inference_ctb_height[i] syntax elements for the i-th subpicture.

[0090] The actual size of the rightmost and bottommost CTB of the ith subpicture (specified by subpic_split_inference_ctb_width[i] and subpic_split_inference_ctb_height[i]) is in units of the minimum coding block size (MinCbSizeY, MinCbSizeY) specified in the SPS. The actual size of the CTB cannot exceed the maximum size of a coding tree block specified in the SPS (CtbSizeY, CtbSizeY). As a result, the range of subpic_split_inference_ctb_height and subpic_split_inference_ctb_width is 0 to 1 minus the ratio of the maximum CTB size to the minimum coding block size.

[0091] In one alternative, to simplify the analysis of the SPS, subpic_split_inference_ctb_width[i] and subpic_split_inference_ctb_height[i] are expressed in units of several luma samples (usually four luma samples). In fact, the size of the CTB and the minimum size of the coding block can be specified in the SPS and defined after the subpic_split_inference_ctb_width[i] and subpic_split_inference_ctb_height[i] syntax elements. In this case, the location of the split inference boundary is expressed as an integer multiple of the size of the smallest coding block of the coding tree block. Furthermore, if subpic_split_inference_ctb_width[i] and subpic_split_inference_ctb_height[i] are defined in different parameter set NAL units, the actual size of each subpicture can be calculated regardless of the CTB and coding block sizes.

[0092] The following pseudocode determines the boundary of the ith subpicture that triggers a split in the coding tree: The location of the vertical boundary is represented by the variable SubPicRightSplitInferenceBoundary[i], and the horizontal boundary is represented by the variable SubPicBotSplitInferenceBoundary[i].

[0093] The position in luma samples of the vertical border of the ith subpicture is equal to the position of the CTB-aligned right border of the ith subpicture minus the subpic_split_inference_ctb_width[i] syntax element defined in the SPS. The position in luma samples of the horizontal border of the ith subpicture is equal to the position of the CTB-aligned bottom border of the ith subpicture minus the subpic_split_inference_ctb_height[i] syntax element defined in the SPS.

[0094] The variables SubPicRightSplitInferenceBoundary[i] and SubPicBotSplitInferenceBoundary[i] are derived as follows: for (i=0;i <NumSubPics;i++){ SubPicRightSplitInferenceBoundary[i]=(SubPicLeft[i]+SubPicWidth[i])*(subpic_grid_col_width_minus1+1)*4-CtbSizeY+subpic_split_inference_ctb_width[i]*MinCbSizeY; SubPicBotSplitInferenceBoundary[i]=(SubPicTop[i]+SubPicHeight[i])*(subpic_grid_row_height_minus1+1)*4-CtbSizeY+subpic_split_inference_ctb_height[i]*MinCbSizeY;} Thus, the proposed new syntax elements related to subpictures, subpic_split_inference_flag, subpic_split_inference_ctb_width[i], and subpic_split_inference_ctb_height[i], allow the encoder to indicate to the decoder the information it needs to be able to infer the splitting of an incomplete CTB and to determine coding blocks that are outside the useful portion of the subpicture.

[0095] The coding tree syntax includes a flag, split_cu_flag, that indicates whether a coding block is split. A split is inferred when a coding tree block is not entirely within the bounds of the useful portion of a subpicture, and as a result, the flag is not coded.

[0096] split_cu_flag equal to 0 specifies that the coding block is not split. split_cu_flag equal to 1 specifies that the coding block is split into four coding blocks using quad partitioning, as indicated by the syntax element split_qt_flag, or split into three coding blocks using ternary partitioning, or split into two coding blocks using binary partitioning, as indicated by the syntax element mtt_split_cu_binary_flag. The binary or ternary partitioning can be either vertical or horizontal, as indicated by the syntax element mtt_split_cu_vertical_flag.

[0097] If split_cu_flag is not present in the encoding of a coding block, the value of split_cu_flag is inferred as follows:

[0098] The value of split_cu_flag is inferred to be equal to 1 if one or more of the following conditions are true: x0+cbWidth is greater than SubPicRightSplitInferenceBoundary[SubPicIdx].

[0099] y0+cbHeight is greater than SubPicBotSplitInferenceBoundary[SubPicIdx].

[0100] Otherwise, the value of split_cu_flag is inferred to be equal to 0.

[0101] The above pseudocode determines whether a coding block located at position (x0, y0) in luma sample coordinates with width (or height) equal to cbWidth (or cbHeight) is completely inside the useful part of the current subpicture with index equal to SubPicIdx. If it is outside this region, the encoder (and decoder) infers that the coding block is split (split_cu_flag equal to 1).

[0102] When a coding block is inferred to be split because its boundary is outside the useful part of the actual pixel boundary of the subpicture, there are three possible partitions. The first one is a quad-tree partition, i.e. the coding block is further partitioned into four square coding blocks of equal size. The second is a partition into two coding blocks separated by either a vertical or horizontal boundary that spans the entire coding block. This partition is a binary tree partition. Finally, the third partition is a ternary tree partition, which splits the coding block into three blocks separated by a vertical or horizontal boundary.

[0103] In addition to the split inference process, the encoder (and similarly the decoder) applies a specific encoding process to coding blocks outside the useful part of the subpicture. In the first embodiment, these coding blocks are not encoded. For each split coding block, the encoder checks whether the y-axis (or x-axis) coordinate of the top boundary (or left boundary) of the block is equal to the SubPicBotSplitInferenceBoundary (or SubPicRightSplitInferenceBoundary) value of the current subpicture. In such a case, the encoder skips encoding the block. As a result, no coded syntax elements are provided for these coding blocks. As a result, when encoding neighboring coding units, these skipped coding blocks are considered to be unavailable for inter-prediction and intra-prediction. Similarly, the temporal prediction is restricted to avoid using pixel information from these skipped coding blocks.

[0104] Figure 7 shows a general decoding process of an embodiment of the present invention. In step 701, the decoder determines the size of the sub-pictures of a frame, typically their width and height, by parsing the SPS and PPS of the stream. In particular, it determines the slice NAL units that belong to each sub-picture. In step 702, the decoder decodes the slices of the sub-pictures that form the picture. The decoding process of the coded CTBs in each slice is explained in more detail with reference to Figure 8. Step 703 consists in generating an output picture from the decoded picture. In particular, the decoder can optionally apply some cropping operations to the decoded sub-pictures.

[0105] Figure 8 shows the process 702 of decoding CTBs coded in slices. In step 801 it is checked whether there are any slices left to decode. If all slices have been decoded the process ends. For a given slice, the decoder determines in step 802 the identifier of the sub-picture to which the slice belongs.

[0106] Then, in step 803, it is checked whether there are any CTBs remaining to be decoded in the current slice. For a given CTB, in step 804, the decoder parses the syntax elements of the CTB. When the CTB is on the rightmost or bottommost boundary of the subpicture with the identifier determined in 802, the decoder applies a split inference process to the CTB if the process is required for the current slice. Then, once the CTB is split, in step 805, the decoder decodes each coding block resulting from the inference process and falling within the subpicture. Coding blocks that are outside the useful portion of the subpicture are not decoded since they are not present in the bitstream. In some embodiments, coding blocks that are outside the useful portion of the subpicture are decoded if they are coded with padding data.

[0107] The subpicture decoding process results in subpictures with sizes that are not multiples of the CTB size in the right column or below the incomplete CTB. When composing the subpictures to compose the final frame for rendering, the decoder must determine the values ​​of the pixels that are in these bands of pixels. An example of a band of undetermined pixels is shown in FIG. 5b, band 504. This is because subpictures in a frame are restricted to be composed of an integer number of CTBs. As a result, the subpicture contains a set of matching decoded pixels that correspond to the region between the origin of the subpicture and the split inference boundary. The remaining region corresponds to undefined or pixel values ​​that are not intended to be displayed. In some embodiments, the decoder may set the values ​​of the pixels in the remaining band to zero. In some other embodiments, the decoder may set the values ​​of the pixels in the remaining band by replicating the value of the closest matching pixel.

[0108] Since these bands of undetermined pixels are not intended to be displayed, they can be suppressed in the resulting frame by shifting adjacent subpictures. For example, subpicture 1 in bitstream 502 of FIG. 5b LQ and 3 LQ may be left shifted by the width of band 504.

[0109] In this embodiment, the decoder shifts the decoded pixels of subpictures to the right and below the current subpicture so that they are aligned with the right and bottom inferred split boundaries. The decoded positions in the luma samples of the coding block take into account that some subpictures are not complete.

[0110] The decoder has all the information necessary to identify the bands of undetermined pixels, and in one embodiment, in post-decoding processing after the decoder has decoded all the sub-pictures, it proceeds to shift the sub-pictures to remove these bands.

[0111] In another embodiment, the decoder can integrate the shifting operation in the decoding process, where new variables are defined by the decoder and used during the decoding of each CTB to correctly decode the coding block in its final position.

[0112] The SubPicOffsetX[i] and SubPicOffsetY[i] variables indicate the offsets to be subtracted, respectively, from the X-axis and Y-axis coordinates of a pixel in the coding block of the i-th subpicture to be placed in the correct position.

[0113] For example, the xCtbShifted and yCtbShifted variables are the new coordinates of the top-left pixel of the CTB at the CtbAddrInRs address in the raster scan order of the picture after the shift operation.

[0114] The following pseudocode calculates these coordinates, where CtbAddressInRs is the raster scan address of the CTB in the picture, PicWidthInCtbsY is the width of the picture in CTB units, CtbLog2SizeY is the log2 of the size of the CTB in luma samples, and SubPicIdx is the index of the subpicture containing the current CTB.

[0115] xCtbShifted=(CtbAddrInRs % PicWidthInCtbsY)< <CtbLog2SizeY yCtbShifted=(CtbAddrInRs / PicWidthInCtbsY)< <CtbLog2SizeY if(subpic_split_inference_flag){ xCtbShifted-=SubPicOffsetX[SubPicIdx] yCtbShifted-=SubPicOffsetY[SubPicIdx] } The first two lines calculate the x-axis and y-axis coordinates of the origin of the coding tree block in luma sample units. These values ​​do not take into account skipped pixels, if any, in adjacent subpictures. Then, if subpic_split_inference_flag is equal to 1, indicating that the subpicture may contain skipped blocks, the sum of the widths of the skipped pixels of the subpictures located to the left (or above) of the coding tree block is subtracted from the xCtbShifted (or yCtbShifted) coordinate.

[0116] The encoder determines the shift offset for the x-axis (SubPicOffsetX[i]) and y-axis (SubPicOffsetY[i]) coordinates of each SubPicture as follows: First, for a given subpicture, it determines the set of subpictures to the left of the current subpicture. For each subpicture in this set, the shift offset is equal to the difference between the right boundary of the last CTB row and the right subpicture inference boundary. Typically, this value is equal to CtbSizeY-subpic_split_inferernce_ctb_width[j]*MinCbSizeY, where j is the index of the subpicture in the set and CtbSizeY is the size of the CTB in luma samples or pixels. The value of the shift offset for the x-axis is equal to the sum of CtbSizeY-subpic_split_inferernce_ctb_width[j]*MinCbSizeY for each j that is equal to the index of the subpicture in the set of the left subpicture. Similarly, the value of the y-axis shift offset is equal to the sum of CtbSizeY-subpic_split_inferernce_ctb_height[j]*MinCbSizeY for each j equal to the index of the subpicture in the set of upper subpictures above the current subpicture.

[0117] Figure 9 shows the case where a picture is divided into 16 sub-pictures, where the bands of undetermined pixels are represented by hatched bands. The arrows represent the shifting of the sub-pictures to remove the bands of undetermined pixels. If all vertical bands, respectively horizontal bands, have the same size, the resulting image can be understood to be rectangular. It is also true that each column of a sub-picture contains the same number of horizontal bands and each row of a sub-picture contains the same number of vertical bands.

[0118] It should be understood that if these conditions are not met, the resulting image will not be rectangular.

[0119] In one embodiment, the possibility to specify different inference boundaries for sub-pictures is constrained to avoid defining pictures with non-rectangular picture shapes after shifting operations: in particular, the height (or width) in luma samples of each column (or row) of a sub-picture must be equal after shifting or cropping of the decoded frame.

[0120] The possibility of specifying different inference boundaries in a given row (or column) of a subpicture may lead to complex operations of shifting the decoded samples in each subpicture with different shift offsets. For this reason, in one embodiment, a constraint can be defined that the subpicture inference boundaries are aligned across the picture. For example, Figure 10 shows the case where a picture is divided into 16 subpictures that respect this constraint.

[0121] FIG. 11 illustrates the concept of a picture-wide boundary.

[0122] Alternatively, the encoder determines two types of boundaries for subpictures. First, a subpicture boundary is aligned with other subpicture boundaries that collectively span the entire picture width or height. This type of subpicture boundary is called a picture-wide boundary. For example, the bottom boundary of subpicture 0 in Figure 11 is a picture-wide boundary. Conversely, the bottom boundary of subpicture 1 is not picture-wide.

[0123] In one embodiment, the encoder constrains the subpicture arrangement so that the last CTB row of a subpicture with a bottom border that is not picture-wide does not use inference split. In other words, the split inference boundary is aligned to the CTB boundary, e.g., the size of subpic_split_inference_ctb_height[i] is equal to the size of the CTB. The same principle applies to the last CTB column of a subpicture with a right border that is picture-wide. As an alternative, the encoder determines the type of the border of each subpicture and encodes the value of subpic_split_inference_ctb_width[i] (or subpic_split_inference_ctb_height[i]) if the right (or bottom) border of the subpicture is a picture-wide border. If the right (or bottom) border is not picture-wide, subpic_split_inference_ctb_width[i] (or subpic_split_inference_ctb_height[i]) is not coded and is inferred to be equal to the size of the CTB.

[0124] The encoder may apply a second constraint to subpictures whose bottom (or right) boundary is the same picture-wide boundary. For example, the constraint may be that subpic_split_inference_height[i] is equal for all i-th subpictures of a set of subpictures when the boundary is below the subpicture. Similarly, subpic_split_inference_width[i] may be equal for all i-th subpictures of a set of subpictures that have a common right picture-wide boundary.

[0125] In addition, the encoder may apply a third constraint that the split inference boundary is aligned with the CTB boundary when the subpicture is not independently decodable. Typically, when subpic_treated_as_pic_flag[i] is equal to 0 or loop_filter_across_subpic_enabled_flag[i] is equal to 1 for the i-th subpicture. This constraint ensures that the split inference mechanism is only used in the context of subpicture merging. Next, an embodiment that follows these constraints is described.

[0126] In another embodiment, the syntax of the SPS is modified to specify the location of the vertical split inference boundary and the location of the horizontal split inference boundary across the entire picture. In this embodiment, picture level signaling is used since the sub-picture boundaries subject to split inference pertain only to picture-wide boundaries.

[0127] There are several alternatives for specifying the location of these boundaries: The location of the split inference boundaries may be signaled in a parameter set NAL unit such as the SPS or PPS. The parameter set must describe information related to subpicture and picture information.

[0128] In one alternative, the position of the inference boundary is defined relative to the origin of the picture, for example in luma sample units.

[0129] For example, the PPS syntax includes the following elements: [Table 6] Where: pps_subpic_split_inference_flag is a flag indicating the use of subpicture split inference. Typically, split inference is applied to the picture-wide boundary. pps_subpic_split_inference_flag equal to 1 indicates the presence of pps_split_inference_boundary_pos_x and pps_split_inference_boundary_pos_y, and pps_subpic_split_inference_flag equal to 0 indicates the absence of pps_split_inference_boundary_pos_x and pps_split_inference_boundary_pos_y.

[0130] pps_split_inference_boundary_pos_x is used to calculate the value of PpsSplitInferenceBoundaryPosX, which specifies the position of the vertical split inference boundary in luma samples. pps_split_inference_boundary_pos_x is in the range from 1 to Ceil(pic_width_in_luma_samples÷4)-1. This coding syntax element is coded, for example, using a fixed length code equal to 13 bits or log2(pic_width_in_luma_samples / 4). It is also possible to use an Exp-Golomb code.

[0131] The position of the vertical split inference boundary PpsSplitInferenceBoundaryPosX is derived as follows. PpsSplitInferenceBoundaryPosX=pps_split_inference_boundary_pos_x * 4 pps_split_inference_boundary_pos_y is used to calculate the value of PpsSplitInferenceBoundaryPosY, which specifies the position of the horizontal split inference boundary in luma samples. pps_split_inference_boundary_pos_y is in the range from 1 to Ceil(pic_height_in_luma_samples÷4)-1. This coding syntax element is, for example, coded using a fixed length code equal to 13 bits or log2(pic_width_in_luma_samples / 4). It is also possible to use an Exp-Golomb code.

[0132] The position of the horizontal split inference boundary PpsSplitInferenceBoundaryPosY is derived as follows. PpsSplitInferenceBoundaryPosY=pps_split_inference_boundary_pos_y * 4; The coding tree split inference process compares the coding unit boundary with the location of the split inference boundaries. For example, a coding unit split is inferred if the right or bottom boundary of the coding block is greater than one of the split inference boundaries that cross the current subpicture. As a result, the split inference in split_cu_flag is, for example, as follows: If split_cu_flag is not present, the value of split_cu_flag is inferred as follows: - The value of split_cu_flag is inferred to be equal to 1 if one or more of the following conditions are true: - x0+cbWidth is PpsSplitInferenceBoundaryPosX and (SubPicLeft[SubPicIdx])*(subpic_grid_col_width_minus1 + 1)*4) <PpsSplitInferenceBoundaryPosXおよび(SubPicLeft[SubPicIdx]+ SubPicWidth[i])*(subpic_grid_col_width_minus1 + 1)* 4)> Greater than PpsSplitInferenceBoundaryPosX - y0+cbHeight is PpsSplitInferenceBoundaryPosY and (SubPicTop[SubPicIdx])*(subpic_grid_row_height_minus1 + 1)*4) <PpsSplitInferenceBoundaryPosYおよび(SubPicTop[SubPicIdx]+SubPicHeight[i])*(subpic_grid_row_height_minus1+1)*4)> Greater than PpsSplitInferenceBoundaryPosY - Otherwise, the value of split_cu_flag is inferred to be equal to 0.

[0133] Another equivalent algorithm for determining the value of split_cu_flag is as follows:

[0134] If split_cu_flag is not present, the value of split_cu_flag is inferred as follows: - The value of split_cu_flag is inferred to be equal to 1 if one or more of the following conditions are true: - SubPicInferenceSplitFlag is equal to 0 and x0+cbWidth is greater than pic_width_in_luma_samples.

[0135] - SubPicInferenceSplitFlag is equal to 0 and y0+cbHeight is greater than pic_height_in_luma_samples.

[0136] - SubPicInferenceSplitFlag is equal to 1 and x0+cbWidth is greater than SubPicInferenceBoundaryPosX.

[0137] - SubPicInferenceSplitFlag is equal to 1 and y0+cbHeight is greater than SubPicInferenceBoundaryPosY.

[0138] - Otherwise, the value of split_cu_flag is inferred to be equal to 0.

[0139] SubPicInferenceSplitFlag is equal to 1 if the subpicture containing the coding block crosses the split inference boundary. SubPicInferenceBoundaryPosX is the horizontal coordinate of the vertical split inference boundary that crosses the current subpicture. If the subpicture does not cross a vertical split inference boundary, SubPicInferenceBoundaryPosX is set to a value equal to or greater than the horizontal coordinate of the right boundary of the subpicture to avoid unnecessary split inference. Similarly, SubPicInferenceBoundaryPosY is the vertical coordinate of the horizontal split inference boundary that crosses the current subpicture, if any. Otherwise, if the subpicture does not cross a horizontal split inference boundary, SubPicInferenceBoundaryPosY is set to a value equal to or greater than the horizontal coordinate of the right boundary of the subpicture. The factor "4" is introduced because the split inference boundary is constrained to correspond to the minimum coding block boundary. Assume that the size of the minimum coding block is 4x4. If the size of the smallest coding block is different, then different factors can be used in these equations.

[0140] In another example, the syntax elements mentioned above in the PPS can be defined at the SPS level, which may be advantageous if the split inference boundary does not change at each new PPS NAL unit. The syntax of the SPS includes, for example, the following elements: [Table 7] Here, subpic_split_inference_flag, split_inference_boundary_pos_x, and split_inference_boundary_pos_y have similar semantics as pps_subpic_split_inference_flag, pps_split_inference_boundary_pos_x, and pps_split_inference_boundary_pos_y.

[0141] subpic_split_inference_flag is a flag indicating the use of subpicture split inference. When subpic_split_inference_flag is equal to 1, it indicates the presence of split_inference_boundary_pos_x and split_inference_boundary_pos_y, and when subpic_split_inference_flag is equal to 0, it indicates the absence of split_inference_boundary_pos_x and split_inference_boundary_pos_y.

[0142] split_inference_boundary_pos_x is used to calculate the value of SplitInferenceBoundaryPosX, which specifies the location of the vertical split estimation boundary in luma samples. split_inference_boundary_pos_x may be in the range of 1 to Ceil(pic_width_in_luma_samples÷4)-1.

[0143] The position of the vertical split inference boundary SplitInferenceBoundaryPosX is derived as follows. SplitInferenceBoundaryPosX=split_inference_boundary_pos_x * 4 split_inference_boundary_pos_y is used to calculate the value of SplitInferenceBoundaryPosY, which specifies the position of the horizontal split inference boundary in luma samples. split_inference_boundary_pos_y may be in the range of 1 to Ceil(pic_height_in_luma_samples÷4)-1.

[0144] The position of the horizontal split inference boundary SplitInferenceBoundaryPosY is derived as follows. SplitInferenceBoundaryPosY=split_inference_boundary_pos_y * 4 The Split_inference_boundary_pos_x and split_inference_boundary_pos_y syntax elements are coded using a fixed length code, e.g., equal to 13 bits or log2(pic_width_in_luma_samples / 4) for Split_inference_boundary_pos_x and equal to log2(pic_height_in_luma_samples / 4) for Split_inference_boundary_pos_x. It is also possible to use an Exp-Golomb code.

[0145] In another embodiment, split_inference_boundary_pos_y may range from 1 to Ceil(pic_height_in_luma_samples÷4) and split_inference_boundary_pos_x may range from 1 to Ceil(pic_width_in_luma_samples÷4). In such a case, the maximum value of the range indicates that the split inference boundary is aligned with or off of the picture boundary. As a result, when set to the maximum value, it indicates that no vertical or horizontal split boundary is used. In a variant, two separate flags are used, one for each vertical and horizontal split inference boundary. In this variant, the presence of split_inference_boundary_pos_x and split_inference_boundary_pos_y is conditional on these flag values.

[0146] In another embodiment, the PPS defines one or more horizontal and vertical split inference boundaries. The encoder specifies the number of horizontal split inference boundaries and the number of vertical split inference boundaries. These boundaries are limited to picture-wide boundaries. For example, the syntax of the PPS includes the following elements: [Table 8] pps_num_ver_split_inference_boundaries specifies the number of pps_split_inference_boundaries_pos_x[i] syntax elements present in the PPS. If pps_num_ver_split_inference_boundaries is not present, it is inferred to be equal to 0.

[0147] pps_split_inference_boundaries_pos_x[i] is used to calculate the value of PpsSplitInferenceBoundaryPosX[i], which specifies the position of the i-th vertical split inference boundary in luma samples. pps_split_inference_boundary_pos_x[i] may be in the range of 1 to Ceil(pic_width_in_luma_samples÷4)-1. This coding syntax element is, for example, coded using a fixed length code equal to 13 bits or log2(pic_width_in_luma_samples / 4). It is also possible to use an Exp-Golomb code.

[0148] The position of the i-th vertical split inference boundary PpsSplitInferenceBoundaryPosX[i] is derived as follows.

[0149] PpsSplitInferenceBoundaryPosX[i]=pps_split_inference_boundary_pos_x[i]* 4 The distance between any two vertical split inference boundaries can be greater than or equal to CtbSizeY luma samples, which is the size of the CTB in luma samples.

[0150] pps_num_hor_split_inference_boundaries specifies the number of pps_split_inference_boundaries_pos_y[i] syntax elements present in the PPS. If pps_num_hor_split_inference_boundaries is not present, it is inferred to be equal to 0.

[0151] pps_split_inference_boundaries_pos_y[i] is used to calculate the value of PpsSplitInferenceBoundaryPosY[i], which specifies the position of the i-th horizontal split inference boundary in luma samples. pps_split_inference_boundary_pos_y[i] may be in the range of 1 to Ceil(pic_height_in_luma_samples÷4)-1. This coding syntax element is, for example, coded using a fixed length code equal to 13 bits or log2(pic_height_in_luma_samples / 4). It is also possible to use an Exp-Golomb code.

[0152] The position of the i-th horizontal split inference boundary PpsSplitInferenceBoundaryPosY[i] is derived as follows.

[0153] PpsSplitInferenceBoundaryPosX[i]=pps_split_inference_boundary_pos_y[i]* 4 The distance between any two horizontal split inference boundaries may be greater than or equal to CtbSizeY luma samples, which is the size of the CTB in luma samples.

[0154] In another embodiment, the encoder specifies the picture-wide split inference boundaries / boundary positions relative to the grid formed by the CTBs of the picture. Typically, the SPS or PPS includes syntax elements indicating their positions as CTB row or CTB column indices and the number of vertical and horizontal boundaries for the split inference process. For each CTB row (or column) described in the PPS, the encoder indicates the CTB width (or height) for the inference process.

[0155] In a variation of this embodiment, the encoder and decoder may determine subpicture boundaries that are picture-wide from the subpicture definition. In particular, the encoder determines NumVerSplitInferenceBoundaries, the number of vertical picture-wide subpicture boundaries, and NumHorSplitInferenceBoundaries, the number of horizontal picture-wide subpicture boundaries. It also determines the x-axis location of the ith (raster scan order) vertical picture-wide subpicture boundary and stores it in the PictureWideSubPictureBoundaryPosX[i] variable. It also determines the y-axis location of the ith (raster scan order) horizontal picture-wide subpicture boundary and stores it in the PictureWideSubPictureBoundaryPosY[i] variable.

[0156] Next, the encoder encodes the width (or height) used by the split inference of the CTB row (or column) to the left (or top) of the vertical (or horizontal) picture-picture-wide subpicture boundary.

[0157] For example, a PPS contains the following syntax elements: [Table 9]

[0158] Where: pps_split_inference_ctb_width[i] is used to calculate the value of PpsSplitInferenceBoundaryPosX[i], which specifies the position of the i-th vertical split inference boundary in units of luma samples. pps_split_inference_ctb_width[i] is specified in units of MinCbSizeY-wide coding blocks and may be in the range 0..CtbSizeY / MinCbSizeY-1. This coding syntax element is, for example, coded using a fixed-length code equal to log2(CtbSizeY / MinCbSizeY-1). It is also possible to use an Exp-Golomb code.

[0159] The position (in luma samples) of the i-th vertical split inference boundary PpsSplitInferenceBoundaryPosX[i] is derived as follows:

[0160] PpsSplitInferenceBoundaryPosX[i]=PictureWideSubPictureBoundaryPosX[i]-CtbSizeY + pps_split_inference_ctb_width[i]*MinCbSizeY pps_split_inference_ctb_height[i] is used to calculate the value of PpsSplitInferenceBoundaryPosY[i], which specifies the position of the i-th horizontal split inference boundary in units of luma samples. pps_split_inference_ctb_height[i] is specified in units of MinCbSizeY height coding blocks and may be in the range 0..CtbSizeY / MinCbSizeY-1. This coding syntax element is, for example, coded using a fixed length code equal to log2(CtbSizeY / MinCbSizeY-1). It is also possible to use an Exp-Golomb code.

[0161] The position (in luma samples) of the i-th horizontal split inference boundary PpsSplitInferenceBoundaryPosY[i] is derived as follows:

[0162] PpsSplitInferenceBoundaryPosY[i]=PictureWideSubPictureBoundaryPosY[i]- CtbSizeY + pps_split_inference_ctb_height[i]*MinCbSizeY In another embodiment, the split inferred boundary is inferred from the subpicture division (e.g., from the subpicture grid). In particular, the encoder can describe subpicture boundaries that are not aligned with the CTB boundaries. For example, the subpicture division defines a subpicture grid element size that is smaller than the CTB size. If two consecutive grid elements have two different subpicture indices and belong to the same CTB, this indicates that the first subpicture has a right (or bottom) boundary that is not aligned with the CTB boundary. The second subpicture has a left (or top) boundary that is not aligned with the CTB boundary. In such a case, the inferred split boundary is inferred to align with the right (or bottom) boundary of the first subpicture, and therefore with the left (or top) boundary of the second subpicture. In a variant, a different signaling is used to indicate the right and left boundaries of the subpictures, for example, by explicitly indicating the width and height of each subpicture in units smaller than the CTB size unit. In such cases, the width and height of the subpicture are compared with their values ​​in CTB units to determine the subpictures that are not aligned with the CTB boundaries. In another variation, a specific value of the subpicture element is reserved to indicate that this subpicture grid element has no coded data. Inferring the split inference boundary from the subpicture split avoids explicit signaling of the split inference boundary in the SPS, PPS, or any parameter set or SEI.

[0163] The VVC specification defines conformance windows within a PPS, as shown in the table below. [Table 10] This conformance window is a rectangular region within each picture represented by left, right, top and bottom offsets relative to the picture boundaries. At the end of the decoding process, the decoder applies a cropping process to remove pixels outside the conformance window.

[0164] In the previous embodiment, we described that the encoder skips some coding blocks in the CTB that are crossed by the split inference boundary. The skipped coding blocks are located to the right of the vertical split inference boundary and below the horizontal split inference boundary. In one embodiment, the decoder decodes the CTB with undefined pixels for these coding blocks. For this reason, it is proposed to add syntax elements to the PPS that define a conformance window for each subpicture.

[0165] In one embodiment, the parameter set includes a new syntax element to indicate a rectangular region within the luma samples in each conforming sub-picture: it is all pixels decoded by the decoder whose values ​​are equal to those coded by the encoder, i.e., the values ​​corresponding to the useful part of the sub-picture.

[0166] Typically, the SPS or PPS defines four adaptive window offset parameters for each subpicture of the stream that has the subpic_treated_as_pic_flag flag equal to true. For example, the syntax is: [Table 11] The syntax elements subpic_conf_win_left_offset[i], subpic_conf_win_right_offset[i], subpic_conf_win_top_offset[i], subpic_conf_win_bottom_offset[i], and subpic_conf_win_left_offset[i] specify four adaptation window offset parameters.

[0167] In another embodiment, for example, if the encoder constrains the split inference boundary to span across the picture width and height, the undefined pixels form one or more bands of pixels within the picture. As a result, instead of specifying conformance regions within each subpicture, the PPS describes bands of pixels that should be excluded from the conformance window and cropped.

[0168] For example, the PPS defines the number of bands of pixels to exclude from the conformance window defined for the pictures in the PPS. For each band of pixels, the encoder specifies the width of the band. For example, the syntax of the PPS may include an element: [Table 12] With the following semantics: conformance_exclusion_band_flag equal to 1 indicates the presence of a conformance exclusion band in the PPS. conformance_exclusion_band_flag equal to 0 indicates the absence of a conformance exclusion band in the PPS.

[0169] num_ver_conformance_exclusion_bands specifies the number of conformance_exclusion_ver_band_pos_x[i] and conformance_exclusion_ver_band_width_minus1[i] syntax elements present in the PPS. If num_ver_conformance_exclusion_bands is not present, it is inferred to be equal to 0.

[0170] conformance_exclusion_ver_band_width_minus1[i] plus 1 specifies the width of the ith vertical conformance exclusion band in luma samples and is used to calculate PpsConformanceVerBandPosX[i], which specifies the position of the ith vertical conformance exclusion band boundary in luma samples. conformance. conformance_exclusion_ver_band_width_minus1[i] is in the range 0..CtbSizeY-2. This coding syntax element is coded, for example, using a fixed length code of 8 bits or equal to log2(pic_width_in_luma_samples / CtbSizeY) or equal to log2(CtbSizeY-2). It is also possible to use an Exp-Golomb code.

[0171] conformance_exclusion_ver_band_pos_x[i] is used to calculate the value of PpsConformanceVerBandPosX[i], which specifies the position of the i-th vertical conformance exclusion band boundary in luma samples. conformance_exclusion_ver_band_pos_x[i] is in the range 0..PicWidthInCtbsY. In a variant, the range is 1..PicWidthInCtbsY-1, in order not to specify a vertical band starting at the first or last pixel column of the picture. This coding syntax element is for example coded using a fixed length code equal to 8 bits or log2(pic_width_in_luma_samples / CtbSizeY). It is also possible to use an Exp-Golomb code.

[0172] The position of the i-th vertical exclusion band PpsConformanceVerBandPosX[i] is derived as follows:

[0173] PpsConformanceVerBandPosX[i]=conformance_exclusion_ver_band_pos_x[i]* CtbSizeY - (conformance_exclusion_ver_band_width_minus1[i]+ 1) The distance between any two vertical fit exclusion band boundaries may be greater than or equal to CtbSizeY luma samples, which is the size of the CTB in luma samples. In a variant, there is no limit to the distance between two vertical fit exclusion band boundaries, and each band may overlap another band. In such a case, the decoder must determine the overlapping bands to determine the actual fit area of ​​the picture.

[0174] num_hor_conformance_exclusion_bands specifies the number of conformance_exclusion_hor_band_pos_y[i] and conformance_exclusion_hor_band_height_minus1[i] syntax elements present in the PPS. If num_hor_conformance_exclusion_bands is not present, it is inferred to be equal to 0.

[0175] conformance_exclusion_hor_band_height_minus1[i] plus 1 specifies the height of the ith horizontal conformance exclusion band in luma samples and is used to calculate PpsConformanceHorBandPosY[i], which specifies the position of the ith horizontal conformance exclusion band boundary in luma samples. Conformance. conformance_exclusion_hor_band_height_minus1[i] is in the range 0..CtbSizeY-2. This coding syntax element is, for example, coded using a fixed length code equal to 7 bits or log2(pic_height_in_luma_samples / CtbSizeY). It is also possible to use an Exp-Golomb code.

[0176] conformance_exclusion_hor_band_pos_y[i] is used to calculate the value of PpsConformanceHorBandPosY[i], which specifies the position of the ith horizontal conformance exclusion band boundary in luma samples. conformance_exclusion_hor_band_pos_y[i] is in the range 0..PicHeightInCtbsY, where PicHeightInCtbsY is the height of the picture in CTBs. In a variant, the range is 1..PicHeightInCtbsY-1, in order not to specify horizontal bands starting at the first or last pixel row of the picture. This coding syntax element is for example coded using a fixed length code equal to 8 bits or log2(pic_height_in_luma_samples / CtbSizeY). It is also possible to use an Exp-Golomb code.

[0177] The position of the i-th horizontal exclusion band PpsConformanceHorBandPosY[i] is derived as follows:

[0178] PpsConformanceHorBandPosY [i]=conformance_exclusion_hor_band_pos_y[i]* CtbSizeY -(conformance_exclusion_hor_band_height_minus1[i]+1) The distance between any two horizontal fit exclusion band boundaries may be greater than or equal to CtbSizeY luma samples, which is the size of the CTB in luma samples. In a variant, there is no limit to the distance between two horizontal fit exclusion band boundaries, and each band may overlap another band. In such cases, the decoder must determine the overlapping bands to determine the actual fit area of ​​the picture.

[0179] In a variant, the number of horizontal and vertical exclusion bands is arbitrary. In this case, if the num_ver_conformance_exclusion_bands and num_hor_conformance_exclusion_bands syntax elements are not present, their value is inferred to be equal to 1. The syntax of a PPS may, for example, be as follows: [Table 13] In a variant, the description of horizontal and / or vertical split inference boundary information is optional. For example, the syntax of a PPS (or SPS) is as follows: [Table 14] The semantics of the new syntax (in bold) elements are as follows:

[0180] conformance_exclusion_ver_band_flag equal to 1 indicates the presence of conformance_exclusion_ver_band_width_minus1 and conformance_exclusion_ver_band_pos_x in the PPS that specify a vertical conformance exclusion band. conformance_exclusion_band_flag equal to 0 indicates the absence of conformance_exclusion_ver_band_width_minus1 and conformance_exclusion_ver_band_pos_x in the PPS.

[0181] conformance_exclusion_ver_band_width_minus1 plus 1 specifies the width of the vertical conformance exclusion band in luma samples and is used to calculate PpsConformanceVerBandPosX, which specifies the position of the left boundary of the vertical conformance exclusion band. conformance_exclusion_ver_band_width_minus1 may be in the range 0..CtbSizeY-2.

[0182] conformance_exclusion_ver_band_pos_x is used to calculate the value of PpsConformanceVerBandPosX. conformance_exclusion_ver_band_pos_x may be in the range 1..PicWidthInCtbsY-1.

[0183] The position of the left boundary of the vertical exclusion band PpsConformanceVerBandPosX is derived as follows.

[0184] PpsConformanceVerBandPosX=conformance_exclusion_ver_band_pos_x* CtbSizeY-(conformance_exclusion_ver_band_width_minus1 + 1)

[0185] conformance_exclusion_hor_band_flag equal to 1 indicates the presence of conformance_exclusion_hor_band_height_minus1 and conformance_exclusion_hor_band_pos_y in the PPS that specify a horizontal conformance exclusion band. conformance_exclusion_band_flag equal to 0 indicates the absence of conformance_exclusion_hor_band_height_minus1 and conformance_exclusion_hor_band_pos_y in the PPS.

[0186] conformance_exclusion_hor_band_height_minus1 plus 1 specifies the height of the horizontal conformance exclusion band in luma samples and is used to calculate PpsConformanceHorBandPosY. PpsConformanceHorBandPosY specifies the position of the upper boundary of the horizontal conformance exclusion band. conformance_exclusion_hor_band_height_minus1 may be in the range 0..CtbSizeY-2.

[0187] conformance_exclusion_hor_band_pos_y is used to calculate the value of PpsConformanceHorBandPosY. conformance_exclusion_hor_band_pos_y may be in the range 1..PicHeightInCtbsY-1.

[0188] The position of the upper boundary of the horizontal exclusion band PpsConformanceHorBandPosY is derived as follows.

[0189] PpsConformanceHorBandPosY=conformance_exclusion_hor_band_pos_y * CtbSizeY-(conformance_exclusion_hor_band_height_minus1 + 1)

[0190] In a variant, the width, height and coordinates of the exclusion band are expressed in chroma samples. Their values ​​in luma samples are obtained by multiplying the values ​​expressed in chroma samples by SubWidthC and SubHeightC, which represent the horizontal and vertical sampling ratios between the luma and chroma elements, respectively. For example, if the chroma format is 4:2:0, SubWidthC and SubHeightC are equal to 2. In another variant, the width, height and coordinates of the exclusion band are expressed in minimum CTB size units to further reduce the length of the syntax element.

[0191] In another embodiment, the width and height of the exclusion band are larger than the CTB. Two consecutive exclusion bands of pixels can be merged into one exclusion band, for example when two adjacent subpictures define two adjacent sets of mismatched pixels. This allows for a more compact description of the exclusion band.

[0192] In another embodiment, the number of vertical fitted exclusion bands and the number of horizontal fitted exclusion bands are inferred to be equal to the number of vertical split inference boundaries and the number of horizontal split inference boundaries, respectively.

[0193] In another embodiment, the positions of the vertical and horizontal fitted exclusion bands are inferred to be equal to the positions of the vertical and horizontal split inference boundaries, respectively.

[0194] In another embodiment, the width of a given vertical fitted exclusion band is assumed to be equal to the number of pixels between the vertical split inference boundary and the nearest right CTB boundary at the same location. The same applies to horizontal exclusion bands with horizontal split inference boundaries.

[0195] In another embodiment, subpicture split inference is inferred from the subpicture division: the pixels to be excluded from the matching window are inferred to correspond to the region between the right (or bottom) borders of the subpictures that are not aligned with the CTB boundaries and the right (or bottom) borders of these CTBs.

[0196] In a variant, these regions are grouped into different subpictures. In such a case, the width or height of these subpictures is less than the CTB size. It is therefore possible to associate these non-conforming subpictures with a specific index. The SPS or PPS can provide a list of subpicture indexes that are not conforming and should be excluded from the conformance window. In another embodiment, the number of bands to be excluded from the conformance window and their size are determined from the inferred split boundary positions.

[0197] In another alternative, the number of bands excluded from the fitting window and their size are determined from the inferred split boundary positions.

[0198] The adaptive cropping window includes luma samples with horizontal picture coordinates from SubWidthC * conf_win_left_offset to pic_width_in_luma_samples-(SubWidthC * conf_win_right_offset + 1) and vertical picture coordinates from SubHeightC * conf_win_top_offset to pic_height_in_luma_samples-(SubHeightC * conf_win_bottom_offset+1).

[0199] Additionally, luma samples with horizontal picture coordinates from PpsConformanceVerBandPosX[i] to PpsConformanceVerBandPosX[i]+conformance_exclusion_ver_band_width_minus1[i]+1 are excluded from the conformance window for i in the range 0..num_ver_conformance_exclusion_bands.

[0200] Additionally, luma samples with vertical picture coordinates from PpsConformanceHorBandPosY[i] to PpsConformanceHorBandPosY[i]+conformance_exclusion_hor_band_height_minus1[i]+1 are excluded from the conformance window for i in the range 0..num_hor_conformance_exclusion_bands.

[0201] The width and height of the cropped picture correspond to the variables PicOutputWidthL and PicOutputHeightL, which are derived as follows: The width (or height) of the picture is equal to the width (or height) of the cropping window minus the width (or height) of the exclusion band of pixels, which corresponds to the following pseudo code: PicOutputWidthL = pic_width_in_luma_samples -SubWidthC*(conf_win_right_offset + conf_win_left_offset) for(i=0;i <num_ver_conformance_exclusion_bands; i++) PicOutputWidthL -= conformance_exclusion_ver_band_width_minus1[i]+1 PicOutputHeightL = pic_height_in_pic_size_units - SubHeightC*(conf_win_bottom_offset + conf_win_top_offset) for(i=0;i <num_hor_conformance_exclusion_bands; i++) PicOutputHeightL -= conformance_exclusion_hor_band_height_minus1[i]+1 In one embodiment, a picture and its sub-pictures that can be displaced in a merge operation are constrained to have a size that is a multiple of the CTB size in both directions. Typically, this can be done by extending the input picture with padding pixels if it does not already take this constraint into account. Thus, there is no inferred boundary mechanism to be used. This constraint is signaled in the bitstream. For example, a flag can be used to indicate that a sub-picture is subject to this constraint and can therefore be freely merged and displaced to any position in the resulting picture without boundary issues. To ensure that sub-pictures can be freely merged, an additional constraint is associated with the flag, which is that the sub-pictures are independently decodable. The flag may be defined as applying to all sub-pictures in a picture, or may be defined at the sub-picture level to indicate that the associated sub-pictures can be freely merged. In one embodiment, a flag indicating that the size of a picture and its sub-pictures is a multiple of the CTB size is defined in the sequence parameter set and is valid for all sub-pictures of the sequence.

[0202] In some embodiments, the adaptation window may be defined at the sub-picture level, for example in the SEI message.

[0203] In one embodiment, the syntax of the SPS is modified to include one additional flag that indicates whether the encoding of the sub-pictures in the bitstream constrains the encoding tool and video sequence characteristics to allow merging operations. When set, this flag restricts whether the sub-pictures are independently decodable or not. The syntax is as follows: [Table 15] For example, the semantics of the element are:

[0204] subpic_mergeable_flag equal to 1 indicates that all subpictures of the picture are constrained for merge operations. subpic_mergeable_flag equal to 0 indicates whether the subpictures are constrained.

[0205] pic_width_max_in_luma_samples specifies the maximum width, in luma samples, of each decoded picture that references the SPS. pic_width_max_in_luma_samples is not equal to 0 and is an integer multiple of Max(8, MinCbSizeY). If subpic_mergeable_flag is equal to 1, pic_width_max_in_luma_samples is an integer multiple of CtbSizeY. As a result, if a subpicture is constrained for a merge operation (subpic_mergeable_flag is equal to 1), the coded picture is constrained to use padding when the original width of the picture is not a multiple of the CTB size.

[0206] pic_height_max_in_luma_samples specifies the maximum height, in luma samples, of each decoded picture that references an SPS. pic_height_max_in_luma_samples shall not be equal to 0 and shall be an integer multiple of Max(8, MinCbSizeY). subpic_mergeable_flag If mergeable_flag is equal to 1, pic_height_max_in_luma_samples shall be an integer multiple of CtbSizeY. As a result, if a subpicture is constrained for a merge operation (subpic_mergeable_flag is equal to 1), the coded picture is constrained to use padding if the original height of the picture is not a multiple of the CTB size.

[0207] In this embodiment, the semantics of some elements of the PPS may be constrained to ensure that each picture that references the PPS has a picture size that is a multiple of the CTB size when subpictures are constrained for merging operations. For example, the semantics of pic_width_in_luma_samples and pic_height_in_luma_samples are as follows:

[0208] pic_width_in_luma_samples specifies the width, in luma samples, of each decoded picture that references a PPS. pic_width_in_luma_samples is not equal to 0, is an integer multiple of Max(8, MinCbSizeY), and is less than or equal to pic_width_max_in_luma_samples. If subpic_mergeable_flag is equal to 1 (e.g., in an SPS with an identifier signaled in the PPS), pic_width_in_luma_samples MUST be an integer multiple of CtbSizeY.

[0209] pic_height_in_luma_samples specifies the height, in luma samples, of each decoded picture that references the PPS. pic_height_in_luma_samples is not equal to 0 and is an integer multiple of Max(8, MinCbSizeY) and is less than or equal to pic_height_max_in_luma_samples. If subpic_mergeable_flag is equal to 1, pic_height_in_luma_samples is an integer multiple of CtbSizeY.

[0210] This first set of constraints on a picture ensures that any merge position is possible for a subpicture of a picture in the merged stream if subpic_mergeable_flag is equal to 1. The encoder may have to use padding data to make both the picture width and height a multiple of the CTB size. The encoder can signal the area with the padding data by defining a conformance window in the bitstream. Typically, one conformance window can be defined for each subpicture in the SEI message.

[0211] In another embodiment, the mergeable flag also constrains the temporal prediction mechanism within each subpicture for all subpictures described in the parameter set NAL unit if the subpicture is constrained for merging operations (subpic_mergeable_flag is equal to 1).

[0212] For example, subpic_treated_as_pic_flag[i] equal to 1 specifies that the i-th subpicture of each coded picture in the CLVS is treated as a picture in the decoding process excluding in-loop filtering operations. subpic_treated_as_pic_flag[i] equal to 0 specifies that the i-th subpicture of each coded picture in the CLVS is not treated as a picture in the decoding process excluding in-loop filtering operations. If not present, the value of subpic_treated_as_pic_flag[i] is inferred to be equal to subpic_mergeable_flag. As a result, if a subpicture is constrained for merging operations (subpic_mergeable_flag equal to 1), the subpic_treated_as_pic_flag[i] syntax element is not present in the parameter set NAL unit and is inferred to be equal to 1 indicating that the temporal prediction of the subpicture is constrained to each subpicture boundary. Otherwise, the temporal prediction may be either constrained or unconstrained.

[0213] In another embodiment, if a subpicture is constrained for merging operations (subpic_mergeable_flag equal to 1), the loop filter mechanism is disabled across subpicture boundaries for all subpictures described in the parameter set NAL unit. For example, loop_filter_across_subpic_enabled_flag[i] equal to 1 specifies that in-loop filtering operations are performed across the boundaries of the i-th subpicture in each coded picture in the CLVS. loop_filter_across_subpic_enabled_flag[i] equal to 0 specifies that in-loop filtering operations are not performed across the boundaries of the i-th subpicture in each coded picture in the CLVS. If not present, the value of loop_filter_across_subpic_enabled_pic_flag[i] is inferred to be equal to !subpic_mergeable_flag. As a result, if a subpicture is constrained for merging operations (subpic_mergeable_flag is equal to 1), then the loop_filter_across_subpic_enabled_pic_flag[i] syntax element is inferred to be absent in the parameter set NAL unit and equal to 0 indicating that the loop filter is constrained to not be valid across subpicture boundaries. Otherwise, the loop filter may or may not be constrained.

[0214] In some embodiments, spatial access may be provided only in some time intervals. The proposed flags may be defined correspondingly in the PPS or even in the picture header, which is a non-VCL NAL unit that is defined at the picture level and defines some syntax elements that apply to all slices of the picture.

[0215] In another embodiment, sub-picture signaling associates an identifier of a sub-picture with a sub-picture index. This identifier is unique for a given sub-picture, simplifying the merge operation since it allows to avoid rewriting the sub-picture index in the slice header when the sub-picture is moved to a new position after the merge operation. Thus, one of the non-VCL NAL units can signal the sub-picture identifier in the bitstream when the sub-picture is constrained for a merge operation.

[0216] Typically, a subpicture identifier can be present in the SPS, PPS, or picture header. In particular, sps_subpic_id_present_flag is a flag in the SPS that indicates the presence of a subpicture identifier in the bitstream when equal to 1. The semantics of this syntax element are, for example, as follows:

[0217] sps_subpic_id_present_flag equal to 1 specifies that a subpicture ID mapping is present in the SPS. sps_subpic_id_present_flag equal to 0 specifies that a subpicture ID mapping is not present in the SPS. If subpic_mergeable_flag is equal to 1, then sps_subpic_id_present_flag shall be equal to 1. As a result, if a subpicture is constrained for merging operations, signaling of the subpicture identifier is provided in the bitstream. In a variant, if subpic_mergeable_flag is equal to 1, then sps_subpic_id_present_flag is not present in the SPS and is inferred to be 1. The corresponding syntax of the SPS may include the following syntax elements: [Table 16]

[0218] In another embodiment, the presence of a sub-picture identifier in the picture header may make the merging operation more complicated. In fact, if the identifier signaling is done in the picture header, it overrides the mapping of the sub-picture identifier done in the parameter set NAL unit. Since a different picture header is transmitted for each frame, the mapping may change for each picture. As a result, a typical merging operation needs to check whether the picture header changes the sub-picture identifier mapping. To avoid this checking operation, if a sub-picture is constrained for the merging operation, the mapping of the identifier in the picture header is disabled. The presence of the sub-picture identifier mapping in the picture header is controlled with the ph_subpic_id_signalling_present_flag syntax element. In this embodiment, the semantics of ph_subpic_id_signalling_present_flag are as follows:

[0219] ph_subpic_id_signalling_present_flag equal to 1 specifies that subpicture ID mapping is signaled to the PH. ph_subpic_id_signalling_present_flag equal to 0 specifies that subpicture ID mapping is not signaled to the PH. If subpic_mergeable_flag is equal to 1, then ph_subpic_id_signalling_present_flag shall be equal to 0. In a variant, if subpic_mergeable_flag is equal to 1, then ph_subpic_id_signalling_present_flag is not present in the picture header and is inferred to be equal to 0.

[0220] A PPS may also signal the mapping of subpicture identifiers. For picture headers, it is necessary to check if the mapping has changed between two PPS NAL units. This additional check increases the complexity of the merge operation. As a result, in one embodiment, if a subpicture is constrained for a merge operation, it is constrained to be identical in each PPS and in each PPS for which subpic_mergeable_flag is equal to 1. The pps_subpic_id[i] syntax element of a PPS specifies the subpicture ID (or identifier) ​​of the i-th subpicture that corresponds to the mapping of subpicture identifiers with the subpicture index. If subpic_mergeable_flag is equal to 1, all PPSs that refer to the same SPS have the same value of pps_subpic_id[i] for i ranging from 0 to pps_num_subpics_minus1.

[0221] In another embodiment, the information indicating that a sub-picture is constrained for a merging operation corresponds to a specific profile of the VVC specification. Typically, a specific value (e.g., 3) of the general_profile_idc syntax element indicates the profile to which the output layer conforms for a sub-picture merging operation. In a variant, the information indicating that a sub-picture is constrained for a merging operation corresponds to a sub-profile. In that case, a specific value (e.g., 3) of general_sub_profile_idc[i] indicates that the bitstream is constrained for a sub-picture merging operation.

[0222] The VVC general constrained information structure is configured by flags described in the profile, layer and level information that allow disabling one or more coding tools. In one embodiment, the general constrained information structure includes any of the following syntax elements:

[0223] - a no_dependent_subpicture_flag syntax element which, when equal to 1, specifies that subpic_treated_as_pic_flag[i] is equal to 1 for any value of i. no_dependent_subpicture_flag equal to 0 imposes no such constraint. In a variant, when equal to 1, it specifies that subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] are equal to 1 for any value of i.

[0224] - a no_picture_header_subpicture_id_mapping syntax element that, when equal to 1, specifies that all picture header NAL units have ph_subpic_id_signalling_present_flag equal to 0. no_picture_header_subpicture_id_mapping equal to 0 imposes no such constraint.

[0225] - a no_pps_subpicture_id_mapping_change syntax element that, when equal to 1, specifies that all PPS NAL units have equal values ​​of pps_subpic_id[i] for any value of i. no_pps_subpicture_id_mapping_change equal to 0 imposes no such constraint.

[0226] In one embodiment, an SEI message is proposed to handle intra-picture regions that require a post-decoding crop operation. This may be the result of a bitstream extraction and merging operation where, for example, some sub-pictures at the right or bottom border of the picture originally contained padding data. This SEI indicates to the decoder a conformance window for at least some sub-pictures that defines picture regions containing padding or unreliable or even meaningless data that the content creator believes should be removed on the image to be displayed.

[0227] For example, the SEI might provide the following syntax: [Table 17]

[0228] With the following semantics: subpic_conf_win_cancel_flag equal to 1 indicates that the SEI message cancels the persistence of any previous subpicture adaptation window SEI messages in the output order applied to the current layer. subpic_conf_win_cancel_flag equal to 0 indicates that subpicture adaptation window information follows.

[0229] subpic_conf_win_num_subpics_minus1 plus 1 specifies the number of subpicture conformance windows present in the SEI message. This value is a function of the number of subpictures present in the picture. Typically, it is required that the value of subpic_conf_win_num_subpics_minus1 is equal to sps_num_subpics_minus1, allowing the definition of a conformance window for each subpicture.

[0230] subpic_conf_win_left_offset[i], subpic_conf_win_right_offset[i], subpic_conf_win_top_offset[i], and subpic_conf_win_bottom_offset[i] specify the samples of the i-th subpicture of the picture in the CLVS that are output from the decoding process with respect to a rectangular region specified in picture coordinates for output relative to the origin of the i-th subpicture, as described in the SPS NAL unit.

[0231] The subpicture adapted cropping window for the i-th subpicture includes luma samples with horizontal picture coordinates from SubPictureLuma_X[i] + SubWidthC * conf_win_left_offset to SubPictureLuma_X[i] + SubPictureLuma_Width[i] - (SubWidthC * subpic_conf_win_right_offset[i] + 1) and vertical picture coordinates from SubPictureLuma_Y[i] + SubHeightC * subpic_conf_win_top_offset[i] to SubPictureLuma_Y[i] + SubPictureLuma_Height[i] - (SubHeightC * subpic_conf_win_bottom_offset[i] + 1). SubPictureLuma_X[i] and SubPictureLuma_Y[i] specify the horizontal and vertical picture coordinates of the first pixel of the ith subpicture, and SubPictureLuma_Width[i] and SubPictureLuma_Height[i] are the width and height of the luma sample of the ith subpicture as described in the SPS. For example, these variables are calculated as follows: SubPictureLuma_X[i]=subpic_ctu_top_left_x[i]* CtbSizeY SubPictureLuma_Y[i]=subpic_ctu_top_left_y[i]* CtbSizeY SubPictureLuma_Width[i]=(subpic_width_minus1[i]+1)* CtbSizeY SubPictureLuma_Height[i]=(subpic_height_minus1[i]+1)* CtbSizeY In some cases, only a subset of subpictures (usually subpictures containing padding data) require a conformance window. In such cases, a new syntax element specifies for each subpicture if a conformance window is signaled. For example, a "for" loop for each subpicture specifies the subpic_conf_win_signalled_flag[i] syntax element for the i-th subpicture described in the subpicture. If equal to 1, the offset parameter is present and a conformance window is specified for the subpicture. Otherwise, subpic_conf_win_signalled_flag[i] is equal to 0, no conformance is signaled for the i-th subpicture, and the offset parameter is not present and is inferred to be equal to 0. In a variant, the number of subpicture conformance windows described in the SEI message is different from the number of subpictures in the picture, and for each signaled subpicture conformance window signaled in the SEI, a list of one or more subpicture indexes in the picture is associated with the index of the subpicture conformance window. This list of indexes indicates the subpictures that use the subpicture adaptation window. For example, a for loop in the SEI message indicates a subpic_conf_win_num_subpics_minus1[i] syntax element, which is the number of subpicture indices associated with the i-th subpicture adaptation window of the SEI message minus 1. Then, a processing loop for j ranging from 0 to subpic_conf_win_num_subpics_minus1[i] defines a subpic_conf_win_subpic_index[i][j] syntax element, where subpic_conf_win_subpic_index[i][j] specifies the j-th index of the subpicture that uses the i-th subpicture adaptation window. In a variant, the SEI message can define a subpicture identifier instead of a subpicture index. In that case, the length in bits of the subpicture identifier is optionally described in the SEI message.

[0232] 12 is a schematic block diagram of a computing device 120 for implementing one or more embodiments of the present invention. The computing device 120 may be a device such as a microcomputer, a workstation, or a light portable device. The computing device 120 may include: a central processing unit 121 such as a microprocessor, denoted CPU; a random access memory 122, denoted RAM, for storing the executable code of the method of an embodiment of the invention, as well as registers adapted to record variables and parameters necessary for carrying out the method according to an embodiment of the invention, the memory capacity of which can be expanded, for example, by an optional RAM connected to an expansion port; a read-only memory 123, denoted ROM, for storing a computer program for implementing an embodiment of the present invention; Network interface 124 is typically connected to a communications network over which digital data to be processed is transmitted and received. Network interface 124 may be a single network interface or may be comprised of a set of different network interfaces (e.g., wired and wireless interfaces, or different types of wired or wireless interfaces). Data packets are read from the network interface for reception or written to the network interface for transmission under the control of software applications running within CPU 121; The user interface 125 may be used to receive input from a user or to display information to a user; A hard disk 126, designated HD, may be provided as mass storage device; The I / O module 127 may be used to receive / send data from / to external devices such as a video source or a display. A communication bus is connected to the

[0233] The executable code may be stored either in the read-only memory 123, in the hard disk 126 or on a removable digital medium, such as a disk. According to a variant, the executable code of the program may be received by the communications network, via the network interface 124, in order to be stored in one of the storage means of the communications device 120, such as the hard disk 126, before being executed.

[0234] The central processing unit 121 is adapted to control and direct the execution of instructions or parts of the software code of a program or programs according to an embodiment of the invention stored in one of the aforementioned storage means. After power-on, the CPU 121 is capable of executing instructions from the main RAM memory 122 relating to a software application, for example after those instructions have been loaded from a program ROM 123 or a hard disk (HD) 126. Such a software application, when executed by the CPU 121, causes the execution of the steps of the flowcharts of the invention.

[0235] Any step of an algorithm of the present invention may be implemented in software by execution of a program or set of instructions by a programmable computing machine such as a PC ("Personal Computer"), a DSP ("Digital Signal Processor") or a microcontroller, or in hardware by a machine or dedicated component such as an FPGA ("Field Programmable Gate Array") or an ASIC ("Application Specific Integrated Circuit").

[0236] Although the invention has been described herein with reference to specific embodiments, the invention is not limited to those embodiments, and modifications that are within the scope of the invention will be apparent to those skilled in the art.

[0237] Many further modifications and variations will suggest themselves to those skilled in the art by reference to the foregoing exemplary embodiments, which are given by way of example only and are not intended to limit the scope of the invention, which is determined solely by the appended claims. In particular, different features from different embodiments may be interchanged where appropriate.

[0238] Each of the above-described embodiments of the present invention may be implemented alone or in combination in several embodiments, and features from the various embodiments may be combined where necessary or where a combination of elements or features from the individual embodiments in a single embodiment is beneficial.

[0239] Each feature disclosed in this specification (including any accompanying claims, abstract, and drawings), unless stated otherwise, may be replaced by alternative features serving the same, equivalent, or similar purpose. Thus, unless stated otherwise, each feature disclosed is only an example of a generic series of equivalent or similar features.

[0240] In the claims, the word "comprise" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

1. 1. A method of encoding video data including a picture into a bitstream, the picture being divided into sub-pictures including one or more slices, the method comprising: encoding a first flag in a sequence parameter set of the bitstream indicating whether sub-picture information is present; encoding a second flag relating to a sub-picture constraint into the sequence parameter set of the bitstream when the value of the first flag is 1; encoding, when the value of the second flag is 0, a third flag indicating whether in-loop filtering is enabled at a sub-picture boundary for a sub-picture within a picture, and a fourth flag indicating whether the sub-picture is treated as a picture in a decoding process excluding in-loop filtering, into the sequence parameter set of the bitstream; encoding coding tree blocks constituting a picture into the bitstream; having The method of claim 1, wherein when the value of the second flag is 1, the third flag and the fourth flag are not coded in the sequence parameter set of the bitstream.

2. 2. The method of claim 1, wherein when the value of the second flag is one, the third flag is inferred as zero.

3. 3. The method of claim 1, wherein the second flags relate to a number of sub-pictures contained in the picture.

4. 4. The method of claim 3, wherein the second flag is associated with all sub-pictures contained in the picture.

5. 5. A method according to claim 1, wherein when the value of the second flag is 1, the sub-picture boundary is treated as an image boundary and no in-loop filtering is performed at the sub-picture boundary.

6. 6. The method according to claim 1, wherein the first flag, the second flag and the third flag are defined in the sequence parameter set.

7. 7. A method according to any preceding claim, wherein the sub-pictures are identified by sub-picture identifiers.

8. 8. A method according to any one of claims 1 to 7, wherein the sub-picture is a rectangular region made up of one or more slices.

9. 2. The method of claim 1, wherein when the value of the second flag is one, the fourth flag is inferred as one.

10. 10. A method according to any preceding claim, comprising the step of encoding information indicative of a conformance window for a sub-picture.

11. 1. A method of decoding a bitstream of video data including pictures, the pictures being divided into sub-pictures including one or more slices, the method comprising: decoding a first flag indicating whether sub-picture information is present from a sequence parameter set of the bitstream; decoding a second flag relating to a sub-picture constraint from the sequence parameter set of the bitstream when the value of the first flag is 1; decoding, from the sequence parameter set of the bitstream, a third flag indicating whether in-loop filtering is enabled at a sub-picture boundary for a sub-picture within a picture when the value of the second flag is 0, and a fourth flag indicating whether the sub-picture is treated as a picture in a decoding process excluding in-loop filtering; and decoding the bitstream based on at least the first flag; The method of claim 1, wherein when the second flag has a value of 1, the third flag and the fourth flag are not decoded from the sequence parameter set of the bitstream.

12. 12. The method of claim 11, wherein when the value of the second flag is one, the third flag is inferred as zero.

13. 13. A method according to claim 11 or 12, wherein the second flags relate to a number of sub-pictures contained in the picture.

14. 14. The method of claim 13, wherein the second flag is associated with all sub-pictures contained in the picture.

15. 15. The method of any one of claims 11 to 14, wherein when the value of the second flag is 1, the sub-picture boundary is treated as an image boundary and no in-loop filtering is performed at the sub-picture boundary.

16. 16. The method of claim 11, wherein the first flag, the second flag, and the third flag are defined in the sequence parameter set.

17. 17. A method according to any one of claims 11 to 16, wherein the sub-pictures are identified by sub-picture identifiers.

18. 18. A method according to any one of claims 11 to 17, wherein the sub-picture is a rectangular region made up of one or more slices.

19. 12. The method of claim 11, wherein when the value of the second flag is one, the fourth flag is inferred as one.

20. 20. A method according to any one of claims 11 to 19, comprising the step of decoding information indicative of a conformance window for a sub-picture.

21. An apparatus for encoding video data including a picture into a bitstream, the picture being divided into sub-pictures including one or more slices, the apparatus comprising: means for encoding a first flag indicating whether sub-picture information is present in a sequence parameter set of a bitstream; means for encoding a second flag relating to a sub-picture constraint into the sequence parameter set of the bitstream when the value of the first flag is 1; means for encoding, when the value of the second flag is 0, a third flag indicating whether in-loop filtering is enabled at a sub-picture boundary for a sub-picture within a picture, and a fourth flag indicating whether the sub-picture is to be treated as a picture in a decoding process excluding in-loop filtering, into the sequence parameter set of the bit stream; means for encoding coding tree blocks constituting a picture into the bitstream; When the value of the second flag is 1, the third flag and the fourth flag are not coded in the sequence parameter set of the bitstream.

22. 1. An apparatus for decoding a bitstream of video data including pictures, the pictures being divided into sub-pictures including one or more slices, the apparatus comprising: means for decoding a first flag indicating whether sub-picture information is present from a sequence parameter set of a bitstream; means for decoding a second flag relating to a sub-picture constraint from the sequence parameter set of the bitstream when the value of the first flag is 1; means for decoding, from the sequence parameter set of the bitstream, a third flag indicating whether in-loop filtering is enabled at a sub-picture boundary for a sub-picture within a picture when the value of the second flag is 0, and a fourth flag indicating whether the sub-picture is treated as a picture in a decoding process excluding in-loop filtering; means for decoding the bitstream based on at least the first flag; The apparatus, wherein when the value of the second flag is 1, the third flag and the fourth flag are not decoded from the sequence parameter set of the bitstream.

23. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 20.

Images