Video decoding methods, video encoding methods, and bitstream transmission methods

Abstract

The present invention discloses video decoding methods, video encoding methods, and bitstream transmission methods. One embodiment of the present invention provides a method for segmenting a video frame, the method comprising the following steps: decoding from the bitstream a segmentation flag indicating whether the frame can be segmented into one or more sub-units; when the segmentation flag indicates that the frame can be segmented into sub-units, decoding segmentation information from the bitstream; and using the segmentation information to segment the frame into sub-units.

Description

[0001] This application is a divisional application of the original invention patent application No. 202080083722.1 (International Application No.: PCT / KR2020 / 013522, Application Date: October 6, 2020, Invention Title: Method and Decoding Device for Screen Segmentation). Technical Field

[0002] This disclosure relates to video encoding and decoding, and more specifically, to a method for screen partitioning and a decoding device thereof that improves encoding and decoding efficiency through screen partitioning methods suitable for various purposes. Background Technology

[0003] Because video data is larger than audio or still image data, storing or sending video data without compression requires significant hardware resources, including memory.

[0004] Therefore, when storing or sending video data, an encoder is typically used to compress the video data for storage or transmission. The decoder then receives the compressed video data and decompresses and reproduces it. Compression techniques used for this type of video include H.264 / AVC and High Efficiency Video Coding (HEVC), which offers approximately 40% better coding efficiency compared to H.264 / AVC.

[0005] However, as video size, resolution, and frame rate increase, the amount of data to be encoded also increases. Therefore, new compression techniques are needed that offer better coding efficiency and higher image quality compared to existing compression technologies. Summary of the Invention

[0006] Technical issues

[0007] The purpose of this disclosure is to provide an improved video encoding and decoding technique to meet specific needs. Specifically, one aspect of this disclosure relates to a technique for improving encoding and decoding efficiency by dividing the frame to suit various purposes.

[0008] Technical solution

[0009] According to one aspect, this disclosure provides a method for partitioning a screen. The method includes the steps of: decoding from a bitstream a partition flag indicating whether the screen is partitioned into one or more sub-units; when the partition flag indicates partitioning, decoding partition information from the bitstream; and partitioning the screen into sub-units based on the partition information.

[0010] According to another aspect, this disclosure provides a video decoding apparatus. The video decoding apparatus includes: a decoding execution unit configured to decode from a bitstream a segmentation flag indicating whether a frame is divided into one or more sub-units, and to decode partitioning information from the bitstream when the segmentation flag indicates partitioning; and a partitioner configured to divide the frame into sub-units based on the partitioning information.

[0011] Technical effect

[0012] As described above, according to one embodiment of this disclosure, the screen can be divided to suit various purposes, including sequence synthesis, bitstream packing, and parallel processing. Attached Figure Description

[0013] Figure 1 This is an exemplary block diagram of a video encoding device capable of implementing the technology disclosed herein.

[0014] Figure 2 An exemplary block partitioning structure using the QTBTTT structure is shown.

[0015] Figure 3 Several intra-frame prediction modes are illustrated as examples.

[0016] Figure 4 This is an exemplary block diagram of a video decoding device capable of implementing the technology disclosed herein.

[0017] Figures 5 to 7 Shows slices, tiles, and sub-pictures.

[0018] Figure 8 The composition of the displayed content.

[0019] Figure 9 This is a block diagram of a decoder capable of implementing the technology disclosed herein.

[0020] Figure 10 This is a flowchart illustrating an example of a method capable of implementing the techniques disclosed herein.

[0021] Figure 11 and Figure 12 This demonstrates various ways of dividing a screen into slices, blocks, and sub-screens.

[0022] Figure 13 This is a flowchart illustrating the process of dividing a screen into sub-screens.

[0023] Figure 14 This is a flowchart illustrating the process of dividing a screen into sub-screens.

[0024] Figure 15 This is a flowchart illustrating the process of dividing the screen into slices.

[0025] Figure 16This shows the partitioning format of the slice.

[0026] Figure 17 This is a flowchart illustrating the sub-screen mapping.

[0027] Figure 18 This is a flowchart showing the slice address.

[0028] Figure 19 This shows the relationship between slices and sub-pictures.

[0029] Figure 20 This shows the connections between header information.

[0030] Figure 21 Examples of various sub-screen configurations are shown. Detailed Implementation

[0031] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that when adding reference numerals to the constituent elements in the various figures, similar reference numerals refer to similar elements even though the elements are shown in different figures. Furthermore, in the following description of the present disclosure, detailed descriptions of known functions and configurations incorporated herein will be omitted to avoid obscuring the subject matter of the disclosure.

[0032] Figure 1 This is an exemplary block diagram of a video encoding device capable of implementing the technology disclosed herein. The following will refer to... Figure 1 Describe the video encoding device and its components.

[0033] The video encoding device includes a block divider 110, a predictor 120, a subtractor 130, a converter 140, a quantizer 145, an encoder 150, an inverse quantizer 160, an inverse converter 165, an adder 170, a filter unit 180, and a memory 190.

[0034] The various components of a video encoding device can be implemented in hardware, software, or a combination of both. The functions of each component can be implemented in software, and a microprocessor can be implemented to execute the software functions corresponding to each component.

[0035] A video consists of multiple frames. Each frame is divided into multiple regions, and encoding is performed on each region. For example, a frame is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more coding tree units (CTUs). Each CTU is divided into one or more coding units (CUs) in a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks in a tile is encoded as the syntax of the tile or as the syntax of a tile group as a set of multiple tiles, and information applied to all blocks constituting a frame is encoded in the frame parameter set (PPS) or frame header. Furthermore, information commonly referenced by multiple frames is encoded in the sequence parameter set (SPS). Additionally, information commonly referenced by one or more SPSs is encoded in the video parameter set (VPS).

[0036] Block splitter 110 determines the size of the coding tree unit (CTU). Information about the size of the CTU (CTU size) is encoded into SPS or PPS syntax and sent to the video decoding device.

[0037] The block segmenter 110 divides the individual frames that make up the video into multiple CTUs of a predetermined size, and then recursively segments the CTUs using a tree structure. In the tree structure, leaf nodes are used as coding units (CUs), which are the basic units of coding.

[0038] The tree structure can be: QuadTree (QT), where a node (or parent node) is divided into four child nodes (or child nodes) of the same size; BinaryTree (BT), where a node is divided into two child nodes; TernaryTree (TT), where a node is divided into three child nodes in a 1:2:1 ratio; or a structure formed by a combination of two or more of the QT, BT, and TT structures. For example, a QTBT (QuadTree + BinaryTree) structure or a QTBTTT (QuadTree + BinaryTree TernaryTree) structure can be used. Here, BTTT can be collectively referred to as Multi-Type Tree (MTT).

[0039] Figure 2 This shows the QTBTTT split tree structure. (Example:) Figure 2As shown, the CTU can initially be segmented according to the QT structure. QT segmentation can be repeated until the size of the segmented blocks reaches the minimum block size (MinQTSize) allowed for leaf nodes in the QT. A first flag (QT_split_flag) indicating whether each node in the QT structure has been segmented into the four lower-level nodes is encoded by the encoder 150 and signaled to the video decoding device. When the leaf node of the QT is not larger than the maximum block size (MaxBTSize) allowed for the root node in the BT, further segmentation can be performed according to one or more of the BT or TT structures. Multiple segmentation directions can exist in the BT and / or TT structures. For example, two directions can exist: horizontal and vertical segmentation of node blocks. Figure 2 As shown, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether a node is split, a flag indicating the splitting direction (vertical or horizontal), and / or a flag indicating the splitting type (binary or ternary) are encoded by encoder 150 and signaled to the video decoding device.

[0040] As another example of a tree structure, when using a QTBTTT structure to segment blocks, information regarding the CU segmentation flag (split_cu_flag) indicating that a block has been segmented and the QT segmentation flag (split_qt_flag) indicating whether the segmentation type is QT segmentation is encoded by the encoder 150 and signaled to the video decoding device. When the value of split_cu_flag indicates that a block has been segmented, the block of the node becomes a leaf node in the segmentation tree structure and is used as a coding unit (CU), which is the basic unit of encoding. When the value of split_cu_flag indicates that a block has not been segmented, the value of split_qt_flag distinguishes whether the segmentation type is QT or MTT. When the segmentation type is QT, there is no additional information. When the segmentation type is MTT, the flag indicating the MTT segmentation direction (vertical or horizontal) (mtt_split_cu_vertical_flag) and / or the flag indicating the MTT segmentation type (binary or ternary) (mtt_split_cu_binary_flag) is encoded by the encoder 150 and signaled to the video decoding device.

[0041] As another example of a tree structure, when using QTBT, two splitting types are possible: horizontal splitting of a node's block (i.e., symmetrical horizontal splitting) and vertical splitting (i.e., symmetrical vertical splitting) into two blocks of the same size. A splitting flag (split_flag) indicating whether each node in the BT structure is split into lower-level blocks, and splitting type information indicating the splitting type, are encoded by encoder 150 and sent to the video decoding device. Additional types are possible, such as splitting a node's block into two asymmetrical blocks. Asymmetrical splitting types can include types that split a block into two rectangular blocks at a 1:3 size ratio and types that split a node's block diagonally.

[0042] The CU can have various sizes depending on the QTBT or QTBTTT segmentation of the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block".

[0043] Predictor 120 predicts the current block to generate a prediction block. Predictor 120 includes an intra-frame predictor 122 and an inter-frame predictor 124.

[0044] Typically, each current block in a frame can be predicted and encoded. Prediction of the current block can be performed using either intra-frame prediction (based on data from the frame containing the current block) or inter-frame prediction (based on data from a frame encoded before the frame containing the current block). Inter-frame prediction includes both one-way and two-way prediction.

[0045] Intra-predictor 122 uses pixels (reference pixels) in the current frame surrounding the current block, including the current block, to predict pixels in the current block. Multiple intra-prediction modes exist depending on the prediction direction. For example, such as... Figure 3 As shown, multiple intra-frame prediction modes can include non-directional modes (including planar and DC modes) and 65 directional modes. Neighboring pixels and the equations used are defined differently for each prediction mode.

[0046] Intra-predictor 122 can determine the intra-prediction mode to be used for encoding the current block. In some examples, intra-predictor 122 can use multiple intra-prediction modes to encode the current block and select the appropriate intra-prediction mode from the tested modes. For example, intra-predictor 122 can use rate-distortion analysis of multiple tested intra-prediction modes to calculate rate-distortion values ​​and can select the intra-prediction mode with the best rate-distortion characteristics among the tested modes.

[0047] Intra-predictor 122 selects one intra-prediction mode from a plurality of intra-prediction modes and uses neighboring pixels (reference pixels) and an equation determined according to the selected intra-prediction mode to predict the current block. Information about the selected intra-prediction mode is encoded by encoder 150 and sent to video decoding device.

[0048] Inter-frame predictor 124 generates a predicted block for the current block through motion compensation processing. The inter-frame predictor searches for the most similar block in a reference frame that was encoded and decoded earlier than the current frame, and generates a predicted block for the current block based on the found block. Then, the inter-frame predictor generates a motion vector corresponding to the displacement between the current block in the current frame and the predicted block in the reference frame. Typically, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma and chroma components. Motion information, including information about the reference frame used to predict the current block and information about the motion vector, is encoded by encoder 150 and sent to the video decoding device.

[0049] Subtractor 130 generates a residual block by subtracting the prediction block generated by intra-predictor 122 or inter-predictor 124 from the current block.

[0050] Transformer 140 transforms the residual signal in the residual block with pixel values ​​in the spatial domain into transform coefficients in the frequency domain. Transformer 140 can use the total size of the current block as a transform unit to transform the residual signal in the residual block. Alternatively, the transformer can divide the residual block into sub-blocks of transformed and non-transformed regions, and only use the sub-blocks of the transformed regions as transform units to transform the residual signal. Here, the sub-block of the transformed region can be one of two rectangular blocks with a 1:1 size ratio based on the horizontal axis (or vertical axis). In this case, a flag indicating that only this sub-block is transformed (cu_sbt_flag), orientation (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) is encoded by encoder 150 and signaled to the video decoding device. In addition, the size of the sub-block of the transformed region can have a 1:3 size ratio based on the horizontal axis (or vertical axis). In this case, the flag used to distinguish the segment (cu_sbt_quad_flag) is separately encoded by the encoder 150 and signaled to the video decoding device.

[0051] The quantizer 145 quantizes the transform coefficients output from the converter 140 and outputs the quantized transform coefficients to the encoder 150.

[0052] Encoder 150 generates a bitstream by encoding the quantized transform coefficients using an encoding method such as context-based adaptive binary arithmetic code (CABAC). Encoder 150 encodes information such as CTU size, CU segmentation flag, QT segmentation flag, MTT segmentation direction, and MTT segmentation type related to block segmentation, so that the video decoding device segments the blocks in the same way as the video encoding device.

[0053] In addition, encoder 150 encodes information about the prediction type that indicates whether the current block is encoded by intra-frame prediction or inter-frame prediction, and encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (information about the reference frame and motion vectors) according to the prediction type.

[0054] Inverse quantizer 160 inverse quantizes the quantized transform coefficients output from quantizer 145 to generate transform coefficients. Inverse transformer 165 transforms the transform coefficients output from inverse quantizer 160 from the frequency domain to the spatial domain and reconstructs the residual block.

[0055] Adder 170 adds the reconstructed residual block to the prediction block generated by predictor 120 to reconstruct the current block. Pixels in the reconstructed current block are used as reference pixels for intra-frame prediction of the next block.

[0056] Filter unit 180 filters the reconstructed pixels to reduce block artifacts, ringing artifacts, and blur artifacts generated due to block-based prediction and transform / quantization. Filter unit 180 may include a deblocking filter 182 and a sample adaptive offset (SAO) filter 184.

[0057] Deblocking filter 182 filters the boundaries between reconstructed blocks to remove block artifacts caused by block-by-block encoding / decoding, and SAO filter 184 further filters the deblocked video. SAO filter 184 is a filter used to compensate for differences between reconstructed pixels and original pixels caused by lossy encoding.

[0058] The reconstructed blocks filtered by deblocking filter 182 and SAO filter 184 are stored in memory 190. Once all blocks in a frame have been reconstructed, the reconstructed frame is used as a reference frame for inter-frame prediction of blocks in the next frame to be encoded.

[0059] Figure 4 This is an exemplary functional block diagram of a video decoding device capable of implementing the technology disclosed herein. The following will refer to... Figure 4 Describe the video decoding device and its components.

[0060] The video decoding device may include a decoder 410, an inverse quantizer 420, an inverse converter 430, a predictor 440, an adder 450, a filter unit 460, and a memory 470.

[0061] Similar to Figure 1 The video encoding and decoding devices can have their components implemented as hardware or software, or a combination of both. Furthermore, the functions of each component can be implemented as software, and the microprocessor can be implemented to execute the software functions corresponding to each component.

[0062] Decoder 410 determines the current block to be decoded by decoding the bitstream received from the video encoding device and extracting information related to block segmentation, and extracts prediction information and information about the residual signal required to reconstruct the current block.

[0063] Decoder 410 extracts information about the CTU size from the Sequence Parameter Set (SPS) or Picture Parameter Set (PPS), determines the size of the CTU, and segments the picture into CTUs of the determined size. Then, the decoder identifies the CTU as the top level (i.e., the root node of the tree structure) and extracts segmentation information about the CTU to segment the CTU using the tree structure.

[0064] For example, when using the QTBTTT structure to segment a CTU, firstly, the first flag (QT_split_flag) related to QT segmentation is extracted, and each node is segmented into four lower-level nodes. Then, for the nodes corresponding to the leaf nodes of the QT, the second flag (MTT_split_flag) related to MTT segmentation and information about the segmentation direction (vertical / horizontal) and / or segmentation type (binary / ternary) are extracted, and the leaf nodes are segmented according to the MTT structure. In this way, the nodes below the leaf nodes of the QT are recursively segmented according to the BT or TT structure.

[0065] As another example, when partitioning a CTU using a QTBTTT structure, the CU partition flag (split_cu_flag) indicating whether a CU has been partitioned is first extracted. If the corresponding block has been partitioned, the QT partition flag (split_qt_flag) is extracted. When the partition type is not QT but MTT, a flag indicating the MTT partition direction (vertical or horizontal) (mtt_split_cu_vertical_flag) and / or a flag indicating the MTT partition type (binary or ternary) (mtt_split_cu_binary_flag) are also extracted. During the partitioning process, each node may undergo recursive QT partitioning zero or more times, followed by recursive MTT partitioning zero or more times. For example, a CTU may be partitioned immediately by MTT, or it may be partitioned multiple times by QT alone.

[0066] As another example, when using a QTBT structure to split a CTU, the first flag (QT_split_flag) is related to the QT split, and each node is split into four lower-level nodes. For nodes corresponding to leaf nodes of the QT, the split_flag indicating whether the node is further split by BT and the split direction information are extracted.

[0067] When determining the current block to be decoded through tree structure segmentation, decoder 410 extracts information about the prediction type, indicating whether the current block undergoes intra-frame prediction or inter-frame prediction. When the prediction type information indicates intra-frame prediction, decoder 410 extracts the syntactic elements of the intra-frame prediction information (intra-frame prediction mode) for the current block. When the prediction type information indicates inter-frame prediction, decoder 410 extracts the syntactic elements of the inter-frame prediction information, that is, information indicating the motion vector and the reference frame to which the motion vector is referenced.

[0068] Decoder 410 extracts information about the quantized transform coefficients of the current block as information about the residual signal.

[0069] The inverse quantizer 420 inverse quantizes the quantized transform coefficients and inverse transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to reconstruct the residual signal, thereby generating the residual block of the current block.

[0070] Additionally, when the inverse transformer 430 performs inverse transformation only on a portion of the transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only the sub-block of the transform block was transformed, as well as orientation information (vertical / horizontal) (cu_sbt_horizontal_flag) and / or sub-block position information (cu_sbt_pos_flag) about the sub-block. Then, the residual signal is reconstructed by inverse transforming the transform coefficients of the sub-block from the frequency domain to the spatial domain. For regions that have not undergone inverse transformation, the residual signal is padded with "0". Thus, the final residual block for the current block is created.

[0071] Predictor 440 may include intra-predictor 442 and inter-predictor 444. Intra-predictor 442 is enabled when the prediction type of the current block is intra-prediction, and inter-predictor 444 is enabled when the prediction type of the current block is inter-prediction.

[0072] Intra-predictor 442 determines the intra-prediction mode of the current block among multiple intra-prediction modes based on the syntactic elements of the intra-prediction mode extracted from decoder 410, and predicts the current block based on the reference pixels around the current block according to the intra-prediction mode.

[0073] The inter-frame predictor 444 determines the motion vector of the current block and the reference frame to which the motion vector is referenced based on the syntactic elements of the intra-frame prediction mode extracted from the decoder 410, and predicts the current block based on the motion vector and the reference frame.

[0074] Adder 450 reconstructs the current block by adding the residual block output from the inverse transform to the prediction block output from the inter-predictor or intra-predictor. Pixels in the reconstructed current block are used as reference pixels for intra-prediction of blocks to be decoded later.

[0075] Filter unit 460 may include a deblocking filter 462 and a SAO filter 464. Deblocking filter 462 performs deblocking filtering on the boundaries between reconstructed blocks to remove block artifacts caused by block-by-block decoding. SAO filter 464 performs additional filtering on the reconstructed blocks after deblocking filtering to compensate for differences between reconstructed pixels and original pixels caused by lossy coding. The reconstructed blocks filtered by deblocking filter 462 and SAO filter 464 are stored in memory 470. When all blocks in a frame have been reconstructed, the reconstructed frame is used as a reference frame for inter-frame prediction of blocks in frames to be encoded subsequently.

[0076] The HEVC international standard uses two types of methods to partition the image (or screen) for parallel processing: slicing and tile partitioning.

[0077] 1) The slicing method provides the function of dividing the screen into CTUs and encoding / decoding the CTUs according to the raster scan order. Multiple slices can exist within a single slice, and there are two types of slices: independent slices and dependent slices.

[0078] Independent segments are segments that do not depend on another slice during intra-frame prediction, inter-frame prediction, coding mode, and entropy coding. Independent segments have header information. Dependent segments are segments that depend on another slice (referencing another preceding independent segment) during intra-frame prediction, inter-frame prediction, coding mode, and entropy coding. For most of the header information of a dependent segment, the dependent segment references the header information of the preceding independent segment. In other words, only a portion of the header information of the dependent segment is signaled.

[0079] Figure 5 (a) shows two slices divided from the image, which can be separated by slice boundaries. A slice may include one independent slice and zero or more dependent slices. The first slice at the top of the image has one independent slice and two dependent slices separated by a dashed line. The second slice at the bottom of the image has only one independent slice (including zero dependent slices).

[0080] Tables 1 and 2 show the syntactic structure of the slice division method.

[0081] [Table 1]

[0082] [Table 2]

[0083] Table 1 shows the flag indicating whether dependent slices are enabled. This flag can be defined by PPS. Table 2 shows the slice header information.

[0084] As shown in Table 2, a flag indicating whether a specific slice is the first slice in the image (first_slice_segment_in_pic_flag) is first used to indicate whether it is an independent slice. For slices other than the first slice, a flag indicating whether they are dependent slices (dependent_slice_segment_flag) is first used to indicate whether they are dependent slices. Additionally, the address of the related slice (slice_segment_address) is also first used to indicate the address. Here, slice_segment_address refers to the index value of the first CTU in the related slice.

[0085] 2) The tile partitioning method provides the function of encoding / decoding the screen by segmenting / dividing the screen into multiple columns and rows based on CTU. The screen can be divided into tiles based on a uniform partitioning scheme or a separate (non-uniform) partitioning scheme.

[0086] Each partitioned piece does not depend on other pieces during intra-frame prediction and entropy coding, and its dependence on loop filtering is controlled by flags defined in the PPS. There is no separate header for each piece.

[0087] Figure 5 (b) shows a screen divided into 3×3 tiles, with each tile separated by tile boundaries (column and row boundaries). Multiple tiles can be encoded / decoded in raster scan order, and multiple CTUs within a single tile can also be encoded / decoded in raster scan order. The numbers shown in each CTU indicate the order in which encoding / decoding is performed.

[0088] Table 3 shows the syntactic structure of the chunking method.

[0089] [Table 3]

[0090] As shown in Table 3, the tiles_enabled_flag is defined in PPS and signals are used to indicate the on / off state of tile division. When the relevant flag is on, the syntax element specifying the tile size is also signaled.

[0091] Syntactic elements that specify tile size may include: num_tile_columns_minus1, which indicates the value obtained by subtracting 1 from the number of tiles (number of tile columns) in the vertical direction; num_tile_rows_minus1, which indicates the value obtained by subtracting 1 from the number of tiles (number of tile rows) in the horizontal direction; and uniform_spacing_flag, which indicates whether the screen is evenly divided in both the horizontal and vertical directions.

[0092] When the screen is not uniformly divided (uniform_spacing_flag=off), the width information of the tile column (column_width_minus1) and the height information of the tile row (row_height_minus1) are communicated via a signal. Additionally, a flag indicating whether loop filtering is applied at the boundaries between tiles (loop_filter_across_tiles_enabled_flag) is communicated via a signal.

[0093] Figure 6 This illustrates the differences between the above-mentioned slicing method and the block-based partitioning method. Figure 6 (a) shows the slices (4 slices in total) divided according to the slice division method. Figure 6 (b) shows the tiles divided according to the tile partitioning method (4 tiles in total). The numbers represent the number of CTUs.

[0094] There may be only horizontal boundaries between dividing slices. Conversely, there may be both horizontal and vertical boundaries between dividing pieces.

[0095] This disclosure presents image segmentation methods for various purposes.

[0096] Specifically, this disclosure proposes 1) a sub-picture partitioning method for synthesizing different sequences, 2) a slice partitioning method for bitstream packing, 3) a block partitioning method for parallel processing, and 4) a method for expressing layout information for sub-pictures synthesized with other sequences and a method for expressing layout information that can be configured.

[0097] Pieces, slices, and sub-pictures

[0098] Before describing this disclosure in detail, the terms "piece", "slice", and "sub-picture" are defined as parts or segments of a picture.

[0099] A mosaic refers to a frame region obtained by dividing the frame into sub-regions for parallel processing. Mosaics form a structure where each mosaic operates independently during intra-frame prediction and entropy coding. In other words, mosaics do not share reference samples during intra-frame prediction and do not share context during entropy coding. Whether loop filtering is applied to the boundaries between mosaics can be controlled by a separate flag.

[0100] A slice refers to a data packing unit that configures an independent Network Abstraction Layer (NAL) bitstream. A slice can contain multiple pieces, or a piece can contain multiple slices.

[0101] Slices can be of two types. The first type is a rectangular slice where tiles with similar characteristics or any tile is designated as a region. The second type is a raster scan slice (non-rectangular shape) where tiles or CTUs are designated as a region at a specific location and the encoding / decoding of tiles or CTUs is performed in raster scan order. In the latter case, a raster scan slice where tiles are designated as a region at a specific location and the encoding / decoding is performed on a tile-by-tile basis is a "tile-based raster scan slice," and a raster scan slice where CTUs are designated as a region at a specific location and the encoding / decoding is performed on a CTU-by-CTU basis is a "CTU-based raster scan slice."

[0102] Figure 7 Examples of rectangular slices and raster scan slices are shown. Figure 7 In the diagram, areas separated by dashed lines represent CTUs, areas separated by straight lines represent tiles, and areas separated by dotted-dash lines represent slices.

[0103] like Figure 7 As shown in (a), a frame can be divided into a total of 216 (18×12) CTUs, a total of 24 (6×4) tiles, and a total of 9 rectangular slices. Figure 7 As shown in (b), a frame can be divided into a total of 216 (18×12) CTUs, a total of 12 (3×4) tiles and a total of 3 raster scan slices.

[0104] A sub-picture is a designated area used for composition with other content. A sub-picture may include at least one slice.

[0105] Figure 8 This shows an example of different content being combined into sub-pictures. The area filled with diagonal lines in the upper right corner represents the picture in content 1. The area filled with horizontal lines represents the picture in content 2. The area filled with vertical lines represents the picture in content 3.

[0106] like Figure 8As shown on the left, each frame can consist of four slices, and a bitstream can be generated for each slice unit. Furthermore, each slice can be defined as a sub-frame. The bitstream, physically generated in units of slices, is input to a bitstream extractor and merger (BEAMer) and undergoes extraction and compositing processes. Here, the logical unit for extraction and compositing is the sub-frame. As a result, a new frame is generated from the compositing, where the upper left is content 1, the lower left is content 2, and the right is content 3.

[0107] BEAMer extracts bitstreams generated in slices from different content and synthesizes the extracted bitstreams in sub-pictures. For this purpose, the pictures of each content are divided into sub-pictures. In other words, a sub-picture is a region that includes at least one slice used for content synthesis.

[0108] Figure 9 An exemplary block diagram of a decoder 410 capable of implementing the methods of this disclosure is shown. The decoder 410 may include a decoding execution unit 910, a partitioner 920, a mapping unit 930, and a configuration unit 940.

[0109] The video encoding device can determine whether the frame is divided into one or more sub-units, encode the result (segmentation flag), and notify the encoding result with a signal. The decoding execution unit 910 can decode the segmentation flag (S1010) from the bitstream and determine whether the frame is divided into sub-units based on the segmentation flag.

[0110] The sub-units that can be divided from the screen can include CTUs, tiles, slices, and sub-pictures. In other words, the screen can be divided into CTUs, tiles, slices, and sub-pictures.

[0111] A segmentation flag can indicate whether a screen is divided into sub-units or whether a segmentation is applied to the screen. The segmentation flag can be a sub-screen flag indicating whether the screen is divided into one or more sub-screens, or partitioning permission information indicating whether a segmentation is applied to the screen. Specific details of the segmentation flag will be described with reference to the various implementation methods.

[0112] When the segmentation flag indicates that the screen is divided into sub-units, the video encoding device can encode the segmentation-related information (segmentation information) of the screen and notify it with a signal. When the segmentation flag indicates that the screen is divided, the decoding execution unit 910 can decode the segmentation information from the bitstream (S1020).

[0113] The partitioning information may include information about the number of sub-units divided from the screen (quantity information) and information about their size (size information). Additionally, the partitioning information may also include information about the location of the sub-units (location information).

[0114] The divider 920 can divide the screen into one or more sub-units based on partition information (using partition information) (S1030).

[0115] Screen division

[0116] Methods for partitioning a screen may include 1) dividing the screen into fixed-size CTUs, 2) dividing the screen into tiles comprising at least one or more CTUs for parallel processing, and 3) dividing the screen into slices for generating a bitstream. Additionally, methods for partitioning a screen may include 4) dividing the screen into multiple sub-screens for content compositing.

[0117] 1) The screen can be divided into CTUs. As shown in Table 4, the video encoding device can define and signal the size of the CTU at one or more levels between SPS and PPS; and the video decoding device can decode the size of the CTU and determine the CTU size.

[0118] [Table 4]

[0119] The minimum and maximum size of the CTU can be limited equally at both the video encoding and video decoding devices. Table 4 shows an example where the minimum size of the CTU is set to 32 (2 to the power of 5).

[0120] 2) The screen can be divided into tiles. A tile may include at least one or more CTUs. In other words, the size (width and height) of a tile can be expressed as a multiple of the CTU size (tiles can be divided in multiples of CTU).

[0121] Figure 11 (a) shows an example where the screen is divided into 3×2 tiles. The numbers indicated on each tile represent the index value of the corresponding tile, and tile indices can be assigned according to the raster scan order based on how the screen is divided into tiles. In addition to assigning tile index values ​​according to raster scan order, tile index values ​​can also be derived based on specific rules. These specific rules will be described later.

[0122] 3) The screen can be divided into slices. The screen can be divided such that a slice contains multiple tiles, or a tile contains multiple slices. In other words, tiles can be used to perform slice division.

[0123] Figure 11 (b) shows an example where the screen is divided into a total of 6 slices. The numbers indicated on each slice refer to the index value of the corresponding slice, and slice indices can be assigned according to the raster scan order based on how the screen is divided into slices. Slicing can be performed such that... Figure 11 Slice 0 of (b) includes Figure 11(a) contains pieces 0 and 1. Figure 11 Slices 1 and 2 of (b) are included Figure 11 In piece 2 of (a), Figure 11 Slices 3 and 5 of (b) are included Figure 11 In piece 3 of (a), Figure 11 Slice 4 of (b) includes Figure 11 Pieces 4 and 5 of (a).

[0124] In addition to assigning slice index values ​​according to the raster scan order, slice indices can also be derived based on specific rules. These specific rules will be described later. Figure 11 Each of the six slices in (b) can be packed into a bitstream, such as Figure 11 As shown in (c).

[0125] 4) The screen can be divided into sub-screens. A sub-screen may include at least one slice.

[0126] Figure 12 This shows an example where the screen is divided into a total of 5 sub-screens. Figure 12 (a) shows an example of index values ​​(0 to 3) being assigned to a sub-screen, and (b) shows an example of identifier (id) values ​​(A to D) being assigned to a sub-screen.

[0127] The index values ​​of sub-pictures can be assigned according to the raster scan order based on how the image is divided into sub-pictures. The image can be divided into sub-pictures, making... Figure 12 The sub-screen 0 of (a) includes Figure 11 slice (b) 0 and Figure 12 Sub-screen 1 of (a) includes Figure 11 Slices 1 and 2 of (b).

[0128] To distinguish the ID value of a sub-picture from its index value, the ID value of a sub-picture can have a larger value than the allowed value of the index value of the sub-picture within the same picture. For example, when numbers 0 to 3 are assigned to... Figure 12 When the index value of (a) is used, the id value of the sub-screen is ( Figure 12 (b) A through D) can be set to a value greater than 3.

[0129] Based on the above method for dividing the screen, it can be seen that the size of the mosaic, slice, and sub-screen are all multiples of the CTU.

[0130] Implementation Method 1

[0131] Implementation 1 relates to a method for dividing a screen into sub-screens using syntactic elements defined in SPS. Implementation 1 proposes syntactic structures and semantics related to sub-screen division.

[0132] Implementation Method 1-1

[0133] The video encoding device can determine whether a subpic is present in the frame (whether the frame is divided into subpics), encode a subpic flag (subpic_present_flag or subpic_info_present_flag) as information about the determination result, and notify the encoded subpic flag with a signal. The video decoding device decodes the subpic_info_present_flag (S1310) and determines whether a subpic exists based on the subpic_info_present_flag value (S1320).

[0134] `subpic_info_present_flag` indicates whether a subpic exists within the current frame, whether subpic parameters exist, or whether subpic partitioning has been performed. When `subpic_info_present_flag=0`, it indicates that no subpic exists; when `subpic_info_present_flag=1`, it indicates that a subpic exists. The subpic flag can be `subpics_present_flag`.

[0135] In the presence of subpics, the video encoding device can encode information about the number of subpics (num_subpics_in_pic_minus1) and signal the encoded information. The video decoding device can decode num_subpics_in_pic_minus1 (S1330) and determine the number of subpics based on num_subpics_in_pic_minus1.

[0136] `num_subpics_in_pic_minus1` is a syntactic element representing the value obtained by subtracting 1 from the number of subpics present in the frame. If `num_subpics_in_pic_minus1` does not exist, the value of the corresponding syntactic element can be inferred as 0. `num_subpics_in_pic_minus1` can also be implemented using `num_subpics_minus1`.

[0137] Since at least two subpics can exist in a frame, the number of subpics can be determined by subtracting 2 from the number of subpics in the frame (num_subpics_in_pic_minus2).

[0138] The video encoding device can encode as many pieces of information as there are sub-pictures (sub-picture position information and size information) indicating the position and size of each sub-picture and notify the encoded information with signals. The video decoding device can decode as many pieces of information as there are sub-pictures indicating the position and size of each sub-picture (S1340) and determine the position and size of each sub-picture based on the relevant information.

[0139] The position information of a subpic can be represented by the index value of the CTU at the top left position in the subpic (e.g., topleft_ctu_idx) or by the coordinates representing the horizontal and vertical positions of the CTU at the top left position (e.g., subpic_ctu_top_left_x and subpic_ctu_top_left_y).

[0140] The size information of a subpicture can be represented by values ​​indicating the width and height of the subpicture relative to the top-left CTU (e.g., subpic_width and subpic_height) (i.e., the number of CTUs included in the relevant subpicture in the horizontal and vertical directions) or by values ​​obtained by subtracting 1 from the width and height (e.g., subpic_width_minus1 and subpic_height_minus1). The size information of a subpicture can also be represented by the index value (bottomright_ctu_idx) of the CTU at the bottom-right position of the relevant subpicture.

[0141] Video decoding devices can divide a video into sub-pictures based on the decoded syntactic elements (information about the number of sub-pictures, the location of the sub-pictures, and the size of the sub-pictures).

[0142] In addition, mapping information for sub-screens can be defined at the SPS and signaled to them.

[0143] Mapping refers to associating a subpicture with a slice. Mapping information is used to associate a subpicture with a slice and may include the subpicture's ID (subpic_id) and the ID's length information (subpic_id_lens_minus1). Mapping information can also be referred to as the subpicture's configuration information.

[0144] The video encoding device can encode the mapping information and notify it with a signal, and the video decoding device can decode the mapping information from the bitstream S1370. The mapping unit 930 can map sub-pictures to slices based on the mapping information.

[0145] Based on the implementation method, the execution mapping can be determined based on the mapping flag indicating whether to execute the ID mapping of the sub-screen.

[0146] In the above embodiment, the video encoding device can encode the mapping flag and notify it with a signal. The video decoding device can decode the mapping flag (S1350) and determine whether to perform ID mapping based on the decoded mapping flag (S1360).

[0147] Mapping flags can be implemented using either `signaled_subpic_id_flag` indicating whether subpic IDs are signaled or `subpic_mapping_enabled_flag` indicating whether subpic mapping is enabled. Alternatively, mapping flags can be implemented in the form of `sps_subpic_id_present_flag` indicating whether ID mapping is performed in the SPS and `sps_subpic_id_signalling_present_flag` indicating whether mapping information is signaled in the SPS.

[0148] When the mapping flag indicates that mapping should be performed, the video encoding device can encode as many subpic IDs as there are subpics (subpic_id[i]) and signal accordingly. When the mapping flag indicates that mapping should be performed, the video decoding device can decode as many subpic IDs as there are subpics (subpic_id[i]) (S1370) and can map the subpics to slices.

[0149] Table 5 shows the syntax structure of Implementation 1-1. The for loop in Table 5 can be executed in raster scan order.

[0150] [Table 5]

[0151] The `subpic_treated_as_pic_flag` in Table 5 is a syntax element that indicates whether the boundaries of subpictures are treated as picture boundaries. For example, `subpic_treated_as_pic_flag` can indicate that the boundaries of subpictures are treated as picture boundaries during inter-frame prediction (by padding the boundary pixel values). The `loop_filter_across_subpic_enabled_flag` in Table 5 is a syntax element that indicates whether loop filtering is applied to the boundaries between subpictures.

[0152] The video encoding device can encode the same number of `subpic_treated_as_pic_flags` and `loop_filter_across_subpic_enabled_flags` as the number of subpics and signal them accordingly. The video decoding device decodes the same number of `subpic_treated_as_pic_flags` and `loop_filter_across_subpic_enabled_flags` as the number of subpics and determines, based on `subpic_treated_as_pic_flags` and `loop_filter_across_subpic_enabled_flags`, whether to treat the boundaries of subpics as picture boundaries and whether to apply loop filtering to the boundaries between subpics.

[0153] The following shows the method of... Figure 12 The result obtained by applying implementation method 1-1 is shown in (b) as an example of sub-screen division.

[0154] subpic_info_present_flag=true(1), num_subpics_in_pic_minus1=3 (num_subpics_in_pic_minus2=2), signalled_subpic_id_flag=true(1), subpic_id={A, B, C,D} (raster scan order).

[0155] Implementation Methods 1-2

[0156] Implementations 1-2 relate to a method of signaling a subpicture ID independently of a mapping flag (e.g., signaled_subpic_id_flag).

[0157] In Example 1-2, the processing other than steps S1350 and S1360 can be performed in the same manner as in Example 1-1.

[0158] Specifically, the video encoding device can notify the sub-picture ID using a signal instead of the mapping flag. The video decoding device can decode the sub-picture ID without decoding the mapping flag, that is, except for steps S1350 and S1360 (S1370).

[0159] Table 6 shows the syntax structures for embodiments 1-2. The for loops in Table 6 can be executed in raster scan order.

[0160] [Table 6]

[0161] Implementation methods 1-3

[0162] Implementation methods 1-3 relate to a method for signaling a sub-screen ID independently of the mapping flag and the sub-screen's position and size information.

[0163] In embodiments 1-3, steps S1350 and S1360 are excluded from the steps of embodiment 1-1, and steps S1340 and S1370 can be performed together.

[0164] Specifically, after encoding the information about the number of sub-frames and notifying it with a signal, the video encoding device can notify the sub-frame ID along with the sub-frame's position and size information using a signal. After decoding the information about the number of sub-frames (S1330), the video decoding device can decode the sub-frame ID along with the sub-frame's position and size information (S1340 and S1370).

[0165] Table 7 shows the syntax structures of embodiments 1-3. The for loop in Table 7 can be executed in raster scan order.

[0166] [Table 7]

[0167] Implementation methods 1-4

[0168] Implementation methods 1-4 relate to a method for signaling the length information of a sub-screen ID along with the sub-screen ID.

[0169] The length information of the subpic ID indicates the number of bits used to represent the subpic ID and can be implemented in the form of signed_subpic_id_length_minus1 or subpic_id_len_minus1. The subpic ID can be represented using the same number of bits as indicated by the length information of the subpic ID.

[0170] When the mapping flag indicates that ID mapping should be performed, the video encoding device can encode the length information of the sub-screen ID and the sub-screen ID and notify it with a signal. When the mapping flag indicates that ID mapping should be performed (S1360), the video decoding device can decode the sub-screen ID and the length information of the sub-screen ID (S1370).

[0171] Table 8 shows the syntax structures of embodiments 1-4. The for loop in Table 8 can be executed in raster scan order.

[0172] [Table 8]

[0173] Implementation methods 1-5

[0174] Implementation methods 1-5 relate to a method for signaling that does not perform mapping information in SPS.

[0175] Therefore, Implementation 1-5 can exclude steps S1350, S1360 and S1370 related to the signaling of mapping information from the steps of Implementation 1-1.

[0176] Table 9 shows the syntax structures of embodiments 1-5. The for loop in Table 9 can be executed in raster scan order.

[0177] [Table 9]

[0178] Implementation Method 2

[0179] Implementation 2 relates to a method for dividing a screen into tiles. Implementation 2 proposes syntactic structures and semantics related to tile division. Syntactic elements for tile division can be defined at the PPS or screen header level.

[0180] The video encoding device can determine whether to apply segmentation to the screen and notify the determination result by signaling segmentation permission information (segmentation flag). The video decoding device can decode the segmentation permission information (S1410, S1430) and determine whether to apply segmentation (S1420, S1440).

[0181] The partitioning permission information indicates whether the screen is divided into tiles and can be implemented by single_tile_in_pic_flag. For example, when single_tile_in_pic_flag=1, it indicates that the screen is not divided into tiles, while when single_tile_in_pic_flag=0, it indicates that the screen is divided into multiple tiles.

[0182] Partition permission information can also be implemented using a combination of `no_partition_flag` and `single_tile_in_pic_flag`. When `no_partition_flag=1`, it indicates that no partitioning is applied to the screen, while when `no_partition_flag=0`, it indicates that partitioning is applied to the screen. In this case, when `single_tile_in_pic_flag=1`, it indicates that the screen is divided into a single tile, while when `single_tile_in_pic_flag=0`, it indicates that the screen is divided into multiple tiles.

[0183] Partition permission information can also be implemented using a single syntax element, no_pic_partition_flag, based on a combination of no_partition_flag and single_tile_in_pic_flag. In this case, when no_pic_partition_flag=1, it indicates that partitioning is not applied to the screen, while when no_pic_partition_flag=0, it indicates that the screen is divided into multiple tiles.

[0184] When the partition permission information indicates that the application should be partitioned, the video encoding device can determine the number of tiles to be partitioned and encode the information about the number of tiles and notify it with a signal. When the partition permission information indicates that the application should be partitioned, the video decoding device can decode the information about the number of tiles (S1450) and set the number of tiles based on the decoded information about the number of tiles.

[0185] Information regarding the number of tiles can include the number of tiles divided vertically (the number of tile columns) and the number of tiles divided horizontally (the number of tile rows). The number of tile columns can be represented by a value obtained by subtracting 1 from the actual number of tile columns (num_tile_columns_minus1 or num_exp_tile_columns_minus1). The number of tile rows can be represented by a value obtained by subtracting 1 from the actual number of tile rows (num_tile_rows_minus1 or num_exp_tile_rows_minus1).

[0186] The video encoding device can identify the size of each tile column and each tile row, encode information about the size of the tile columns and rows (tile size information), and notify this information with a signal. The video decoding device can decode the tile size information (S1460). The size of the tile columns and rows can be set based on the decoded tile size information.

[0187] The size information of a tile column can be represented by the width value (number of CTUs) of the relevant tile column based on the CTU at the top left position in the relevant tile column (tile_column_width_minus1). The size information of a tile row can be represented by the height value (number of CTUs) of the relevant tile row based on the CTU at the top left position in the relevant tile row (tile_row_height_minus1).

[0188] Since the width of the last tile column can be inferred without a separate definition, the size information of the tile column can be signaled as many times as the number obtained by subtracting 1 from the number of tile columns. Similarly, since the height of the last tile row can be inferred without a separate definition, the size information of the tile row can be signaled as many times as the number obtained by subtracting 1 from the number of tile rows.

[0189] Implementation Method 2-1

[0190] Implementation method 2-1 relates to a method for implementing partition permission information using single_tile_in_pic_flag.

[0191] The video encoding device can encode the `single_tile_in_pic_flag` and notify it with a signal. When `single_tile_in_pic_flag` = 1 (S1440), since tile division is not applied, the video decoding device does not need to execute steps S1450 and S1460. On the other hand, when `single_tile_in_pic_flag` = 0 (S1440), since tile division is applied, the video decoding device can divide the image into tiles by executing steps S1450 and S1460.

[0192] Table 10 shows an example of the syntax structure of Implementation 2-1. The for loop in Table 10 can be executed in raster scan order.

[0193] [Table 10]

[0194] Implementation Method 2-2

[0195] Implementation method 2-2 relates to a method in which partition permission information is implemented using no_partition_flag and single_tile_in_pic_flag.

[0196] The video encoding device can encode no_partition_flag and notify it with a signal, and when no_partition_flag=0, it can encode single_tile_in_pic_flag and notify it with a signal. The video decoding device can decode no_partition_flag (S1410) and determine whether the screen has been partitioned (S1420).

[0197] Since tile partitioning is not applied when no_partition_flag=1 (S1420), the video decoding device does not need to perform subsequent steps. Unlike the above operation, when no_partition_flag=0 (S1420), the video decoding device can decode single_tile_in_pic_flag (S1430) and determine the value of single_tile_in_pic_flag (S1440).

[0198] Since the image is divided into a single tile when single_tile_in_pic_flag=1 (S1440), the video decoding device does not need to execute steps S1450 and S1460. Unlike the above operation, when single_tile_in_pic_flag=0 (S1440), due to the application of tile division, the video decoding device can divide the image into multiple tiles by executing steps S1450 and S1460.

[0199] Table 11 shows an example of the syntax structure for implementation 2-2. The for loop in Table 11 can be executed in raster scan order.

[0200] [Table 11]

[0201] In the examples in Table 11, when no_partition_flag=1, rectangular slicing or configuration cannot be performed on the relevant screen because tile partitioning is not applied. However, raster scan slicing or configuration can be performed on a CTU basis. When single_tile_in_pic_flag=1, since the screen is divided into one tile, one or more slices can be configured based on one tile. On the other hand, when single_tile_in_pic_flag=0, since the screen is divided into multiple tiles, rectangular slices can be configured based on multiple tiles. In this case, raster scan slicing can also be configured on a tile-by-tile basis.

[0202] Figure 11 The values ​​of the syntactic elements in implementation 2-1 of the block partitioning example shown in (a) are as follows.

[0203] single_tile_in_pic_flag=false(0), num_tile_colums_minus1=2, and num_tile_rows_minus1=1

[0204] Implementation Method 3

[0205] Implementation 3 relates to a method for dividing slices using tiles. The slice type to be divided in Implementation 3 is rectangular slices. Implementation 3 proposes syntactic structures and semantics related to slice division. The syntactic elements of slice division can be defined at the PPS or header level.

[0206] Implementation Method 3-1

[0207] The video encoding device can determine the type of slice divided from the frame and notify the determination result by signaling the type flag rect_slice_flag. The video decoding device can decode the type flag (S1530) and determine the slice type based on the type flag (S1540).

[0208] The type flag indicates whether the slice type is a rectangular slice. When rect_slice_flag=0, it indicates a raster scan slice; when rect_slice_flag=1, it indicates a rectangular slice.

[0209] When a subpic exists (subpic_info_present_flag=1), rect_slice_flag can be inferred as "1" without signal notification. In other words, subpic partitioning can be performed when the slice is a rectangular slice. When rect_slice_flag does not exist and subpic_info_present_flag=1, rect_slice_flag can be inferred as 1 (rect_slice_flag=1); when rect_slice_flag does not exist and subpic_info_present_flag=0, rect_slice_flag can be inferred as 0 (rect_slice_flag=0).

[0210] In the case of rectangular slices, the video encoding device can encode a relationship flag (single_tile_per_slice_flag) indicating the relationship between tiles and slices and notify it with a signal. The video decoding device can decode the single_tile_per_slice_flag (S1550) and use the single_tile_per_slice_flag to determine the relationship between tiles and slices (S1560).

[0211] Since tiles and slices have a one-to-one relationship when single_tile_per_slice_flag=1, no signal is needed to notify additional slice partition information, and tile partition information can be used as slice partition information. In this case, the video decoding device can divide the image into one or more slices based on the tile partition information.

[0212] When single_tile_per_slice_flag=0, it indicates a relationship where a slice includes multiple tiles or a tile includes multiple slices; therefore, additional slice partitioning information is signaled.

[0213] The video encoding device can encode and signal information about the number of slices in the frame (num_slices_in_pic_minus1) and whether multiple slices constitute a tile (more_than_one_slice_in_tile_flag[i]). After decoding num_slices_in_pic_minus1 (S1570), the video decoding device can decode more_than_one_slice_in_tile_flag[i] elements, which are the same number as the number of slices (S1580).

[0214] When more_than_one_slice_in_tile_flag=1, it indicates that multiple slices constitute a tile. When more_than_one_slice_in_tile_flag=0, it indicates that a slice includes one or more tiles.

[0215] When more_than_one_slice_in_tile_flag=1, it indicates that multiple slices are included in the relevant tile; therefore, information about the number of slices included in the tile (num_slices_in_tile_minus1) and the size of the slices included in the tile (slice_height_in_ctus_minus1) can be signaled.

[0216] `num_slices_in_tile_minus1` represents the value obtained by subtracting 1 from the number of slices included in the relevant tile. Since a tile can contain at least two slices, `num_slices_in_tile_minus1` can be changed to `num_slices_in_tile_minus2`. `slice_height_in_ctus_minus1` represents the value obtained by subtracting 1 from the height value (number of CTUs) of each slice included in the relevant tile. Here, the height of the last slice can be inferred.

[0217] After decoding the information about the number of slices (num_slices_in_tile_minus1) (S1592), the video decoding device can decode the size information (slice_height_in_ctus_minus1) of slices that are as numerous as the number of slices (S1594).

[0218] When more_than_one_slice_in_tile_flag=0, a slice may include one or more tiles. In this case, the video encoding device may encode and signal the size information (slice size information) representing the size of the relevant slice based on the number of tiles included in the relevant slice. The video decoding device may decode the slice size information represented by the number of tiles (S1596).

[0219] The slice size information based on the number of tiles can be composed of slice_width_in_tiles_minus1[i] and slice_height_in_tiles_minus1[i]. slice_width_in_tiles_minus1 represents the number of tile columns belonging to the relevant slice (the value obtained by subtracting 1 from the number of tile columns). slice_height_in_tiles_minus1 represents the number of tile rows belonging to the relevant slice (the value obtained by subtracting 1 from the number of tile rows).

[0220] Table 12 shows an example of the syntax structure of Implementation 3-1. The for loop in Table 12 can be executed in raster scan order.

[0221] [Table 12]

[0222] Implementation Method 3-2

[0223] Implementation method 3-2 relates to an alternative method for dividing a screen into slices using partition permission information (segmentation flag, no_partition_flag). Here, the partition permission information indicates whether the screen is divided into multiple slices; when no_partition_flag=1, it indicates that the screen is not divided into slices, and when no_partition_flag=0, it indicates that the screen is divided into multiple slices.

[0224] The video encoding device can determine whether the image is divided into multiple slices, set the determination result as the value of the partition permission information, and notify the set information with a signal. The video decoding device can decode the partition permission information (S1510) and determine whether to perform slice division based on the partition permission information (S1520).

[0225] When no_partition_flag=0, the screen is divided into slices; therefore, the slice division steps (S1530 to S1596) can be performed. However, when no_partition_flag=1, the screen is not divided into slices; therefore, steps S1530 to S1596 can be omitted. In other words, slice division can be omitted.

[0226] Table 13 shows an example of the syntax structure of Implementation 3-2. The for loop in Table 13 can be executed in raster scan order.

[0227] [Table 13]

[0228] Figure 11 The values ​​of the syntactic elements in implementation 3-1 of the slice partitioning example shown in (b) are as follows.

[0229] rect_slice_flag=true(1), single_tile_per_slice_flag=false(0), num_slices_in_pic_minus1=5

[0230] {more_than_one_slice_in_tile_flag=0, slice_width_in_tiles_minus1=1, slice_height_in_tiles_minus1=0}

[0231] {more_than_one_slice_in_tile_flag=1, num_slices_in_tile_minus1= 1(num_slices_in_tile_minus2=0), slice_height_in_ctus_minus1=number of CTUs}

[0232] {more_than_one_slice_in_tile_flag=1, num_slices_in_tile_minus1= 1(num_slices_in_tile_minus2=0), slice_height_in_ctus_minus1=number of CTUs}

[0233] {more_than_one_slice_in_tile_flag=0, slice_width_in_tiles_minus1=1, slice_height_in_tiles_minus1=0}

[0234] Figure 16 The results are shown by assigning or deriving slice indices while forming slices based on the above-described slice partitioning method.

[0235] It can be noted that, Figure 16 The derivation of the slice index is different Figure 11 The derivation of the slice index (raster scan order) in (b). Specifically, Figure 11 The slice index 4 of (b) is derived as Figure 16 Index 5 in Figure 11 The slice index 5 of (b) is derived as Figure 16 Index 4 in the table.

[0236] Figure 16 The derivation result of the slice index is obtained based on the specific rules mentioned above, where the specific rules may refer to the slice division method of Implementation Method 3.

[0237] Implementation Method 4

[0238] Implementation 4 relates to a method for notifying mapping information of a sub-screen using a signal. Implementation 4 proposes syntactic structures and semantics related to the method of notifying mapping information using a signal.

[0239] There are two scenarios where the mapping information can be notified via signals. The first scenario involves the sub-screen configuration remaining unchanged throughout the content, and the second scenario involves the sub-screen configuration changing in the middle of the content.

[0240] When the sub-screen configuration remains unchanged, the mapping information of the sub-screen can be notified by signal at the SPS level. For example, when the position and size of the sub-screen are notified by signal in SPS, the mapping information of the sub-screen can also be notified by signal (Implementation Method 1).

[0241] Whether the sub-picture configuration changes or remains unchanged, the sub-picture mapping information can be notified via signals in the picture header or PPS. In other words, the position and size information of the sub-picture can be notified via signals in the SPS, and the sub-picture mapping information can be notified via signals in one or more of the picture header, PPS, slice header, and SEI.

[0242] Since an example of signaling mapping information in SPS has been described in Implementation 1, an example of signaling mapping information in PPS will be described below.

[0243] The video encoding device can set the value of the mapping flag based on whether mapping information (subpic_id) exists (or whether to notify subpic_id with a signal) and encode the mapping flag and notify it with a signal. The video decoding device can decode the mapping flag (S1730) and determine whether subpic_id exists (or whether to notify subpic_id with a signal) based on the decoded mapping flag (S1740).

[0244] As mentioned above, the mapping flag can be implemented by either `signaled_subpic_id_flag` indicating whether the subpic's ID is signaled, or `subpic_mapping_enabled_flag` indicating whether subpic mapping is allowed. When `subpic_mapping_enabled_flag=0`, it means that the mapping information defined in SPS is used, while when `subpic_mapping_enabled_flag=1`, it means that the mapping information defined in PPS is used.

[0245] Alternatively, the mapping flags can be implemented by including `sps_subpic_id_present_flag` indicating whether ID mapping is performed in SPS, `sps_subpic_id_signalling_present_flag` indicating whether mapping information is signaled in SPS, and `pps_subpic_id_signalling_present_flag` indicating whether mapping information is signaled in PPS. In this case, `pps_subpic_id_signalling_present_flag` can be signaled when `sps_subpic_id_present_flag=1` and `sps_subpic_id_signalling_present_flag=0`.

[0246] When the mapping flag indicates the presence of a subpic_id (i.e., indicating the application of mapping), the video encoding device can encode the subpic_id and signal it. When the mapping flag indicates the presence of a subpic_id, the video decoding device can decode the subpic_id (S1760). The video decoding device can decode as many subpic_ids as there are subpics or slices.

[0247] According to the implementation method, in addition to subpic_id, the mapping information may also include the length information of subpic_id (signalled_subpic_id_length_minus1 or subpic_id_len_minus1).

[0248] When the mapping flag indicates the presence of mapping information (S1740), the length information can be signaled from the video encoding device and decoded by the video decoding device (S1750).

[0249] According to the implementation method, whether to perform the processing of signal notification mapping information can be determined based on whether there is a sub-screen in the screen and whether the slice type is a rectangular slice.

[0250] The video decoding device can decode the subpic_info_present_flag and rect_slice_flag notified by the video encoding device via signals (S1710) and determine whether a subpic exists and whether the slice type is a rectangular slice based on the subpic_info_present_flag and rect_slice_flag (S1720). When a subpic exists and the slice type is a rectangular slice, the above-mentioned mapping information signaling processing can be performed (S1730 to S1760); however, when no subpic exists or the slice type is not a rectangular slice, the mapping information signaling processing may not be performed.

[0251] Tables 14 and 15 illustrate two syntactic structures for Implementation 4. Table 14 shows the signaling and decoding of subpic_ids, which are as many as the number of subpics. Table 15 shows the signaling and decoding of subpic_ids, which are as many as the number of slices.

[0252] [Table 14]

[0253] [Table 15]

[0254] The decoded subpic_ids can be arranged in a specific order, as many as the number of subpicks or as many as the number of slices. Here, the specific order can be a reference. Figure 16 The described raster scan order or slice configuration order.

[0255] Based on the examples in Table 14, when Figure 11 The slice of (b) is mapped to Figure 12When the subpic of (b) is a subpic, subpic_mapping_enabled_flag is equal to true(1) and mapping is performed in the order of subpic_id={A, B, C, D} (raster scan order). Based on the example in Table 15, when Figure 11 The slice of (b) is mapped to Figure 12 When the subpic of (b) is a subpic, subpic_mapping_enabled_flag is equal to true(1) and mapping is performed in the order of subpic_id={A, B, B, C, D, C} (raster scan order).

[0256] Based on the examples in Table 14, when Figure 16 Slices mapped to Figure 12 When the subpic of (b) is a subpic, subpic_mapping_enabled_flag is equal to true(1) and mapping is performed in the order of subpic_id={A, B, C, D} (raster scan order). Based on the example in Table 15, when Figure 16 Slices mapped to Figure 12 When the subpic of (b) is a subpic, subpic_mapping_enabled_flag is equal to true(1) and mapping is performed in the order of subpic_id={A, B, B, C, C, D} (slice configuration order).

[0257] Implementation Method 5

[0258] Implementation 5 presents the syntactic structures and semantics defined in the slice header. Tables 16 and 17 show two syntactic structures of Implementation 5.

[0259] [Table 16]

[0260] [Table 17]

[0261] slice_pic_parameter_set_id indicates the PPS ID referenced by the relevant slice, and slice_pic_header_id indicates the image header ID referenced by the relevant slice.

[0262] As shown in Table 16, the video encoding device can use signals to notify the subpic ID (subpic_id), slice address information (slice_address), and slice type (slice_type). To distinguish the subpic_id (or slice_subpic_id) notified by signals in the slice header from the subpic_id (sps_subpic_id or pps_subpic_id) notified by signals in SPS or PPS, the subpic_id notified by signals in SPS or PPS can be referred to as "first mapping information," and the subpic_id notified by signals in the slice header can be referred to as "second mapping information." In other words, the "mapping information" mentioned in this disclosure can include both first and second mapping information.

[0263] The subpic_id in the slice header indicates the ID of the subpic to which the relevant slice belongs. When a subpic exists (when subpic_info_present_flag=1, i.e., when the segmentation flag indicates a division) (Table 16), or when a subpic exists (subpic_info_present_flag=1) and the slice type is a rectangular slice (rect_slice_flag=1) (Table 17), the subpic ID can be indicated by a signal.

[0264] When no subpic ID (first mapping information) is signaled in SPS or PPS, the subpic ID signaled in the slice header (second mapping information) can refer to the subpic's index value. In other words, when the subpic flag is off in SPS (e.g., signaled_subpic_id_flag=0) or off in PPS (e.g., subpic_mapping_enabled_flag=0), no subpic ID is signaled in SPS or PPS; in this case, the subpic ID signaled in the slice header can correspond to the subpic's index value, such as... Figure 12 As shown in (a).

[0265] Conversely, when the sub-frame ID is signaled in SPS or PPS, the sub-frame ID signaled in the slice header can correspond to the sub-frame ID value, such as... Figure 12 As shown in (b). In this case, the video decoding device can map the ID of the sub-frame signaled from the SPS or PPS to the ID of the sub-frame signaled from the slice header.

[0266] Furthermore, as shown in Tables 16 and 17, the video encoding device can encode the slice address information (slice_address) and notify it with a signal, and the configuration unit 940 can use the address information to set the slice index value.

[0267] Address information indicates the address of the relevant slice. Address information can have different meanings depending on the values ​​of the subpic_info_present_flag, the type flag (rect_slice_flag), and / or partition permission information (e.g., no_partition_flag).

[0268] First, when subpic_info_present_flag=1 (S1810), the address information can be the index value of the slice in the subpicture to which the relevant slice belongs (S1820). For example, such as Figure 12 As shown in (b), when a sub-screen exists, the address information can be the slice index value of the sub-screen to which the relevant slice belongs.

[0269] When subpic_info_present_flag=0 (S1810) and rect_slice_flag=1 (S1830), the address information can be the index value of the slice within the entire frame (S1840). For example, as... Figure 11 (b) or Figure 16 In this context, when a sub-screen does not exist, the address information can be the slice index value of the entire screen.

[0270] When `subpic_info_present_flag=0`, no signal is used to notify information about the number of subpics (`num_subpics_in_pic_minus1`). When no information about the number of subpics exists, the number of subpics is inferred to be 1 (`num_subpics_in_pic_minus1=0`). As a result, when `subpic_info_present_flag=0` (S1810) and `rect_slice_flag=1` (S1830), the address information can be "the index value of the slice within the entire subpic." In this text, "subpic" can refer to the entire picture.

[0271] When `subpic_info_present_flag=0` (S1810) and `rect_slice_flag=0` (S1830), in other words, in the case of raster scan slices, the address information can be either the tile index or the CTU index (S1860, S1870). When the raster scan slice is designated as a region at a specific location, and tile encoding / decoding is performed in the raster scan order, the address information can be the tile index (S1860). On the other hand, when the raster scan slice is designated as a region at a specific location, and CTU encoding / decoding is performed in the raster scan order (without tile division), the address information can be the CTU index (S1870).

[0272] According to the implementation method, whether the address information corresponds to the index value of the block or the index value of the CTU can be determined based on the value of the partition permission information.

[0273] For example, when the screen is divided into one or more tiles (no_partition_flag=0, S1850), the address information can be the index value of the tile (S1860), and when the screen is not divided (no_partition_flag=1, S1850), the address information can be the index value of the CTU (S1870).

[0274] Figure 19 The relationship between sub-pictures and slices based on the method proposed in this disclosure is shown.

[0275] exist Figure 19 In (a), each of the six rectangles represents a slice, and a slice consists of a total of four sub-pictures, as shown below. Figure 12 As in example (b), sub-pictures are distinguished by different patterns. The puzzle consists of a total of 6 (3×2) pieces, as shown... Figure 11 The partitioning format of (a) is like that.

[0276] Since there are a total of six slices, therefore... Figure 19 As shown in (b), slices are generated from a total of six NAL bitstreams, and the frame can be divided into four sub-frames.

[0277] Figure 19 The syntactic element values ​​of the six slice heads in example (b) are as follows.

[0278] #1: (subpic_id=A, slice_address=0)

[0279] #2: (subpic_id=B, slice_address=0)

[0280] #3: (subpic_id=B, slice_address=1)

[0281] #4: (subpic_id=C, slice_address=0)

[0282] #5: (subpic_id=C, slice_address=1)

[0283] #6: (subpic_id=D, slice_address=0)

[0284] Figure 20 The diagram shows the connections between the slice NAL, the header NAL, the PPS NAL, and the SPS NAL.

[0285] Video decoding devices can use the pps_id and pic_header_id of the slice header to connect the slice header data with the PPS data and the picture header data. When the slice header includes sps_id, data can be directly connected from the slice header to the SPS.

[0286] The video decoding device can obtain sub-picture mapping information from SPS, tile partitioning information and slice configuration information from PPS, and sub-picture mapping information from the video header.

[0287] The receiving end (video decoding device) can be divided into a BEAMer for synthesizing content and a decoder for decoding the synthesized data. The BEAMer combines multiple pieces of content into a single synthesized data by extracting and synthesizing them. The synthesized data corresponds to the input data fed into the decoder.

[0288] The following will describe various sub-screen configuration examples. These examples can be divided into cases where the sub-screen configuration remains unchanged and cases where the sub-screen configuration changes.

[0289] 1) Figure 21 (c) shows an example where the sub-screen configuration remains unchanged.

[0290] from Figure 19 (b) Sub-screen configuration Figure 11 The block partitioning format of (a) and Figure 19 The slice configuration of (a) can ultimately be derived. Figure 21 The two contents shown in (a) and (b) are distinguishable in different directions. Figure 21 The content of (a) and Figure 21 The content of (b).

[0291] BEAMer can combine two pieces of content to generate a combined piece of content, such as Figure 21 As shown in (c). In response to Figure 21 The content of (a) and Figure 21 After receiving all the data in (b), BEAMer can extract a total of six slice bitstreams from all the received data as follows and input the received data into the decoder.

[0292] 1) Figure 21 (b) A-0 slice bitstream of content

[0293] 2) Figure 21 (a) B-0 slice bitstream of content

[0294] 3) Figure 21 (a) B-1 slice bitstream of content

[0295] 4) Figure 21 (a) C-0 slice bitstream of content

[0296] 5) Figure 21 (a) C-1 slice bitstream of content

[0297] 6) Figure 21 (b) D-0 slice bitstream of content

[0298] For example, it can be obtained from Figure 21 Content extraction of (b) Figure 21 Sub-pictures A and D of (c) can be composited, and can be obtained from Figure 21 Content extraction of (a) Figure 21 Sub-pictures B and C of (c) are composited.

[0299] In addition to the slice bitstream, header information such as SPS and PPS is input to the decoder. Additionally, a frame header indicating the mapping information of sub-frames can also be input to the decoder. In this case, Figure 21 The syntactic value of the header in example (c) can be compared with... Figure 16 or Figure 11 The syntactic value of (b) is the same.

[0300] 2) Figure 21 (d) shows an example of a change in sub-screen configuration. Figure 21 (d) shows the situation where the sub-screen configuration changes while the two content items are merged.

[0301] BEAMer can combine two pieces of content to generate a combined piece of content, such as Figure 21 As shown in (d). In response to Figure 21 The content of (a) and Figure 21After receiving all the data in (b), BEAMer can extract a total of six slice bitstreams from all the received data as follows and input the extracted data into the decoder.

[0302] 1) Figure 21 (b) D-0 slice bitstream of content

[0303] 2) Figure 21 (b) C-0 slice bitstream of content

[0304] 3) Figure 21 (b) C-1 slice bitstream of content

[0305] 4) Figure 21 (a) B-0 slice bitstream of content

[0306] 5) Figure 21 (a) B-1 slice bitstream of content

[0307] 6) Figure 21 (a) A-0 slice bitstream of content

[0308] For example, it can be obtained from Figure 21 Content extraction of (b) Figure 21 Sub-pictures D and C of (d) can be composited, and can be obtained from Figure 21 Content extraction of (a) Figure 21 Sub-pictures B and A of (d) are composited.

[0309] When the sub-picture configuration changes, the mapping information of the changed sub-picture should be defined in the picture header and notified by a signal. For example, the sending end (video encoding device) can generate and notify the picture header information by a signal, or the receiving end (video decoding device) can generate the picture header information.

[0310] When the sub-picture configuration changes, in addition to the slice bitstream, header information such as SPS and PPS is also input to the decoder. Additionally, a picture header indicating the sub-picture's mapping information can also be input to the decoder. In this case, Figure 21 The syntax values ​​for the image header of example (d) are as follows.

[0311] As shown in Table 14, the signal notifies that the number of subpic_ids is the same as the number of subpics, and that subpic_mapping_enabled_flag=true(1), and that subpic_id={D, C, B, A} is in raster scan order. As shown in Table 15, the signal notifies that the number of subpic_ids is the same as the number of slices, and that subpic_mapping_enabled_flag=true(1), and that subpic_id={D, C, C, B, A, B} is in raster scan order. As shown in Table 15, the signal notifies that the number of subpic_ids is the same as the number of slices, and that subpic_mapping_enabled_flag=true(1), and that subpic_id={D, C, C, B, B, A} is in slice configuration order.

[0312] As mentioned above, it can be achieved through Figure 21 (d) Confirmation allows for easy representation of sub-screen configurations in the screen header without altering SPS, PPS, slice header, and / or slice data information.

[0313] Although exemplary embodiments have been described for illustrative purposes, those skilled in the art will understand that various modifications and changes can be made without departing from the concept and scope of the embodiments. Exemplary embodiments have been described for the sake of brevity and clarity. Therefore, those skilled in the art will understand that the scope of the embodiments is not limited to those expressly described above, but includes the claims and their equivalents.

[0314] Cross-references to related applications

[0315] This application claims priority to Korean Patent Application No. 10-2019-0123858, filed on October 7, 2019, and Korean Patent Application No. 10-2020-0032688, filed on March 17, 2020, the entire contents of which are incorporated herein by reference.

Claims

1. A video decoding method for segmenting a frame, the video decoding method comprising the following steps: Decode a segmentation flag from the bitstream indicating whether the frame has been divided into one or more sub-units; When the segmentation flag indicates a segmentation, a slice flag indicating one of multiple slice types is decoded from the bitstream, wherein the multiple slice types include rectangular slice type and raster scan slice type; and When the segmentation flag indicates a division and when the slice flag indicates the rectangular slice type, the partitioning information related to dividing the screen into multiple slices is decoded from the bitstream. The partitioning information includes information about the number of slices and the size of the slices. When a slice is divided into multiple tiles, the size information of the slice includes information about the number of tile columns included in the slice and information about the number of tile rows included in the slice.

2. The video decoding method according to claim 1, wherein, When the segmentation flag indicates a division, the partitioning information includes information about the number of tiles and the size of the tiles.

3. The video decoding method according to claim 1, further comprising the following steps: Decode mapping information from the bitstream, the mapping information being used to map one or more sub-pictures to one or more slices belonging to each of the sub-pictures; and The sub-screen is mapped to the slice based on the mapping information.

4. The video decoding method according to claim 3, further comprising the following steps: The bitstream decoding indicates whether a mapping flag is applied to the image. Wherein, when the mapping flag indicates the application of mapping, the mapping information includes first mapping information decoded from the SPS or PPS of the bitstream.

5. The video decoding method according to claim 3, wherein, When the segmentation flag indicates a segmentation, the mapping information includes second mapping information decoded from the slice header of the bitstream.

6. The video decoding method according to claim 1, further comprising the following steps: Decode the address information of the target slice from the slice header of the bitstream; as well as The index of the target slice within the sub-screen containing the target slice is set based on the address information. Wherein, when the slice flag decoded from the bitstream indicates the rectangular slice type, the address information indicates the index of the slice within the frame, and When the slice flag indicates the raster scan slice type, the address information indicates the index of the tile included in the image.

7. A video encoding method, the video encoding method comprising the following steps: Encode the segmentation markers that indicate whether the screen is divided into one or more sub-units; When the segmentation marker indicates division, the slice marker indicating one of multiple slice types is encoded, wherein the multiple slice types include rectangular slice type and raster scan slice type; and When the segmentation flag indicates a division and when the slice flag indicates the rectangular slice type, the partitioning information related to dividing the screen into multiple slices is encoded. The partitioning information includes information about the number of slices and the size of the slices. When a slice is divided into multiple tiles, the size information of the slice includes information about the number of tile columns included in the slice and information about the number of tile rows included in the slice.

8. A method for transmitting a bitstream including encoded video data, the method comprising the steps of: The bitstream is generated by encoding video data; as well as The bitstream is sent to the video decoding device. The steps for generating the bitstream include: Encode the segmentation markers that indicate whether the screen is divided into one or more sub-units; When the segmentation marker indicates division, the slice marker indicating one of multiple slice types is encoded, wherein the multiple slice types include rectangular slice type and raster scan slice type; and When the segmentation flag indicates a division and when the slice flag indicates the rectangular slice type, the partitioning information related to dividing the screen into multiple slices is encoded. The partitioning information includes information about the number of slices and the size of the slices. When a slice is divided into multiple tiles, the size information of the slice includes information about the number of tile columns included in the slice and information about the number of tile rows included in the slice.

Images