Method and apparatus for encoding and decoding video using sub-image partitions

Abstract

The present invention relates to a method and an apparatus for encoding and decoding a video using sub-image partitions. A method for processing a bitstream is provided in which an image sequence divided into a plurality of sub-images is encoded. The method comprises the steps of: decoding, from a bitstream, division information indicating a division structure in which an image belonging to a sequence is divided into sub-images; decoding ID information related to the sub-images, and mapping an ID to each of the sub-images by using the ID information; and reconstructing the block within the at least one sub-image by using the mapped ID.

Description

[0001] This application is a divisional application of the PCT patent application entered into China with Chinese patent application number 202080056612.6, invention name “Method and device for encoding and decoding video using sub-image partitioning”, and application date August 10, 2020.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to Korean Patent Application No. 10-2019-0097448, filed on August 9, 2019, and Korean Patent Application No. 10-2019-0120805, filed on September 30, 2019, which are hereby incorporated by reference in their entirety. Technical Field

[0004] The present invention relates to encoding and decoding of video, and more particularly, to partitioning each image into independently displayable sub-images and encoding and decoding each sub-image. Background Art

[0005] Since the amount of video data is larger than that of voice data or still image data, storing or transmitting the video data without compression processing requires a large amount of hardware resources (including memory).

[0006] Accordingly, when storing or transmitting video data, an encoder is usually used to compress the video data for storage or transmission. Then, a decoder receives the compressed video data, decompresses and reproduces the video data. Compression technologies used for such videos include H.264 / AVC and High Efficiency Video Coding (HEVC), which has an approximately 40% improvement in coding efficiency over H.264 / AVC.

[0007] However, the size, resolution and frame rate of video images are gradually increasing, so the amount of data to be encoded is also increasing. Accordingly, a new compression technology with better encoding efficiency and higher image quality than existing compression technologies is needed.

[0008] Furthermore, due to the emergence of various applications such as 360 video, a technology of displaying not only the entire area of ​​a decoded image but also a partial area of ​​the image is required. Summary of the invention

[0009] Technical issues

[0010] The present invention relates to a technique for partitioning each image into sub-images that can be displayed independently of each other, and a technique for encoding and decoding each sub-image.

[0011] Technical Solution

[0012] According to one aspect of the present invention, there is provided a method for processing a bitstream, the bitstream being generated by encoding an image sequence partitioned into a plurality of sub-images. The method comprises: decoding partition information indicating a partition structure from the bitstream, in which the images belonging to the sequence are partitioned into sub-images; decoding ID information of the sub-images, and mapping an ID to each sub-image by using the ID information; and reconstructing a block within at least one sub-image by using the mapped ID.

[0013] The partition structure defined by the partition information may be the same for all images in a sequence.The ID information may be structured to allow mapping of different IDs to co-located sub-images within an image belonging to a sequence.

[0014] According to another aspect of the present invention, a video encoding method is provided for generating a bit stream by encoding an image sequence partitioned into a plurality of sub-images. The method comprises: encoding partition information indicating a partition structure in which images belonging to the sequence are partitioned into sub-images; encoding ID information of the sub-images; and encoding a block within at least one sub-image by using the mapped ID.

[0015] The partition structure defined by the partition information may be the same for all images in the sequence.The ID information is constructed to allow mapping of different IDs to co-located sub-images within images belonging to the sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is an exemplary block diagram of a video encoding device capable of implementing the techniques of the present invention.

[0017] Figure 2 is a schematic diagram showing block partitioning using the QTBTTT structure.

[0018] Figure 3 is a schematic diagram showing multiple intra prediction modes.

[0019] Figure 4 is an exemplary block diagram of a video decoding device capable of implementing the techniques of the present invention.

[0020] Figure 5 is an exemplary diagram showing the structure of a bit stream.

[0021] Figure 6 is an exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0022] Figure 7is another exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0023] Figure 8 is another exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0024] Fig. 9 , Fig.10a and Fig.10b is a further exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0025] Fig.11 is a further exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0026] Fig.12 is an exemplary view showing a reference structure of each sublayer.

[0027] Fig.13 is another exemplary view showing a reference structure of each sub-layer. DETAILED DESCRIPTION

[0028] Hereinafter, some embodiments of the present invention will be described in detail with reference to the attached drawings. It should be noted that when assigning reference numerals to constituent elements in the various drawings, the same reference numerals represent the same elements even though the elements are shown in different drawings. In addition, in the following description of the present invention, the detailed description of the known functions and configurations incorporated herein will be omitted to avoid blurring the subject matter of the present invention.

[0029] Figure 1 is an exemplary block diagram of a video encoding device capable of implementing the technology of the present invention. Figure 1 A video encoding apparatus and elements of the apparatus are described below.

[0030] The video encoding apparatus includes an image divider 110 , a predictor 120 , a subtractor 130 , a transformer 140 , a quantizer 145 , a reorganizer 150 , an entropy encoder 155 , an inverse quantizer 160 , an inverse transformer 165 , an adder 170 , a loop filter unit 180 , and a memory 190 .

[0031] Each element of the video encoding device may be implemented in hardware or software, or a combination of hardware and software. The functions of each element may be implemented as software, and a microprocessor may be implemented to execute the software functions corresponding to each element.

[0032] A video includes multiple images. Each image is divided into multiple regions, and encoding is performed on each region. For example, an image is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more coding tree units (coding treeunit, CTU). Each CTU is divided into one or more coding units (coding unit, CU) through a tree structure. The information applied to each CU is encoded as the syntax of the CU, and the information commonly applied to the CU included in a CTU is encoded as the syntax of the CTU. In addition, the information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and the information applied to all blocks constituting one or more images is encoded in the picture parameter set (Picture Parameter Set, PPS) or the picture header. In addition, the information commonly referenced by a sequence consisting of multiple images is encoded in the sequence parameter set (Sequence Parameter Set, SPS). The information commonly applied to a tile or tile group can be encoded as the syntax of the tile header or tile group header.

[0033] The image divider 110 determines the size of a coding tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as a syntax of an SPS or a PPS and transmitted to a video decoding apparatus.

[0034] The image divider 110 divides each image constituting a video into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. In the tree structure, a leaf node is used as a coding unit (CU), which is a basic unit of coding.

[0035] The tree structure can be a quadtree (QuadTree, QT), a binary tree (BinaryTree, BT), a ternary tree (TernaryTree, TT), or a structure formed by a combination of two or more QT structures, BT structures and TT structures, wherein the quadtree (QT) is a node (or parent node) that is divided into four slave nodes (or child nodes) of the same size, the binary tree (BT) is a node that is divided into two slave nodes, and the ternary tree (TT) is a node that is divided into three slave nodes at a ratio of 1:2:1. For example, a quadtree plus binary tree (QuadTree plus BinaryTree, QTBT) structure can be used, or a quadtree plus binary tree ternary tree (QuadTree plus BinaryTree TernaryTree, QTBTTT) structure can be used. Here, BTTT can be collectively referred to as a multiple-type tree (MTT).

[0036] Figure 2 The QTBTTT segmentation tree structure is shown as an example. Figure 2 As shown, the CTU can be first split into a QT structure. The QT splitting can be repeated until the size of the split block reaches the minimum block size MinQTSize of the leaf node allowed in the QT. The first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder 155 and notified to the video decoding device by a signal. When the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it can be further split into one or more BT structures or TT structures. The BT structure and / or the TT structure can have multiple splitting directions. For example, there can be two directions, namely, a direction of splitting the blocks of nodes horizontally and a direction of splitting the blocks vertically. As Figure 2 As shown, when the MTT splitting starts, the second flag (mtt_split_flag) indicating whether the node is split, the flag indicating the splitting direction (vertical or horizontal) in the case of splitting, and / or the flag indicating the splitting type (binary or trifurcated) are encoded by the entropy encoder 155, and the signal is notified to the video decoding device. Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into 4 nodes of the lower layer, the CU split flag (split_cu_flag) indicating whether the node is split can be encoded. When the value of the CU split flag (split_cu_flag) indicates that the splitting is not performed, the block of the node becomes a leaf node in the split tree structure and is used as a coding unit (CU), which is a basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that the splitting is performed, the video encoding device starts to encode the flags from the first flag in the above-mentioned manner.

[0037] When QTBT is used as another example of a tree structure, there may be two types of segmentation, namely, a type of horizontally segmenting a block into two blocks of the same size (i.e., symmetrical horizontal segmentation) and a type of vertically segmenting a block into two blocks of the same size (i.e., symmetrical vertical segmentation). A segmentation flag (split_flag) indicating whether each node of the BT structure is segmented into blocks of the lower layer and segmentation type information indicating the segmentation type are encoded by the entropy encoder 155 and transmitted to the video decoding device. There may be an additional type of segmenting a block of a node into two asymmetric blocks. An asymmetric segmentation type may include a type of segmenting a block into two rectangular blocks at a size ratio of 1:3, or a type of segmenting a block of a node diagonally.

[0038] A CU may have various sizes according to the QTBT or QTBTTT partitioning of a CTU. Hereinafter, a block corresponding to a CU to be encoded or decoded (i.e., a leaf node of a QTBTTT) is referred to as a "current block". When QTBTTT partitioning is adopted, the shape of the current block may be a square or a rectangle.

[0039] The predictor 120 predicts the current block to generate a predicted block. The predictor 120 includes an intra predictor 122 and an inter predictor 124.

[0040] The intra-frame predictor 122 predicts pixels in the current block using pixels (reference pixels) located around the current block in the current image including the current block. There are multiple intra-frame prediction modes according to the prediction direction. For example, Figure 3 As shown, the plurality of intra prediction modes may include two non-directional modes and 65 directional modes, the two non-directional modes including a planar mode and a DC mode. Neighboring pixels and equations to be used are defined differently for each prediction mode.

[0041] The intra-frame predictor 122 may determine an intra-frame prediction mode to be used when encoding the current block. In some examples, the intra-frame predictor 122 may encode the current block using several intra-frame prediction modes and select an appropriate intra-frame prediction mode to be used from the tested modes. For example, the intra-frame predictor 122 may calculate a rate-distortion value using a rate-distortion analysis of several tested intra-frame prediction modes and may select an intra-frame prediction mode having the best rate-distortion characteristic from the tested modes.

[0042] The intra predictor 122 selects an intra prediction mode from a plurality of intra prediction modes and predicts the current block using adjacent pixels (reference pixels) and equations determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0043] The inter-frame predictor 124 generates a prediction block of the current block by motion compensation. The inter-frame predictor 124 searches for a block most similar to the current block in a reference image that has been encoded and decoded earlier than the current image, and uses the searched block to generate a prediction block of the current block. Then, the inter-frame predictor generates a motion vector corresponding to the displacement between the current block in the current image and the prediction block in the reference image. Usually, motion estimation is performed on the luminance (luma) component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chrominance component. The motion information including information about the reference image and information about the motion vector for predicting the current block is encoded by the entropy encoder 155 and transmitted to the video decoding device. The inter-frame predictor 124 can perform interpolation on the reference image or reference block to increase the accuracy of the prediction. That is, sub-pixels are interpolated between two consecutive integer pixels by applying filter coefficients to a plurality of consecutive integer pixels including two integer pixels. When the process of searching for the block most similar to the current block is performed for the interpolated reference image, the motion vector can be expressed not with integer pixel precision but with fractional unit precision. The precision or resolution of the motion vector can be set differently for each unit of the target area to be encoded, for example, a slice, a tile, a CTU, or a CU.

[0044] The subtractor 130 subtracts the prediction block generated by the intra predictor 122 or the inter predictor 124 from the current block to generate a residual block.

[0045] The transformer 140 may divide the residual block into one or more sub-blocks and apply a transform to the one or more sub-blocks, thereby transforming the residual value of the transform block from the pixel domain to the frequency domain. In the frequency domain, the transform block is referred to as a coefficient block or a transform block containing one or more transform coefficient values. A two-dimensional transform kernel may be used for the transform, while a one-dimensional transform kernel may be used for horizontal transform and vertical transform, respectively. The transform kernel may be based on a discrete cosine transform (DCT), a discrete sine transform (DST), etc.

[0046] The transformer 140 may transform the residual signal in the residual block using the entire size of the residual block as a transform unit. Alternatively, the residual block may be partitioned into a plurality of sub-blocks, and the residual signal in the sub-block may be transformed using the sub-block as a transform unit.

[0047] The transformer 140 may transform the residual block separately in the horizontal direction and the vertical direction. For the transformation, various types of transformation functions or transformation matrices may be used. For example, a pair of transformation functions for transforming in the horizontal direction and the vertical direction may be defined as a multiple transform set (MTS). The transformer 140 may select a pair of transformation functions with the best transformation efficiency in the MTS, and transform the residual block in the horizontal direction and the vertical direction, respectively. The information (mts_idx) about the transformation function pair selected from the MTS is encoded by the entropy encoder 155 and notified to the video decoding device with a signal.

[0048] The quantizer 145 quantizes the transform coefficients output from the transformer 140 using the quantization parameters, and outputs the quantized transform coefficients to the entropy encoder 155. For some blocks or frames, the quantizer 145 may directly quantize the relevant residual block without transformation. The quantizer 145 may apply different quantization coefficients (scaling values) according to the position of the transform coefficients in the transform block. The quantization coefficient matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to the video decoding device.

[0049] The reorganizer 150 can reorganize the coefficient values ​​of the quantized residual values. The reorganizer 150 can change the 2D coefficient array into a 1D coefficient sequence by coefficient scanning. For example, the reorganizer 150 can scan the coefficients from the DC coefficient to the coefficients in the high frequency region using a zig-zag scan or a diagonal scan to output a 1D coefficient sequence. According to the size of the transform unit and the intra-frame prediction mode, a vertical scan or a horizontal scan can be used instead of a zig-zag scan, wherein the vertical scan scans the 2D coefficient array in the column direction, and the horizontal scan scans the coefficients of the 2D block shape in the row direction. That is, the scanning mode to be used can be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan according to the size of the transform unit and the intra-frame prediction mode.

[0050] The entropy encoder 155 encodes the one-dimensional quantized transform coefficient output from the reassembler 150 using various encoding techniques such as Context-based Adaptive Binary Arithmetic Code (CABAC) and exponential Golomb to generate a bitstream.

[0051] The entropy encoder 155 encodes information related to block segmentation (e.g., CTU size, CU segmentation flag, QT segmentation flag, MTT segmentation type, and MTT segmentation direction) so that the video decoding device can segment the block in the same manner as the video encoding device. In addition, the entropy encoder 155 encodes information about the prediction type indicating whether the current block is encoded by intra-frame prediction or inter-frame prediction, and encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (information about the reference image index and motion vector) according to the prediction type. In addition, the entropy encoder 155 encodes information related to quantization, that is, information about quantization parameters and information about quantization matrices.

[0052] The inverse quantizer 160 inversely quantizes the quantized transform coefficient output from the quantizer 145 to generate a transform coefficient. The inverse transformer 165 transforms the transform coefficient output from the inverse quantizer 160 from the frequency domain to the spatial domain, and reconstructs a residual block.

[0053] The adder 170 adds the reconstructed residual block and the prediction block generated by the predictor 120 to reconstruct the current block. Pixels in the reconstructed current block are used as reference pixels when performing intra prediction of a subsequent block.

[0054] The loop filter unit 180 filters the reconstructed pixels to reduce blocking artifacts, ringing artifacts, and blurring artifacts caused by block-based prediction and transform / quantization. The loop filter unit 180 may include one or more of a deblocking filter 182, a sample adaptive offset (SAO) filter 184, or an adaptive loop filter (ALF) 186.

[0055] The deblocking filter 182 filters the boundaries between the reconstructed blocks to remove block artifacts caused by block-by-block encoding / decoding, and the SAO filter 184 performs additional filtering on the deblock filtered video. The SAO filter 184 is a filter for compensating for the difference between the reconstructed pixels and the original pixels caused by lossy encoding, and performs filtering in a manner of adding a corresponding offset to each reconstructed pixel. The ALF 186 performs filtering on the target pixel to be filtered by applying a filter coefficient to the target pixel and the neighboring pixels of the target pixel. The ALF 186 can divide the pixels included in the image into predetermined groups, and then determine a filter to be applied to the corresponding group to perform filtering differentially on each group. Information about the filter coefficient to be used for the ALF can be encoded and notified to the video decoding device with a signal.

[0056] The reconstructed blocks filtered by the loop filtering unit 180 are stored in the memory 190. Once all blocks in one image are reconstructed, the reconstructed image may be used as a reference image for inter-frame prediction of blocks in a subsequent image to be encoded.

[0057] Figure 4 is an exemplary functional block diagram of a video decoding device capable of implementing the technology of the present invention. Figure 4 A video decoding apparatus and elements of the apparatus are described below.

[0058] The video decoding apparatus may include an entropy decoder 410 , a reorganizer 415 , an inverse quantizer 420 , an inverse transformer 430 , a predictor 440 , an adder 450 , a loop filtering unit 460 , and a memory 470 .

[0059] Similar to Figure 1 In the video encoding device of the present invention, each element of the video decoding device can be implemented by hardware, software, or a combination of hardware and software. In addition, the function of each element can be implemented by software, and the microprocessor can be implemented to execute the software function corresponding to each element.

[0060] The entropy decoder 410 determines a current block to be decoded by decoding a bit stream generated by a video encoding device and extracting information related to block partitioning, and extracts prediction information required to reconstruct the current block, information about a residual signal, and the like.

[0061] The entropy decoder 410 extracts information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), determines the size of the CTU, and divides the image into CTUs of the determined size. Then, the decoder determines the CTU as the highest level (i.e., the root node) of the tree structure, and extracts partition information about the CTU to partition the CTU using the tree structure.

[0062] For example, when the CTU is segmented using the QTBTTT structure, the first flag (QT_split_flag) related to the segmentation of the QT is extracted to segment each node into four nodes of the sublayer. For the node corresponding to the leaf node of the QT, the second flag (MTT_split_flag) related to the segmentation of the MTT and the information about the segmentation direction (vertical / horizontal) and / or the segmentation type (binary / trifurcated) are extracted to segment the corresponding leaf node with the MTT structure. Thus, each node below the leaf node of the QT is recursively segmented with the BT or TT structure.

[0063] As another example, when a CTU is split using a QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU is split may be extracted. When the corresponding block is split, a first flag (QT_split_flag) may be extracted. In a split operation, after zero or more recursive QT splits, zero or more recursive MTT splits may occur for each node. For example, a CTU may directly undergo MTT splitting without undergoing QT splitting, or may only undergo QT splitting multiple times.

[0064] As another example, when the CTU is segmented using the QTBT structure, the first flag (QT_split_flag) related to the QT segmentation is extracted, and each node is segmented into four nodes of the lower layer. Then, the segmentation flag (split_flag) indicating whether the node corresponding to the leaf node of the QT is further segmented with the BT and the segmentation direction information are extracted.

[0065] Once the current block to be decoded is determined by partitioning through the tree structure, the entropy decoder 410 extracts information about the prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 410 extracts syntax elements of intra-prediction information (intra-prediction mode) of the current block. When the prediction type information indicates inter-prediction, the entropy decoder 410 extracts syntax elements for inter-prediction information, that is, information indicating a motion vector and a reference image referenced by the motion vector.

[0066] The entropy decoder 410 also extracts information about the quantized transform coefficient of the current block as information related to quantization and information about the residual signal.

[0067] The reorganizer 415 may change a sequence of one-dimensional quantized transform coefficients entropy-decoded by the entropy decoder 410 into a 2-dimensional coefficient array (ie, block) in the reverse order of coefficient scanning performed by the video encoding apparatus.

[0068] The inverse quantizer 420 inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantizer 420 may apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantizer 420 may perform inverse quantization by applying a quantization coefficient (scaling value) matrix from a video encoding device to the two-dimensional quantized transform coefficient array.

[0069] The inverse transformer 430 inversely transforms the inversely quantized transform coefficient from the frequency domain to the spatial domain to reconstruct the residual signal, thereby generating a reconstructed residual block of the current block. In addition, when MTS is applied, the inverse transformer 430 determines the transform function or transform matrix to be applied in the horizontal direction and the vertical direction, respectively, using the MTS information (mts_idx) signaled from the video encoding device, and inversely transforms the transform coefficient in the transform block in the horizontal direction and the vertical direction using the determined transform function.

[0070] The predictor 440 may include an intra predictor 442 and an inter predictor 444. The intra predictor 442 is activated when the prediction type of the current block is intra prediction, and the inter predictor 444 is activated when the prediction type of the current block is inter prediction.

[0071] The intra predictor 442 determines an intra prediction mode of a current block among a plurality of intra prediction modes based on a syntax element of the intra prediction mode extracted from the entropy decoder 410 , and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0072] The inter predictor 444 determines a motion vector of a current block and a reference image referred to by the motion vector using a syntax element of the intra prediction mode extracted from the entropy decoder 410 , and predicts the current block based on the motion vector and the reference image.

[0073] The adder 450 reconstructs the current block by adding the residual block output from the inverse transformer to the prediction block output from the inter-frame predictor or the intra-frame predictor. Pixels in the reconstructed current block are used as reference pixels when intra-predicting a subsequent block to be decoded.

[0074] The loop filtering unit 460 may include at least one of a deblocking filter 462, an SAO filter 464, or an ALF 466. The deblocking filter 462 performs deblocking filtering on the boundaries between the reconstructed blocks to remove block artifacts caused by block-by-block decoding. The SAO filter 464 performs filtering in a manner of adding the reconstructed blocks after deblocking filtering the corresponding offsets so as to compensate for the difference between the reconstructed pixels and the original pixels caused by lossy coding. The ALF 466 performs filtering on the target pixel to be filtered by applying the filter coefficients to the target pixel and the neighboring pixels of the target pixel. The ALF 466 can divide the pixels in the image into predetermined groups, and then determine a filter to be applied to the corresponding group to perform filtering differentially on each group. The filter coefficients of the ALF are determined based on information about the filter coefficients decoded from the bitstream.

[0075] The reconstructed blocks filtered by the loop filtering unit 460 are stored in the memory 470. When all blocks in one image are reconstructed, the reconstructed image is used as a reference image for inter-frame prediction of blocks in a subsequent image to be encoded.

[0076] As described above, the video encoding device transmits a bit stream containing encoded data about the video, and the video decoding device decodes the bit stream to reconstruct each image constituting the video. The bit stream may be composed of a plurality of transmission units, namely, the network abstraction layer (NAL) unit. Figure 5 As shown, the NAL unit may include a NAL unit header and data carried by the NAL unit. In order to match the size of the NAL unit in bytes, a bit may be added to the rear of the data constituting the NAL unit. The NAL unit header includes a NAL unit type for indicating the type of data carried by the NAL unit and a temporal layer ID of the NAL unit.

[0077] According to the type of data included in the data field, NAL units can be classified into video coding layer (VCL) type and non-VCL type. A VCL type NAL unit is a NAL unit that contains image data of the coded pixel group in the data field, and generally contains data about the coded slice. This means that the unit in which the image data is transmitted is a slice. A non-VCL type NAL unit includes parameters required for decoding the data of the pixel group in the data field. NAL units including high-level syntax, such as an SPS including parameters shared at the sequence level (hereinafter referred to as a "first parameter set") or a PPS including parameters shared at one or more picture levels (hereinafter referred to as a "second parameter set"), correspond to non-VCL type NAL units. Whether the transmitted NAL unit is related to the first parameter set, the second parameter set, or the coded slice is indicated by the NAL unit type contained in the NAL unit header.

[0078] With the emergence of various applications such as 360 video, a technology is needed to display not only the entire area of ​​a decoded image but also a partial area of ​​an image. In order to support this technical requirement, another aspect of the present invention provides a method for partitioning an image into a plurality of sub-images and encoding and decoding each sub-image. The present invention is capable of independently encoding or independently transmitting the sub-images constituting each image. In addition, data corresponding to each of the sub-images can be independently extracted or decoded from the entire bit stream. Further, the sub-images from the partitions of the image can be displayed independently of each other.

[0079] The video encoding device may encode a flag indicating whether sub-image partitioning is allowed in the video sequence. If the flag indicates that sub-image partitioning is allowed, the video encoding device encodes information indicating the layout of the sub-image. The video decoding device may identify whether sub-image partitioning is allowed in the video sequence by decoding the flag, and may determine the layout of the sub-image within the image by decoding the layout information.

[0080] To represent the layout of a sub-image partitioned from an image, a coding unit group consisting of one or more coding units (CUs) may be defined. Depending on the purpose and size, a coding unit group may correspond to a grid, CTU, slice, or tile of a predefined size. A sub-image is defined by one or more coding unit groups.

[0081] Figure 6 is an exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0082] The coding unit group can be classified into a first coding unit group and a second coding unit group, wherein the first coding unit group is a basic unit of an image or a sub-image, and the second coding unit group is composed of multiple first coding unit groups. For example, the first coding unit group can be a CTU or a grid of a predefined size, and the second coding unit group can be a slice corresponding to a transmission unit or a tile or tile group defined for independent encoding / decoding. A sub-image can be composed of one second coding unit group or multiple second coding unit groups.

[0083] The video encoding device can use a signal to notify the size of the first coding unit group (for example, the horizontal length or vertical length of the first coding unit group), the number of first coding unit groups in the horizontal direction and / or vertical direction of the image, etc., and the video decoding device can identify the size of the first coding unit group and the size of the image by utilizing the information notified by the signal.

[0084] The layout information of the sub-images within the image may be represented as partition information indicating a partition structure for partitioning the image into sub-images, and information indicating an ID or index of the sub-images within the image. The partition information for indicating the partition structure may be represented based on the first coding group. That is, the partition information may be represented by the size of the first coding unit group (which is a basic unit of the sub-image) (e.g., the width and height of the first coding unit group) and the number of first coding unit groups in the horizontal and vertical directions of the sub-image. In addition, the partition information may further include identification information for identifying the first coding unit group located at the upper left and / or lower right of each sub-image.

[0085] Figure 7 is another exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0086] Figure 7 It can correspond to the case where a second coding unit group constitutes a sub-image. Figure 7 In the example, a grid corresponds to a first coding unit group, and a second coding unit group (which is a group of grids) corresponds to a sub-image.

[0087] As in Figure 6 In this example, the partition information may further include the size of the grid (the grid is the basic unit of the sub-image), the number of grids in the horizontal and vertical directions of the sub-image, and identification information for identifying the grid located at the upper left and / or lower right of the sub-image.

[0088] Alternatively, the partition information may be expressed as an ID or address of a grid constituting a sub-image. To this end, the ID or address of the grid within the image may be assigned in ascending order starting from 0 according to a specific scanning order, such as a raster scanning order.

[0089] In the above, the partition structure of the sub-image has been defined based on the basic unit of rectangle. However, the partition structure of the sub-image can be defined in various forms.

[0090] Figure 8 is another exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0091] In this example, a plurality of partition types are predefined, by which the image is partitioned into sub-blocks. The video encoding device may signal an index indicating any one of the plurality of predefined partition types, and the video decoding device may determine the partition type by the partition type by which the image is partitioned into sub-blocks through the index received from the video encoding device.

[0092] Fig. 9 , Fig.10a and Fig.10b is a further exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0093] In this example, the basic unit of the sub-image can have various shapes. Fig. 9 As shown, the basic unit may be a rectangle or a triangle. The video encoding device signals the index indicating the basic unit shape to be used among the plurality of basic unit shapes to the video decoding device. In addition, the video encoding device signals information about the size (M and N) of the basic unit shape to be used.

[0094] The sub-image partition structure may be represented by at least one of the index of the first basic unit of the sub-image, the index of the last basic unit, and the shape of the sub-image. To this end, the index of the basic unit may be assigned according to a predefined scanning order. For example, the scanning order may be as follows: Fig.10a and Fig.10b Raster scan order shown.

[0095] like Fig.10a As shown, if the shape of the basic unit is rectangular and the shape of the sub-image is rectangular, the structure of the sub-image can be defined only by the index of the last basic unit of each sub-image. Fig.10a , the first sub-image may be defined by the value of index K, and the second sub-image may be defined by the value of index J. The video encoding device notifies the video decoding device of information about the K value and the J value with a signal. The information about the J value may be represented by a difference value from the K value. The video decoding device may obtain the J value by using the transmitted difference value and the K value.

[0096] Although the drawings show a raster scanning order in which the index increases sequentially from the upper left to the lower right, other scanning orders may be used. In addition, the index may be represented by two values ​​in the horizontal direction (x-axis direction) and the vertical direction (y-axis direction). In this case, each sub-image is defined by the two-dimensional coordinates of the basic unit. For example, in Fig.10a In the image, each sub-image is defined by an index of 4 in the horizontal direction and an index of 2 in the vertical direction, i.e., (4, 2). The video encoding device signals the horizontal and vertical indices of the last basic unit of each sub-image within the image to the video decoding device.

[0097] Even when the sub-images have shapes other than rectangles, the sub-image partition structure can be defined by the index of the last basic unit within each sub-image. Figure 8 The partition type 5 of the embodiment of the present invention partitions the image in such a way that the sub-image partition structure can be defined by the index of the last basic unit of each sub-image. For example, the current sub-image is composed of such basic units, the index of which is greater than the index of the last basic unit corresponding to the previous sub-image and equal to or less than the index of the last basic unit corresponding to the current sub-image.

[0098] Meanwhile, in the above-described example, if the number of sub-images within an image is signaled, index information of a basic unit corresponding to the last sub-image within the image may be omitted.

[0099] When the shape of the basic unit is Fig.10b When the triangle is shown, the sub-image partition structure can be defined in a manner similar to the above method.

[0100] Fig.11 is a further exemplary view for explaining a partition structure for partitioning an image into sub-images.

[0101] In this example, the sub-image partition structure is defined by utilizing the position information of pixels within the image rather than the basic unit of the sub-image. If the image is partitioned into six rectangular sub-images, such as Fig.11 As shown in , the partition structure can be defined only by the coordinate information (a and b) of the pixel. If there is a restriction that the size of the sub-image must be a multiple of n, the video encoding device can transmit the value obtained by dividing the pixel coordinate value by n. In other words, the values ​​corresponding to a / n and b / n are notified by signal as the coordinate information (a and b) of the pixel. The video decoding device can calculate a and b by multiplying the received value by n. In some embodiments, the coordinates of b can be transmitted in the form of a difference from the coordinates of a.

[0102] The video encoding device transmits information about the partition structure (partition information) and ID information of the sub-image within the image to the video decoding device as layout information. The video decoding device determines the layout of the sub-image within the image by using the received layout information. The video decoding device can extract the sub-image ID transmitted on a per-slice basis from the video encoding device, thereby finding the sub-image including the corresponding slice. In addition, through the above-mentioned decoding process, the sub-image is reconstructed by decoding the blocks within one or more slices constituting each sub-image.

[0103] As described above, each of the multiple images constituting a video sequence can be partitioned into multiple sub-images, and only some sub-images within each image belonging to a sequence can be extracted and encoded / decoded. In addition, only these sub-images can be displayed. Accordingly, the present invention defines such a sub-sequence or sub-layer, which consists only of sub-images at the same position or with the same ID within each image belonging to a video sequence.

[0104] Since a sub-layer is required to constitute a piece of content, it is desirable that the sub-images belonging to the sub-layer are related to each other and that the sub-images have the same size. Accordingly, all images constituting a video sequence may have the same sub-image partition structure, and co-located sub-images within an image may have the same ID. However, in the case of applications such as 360-degree video streaming based on a user's viewport, the ID of a sub-image may change within a video sequence depending on the user's viewing direction. Accordingly, sub-images with the same ID may be located at different positions or have different sizes in order to support such applications. In other words, co-located sub-images within an image belonging to a sequence may have different IDs. Therefore, the ID information may be signaled in such a way as to allow different IDs to be mapped to co-located sub-images within an image belonging to a sequence.

[0105] Hereinafter, a technique for signaling layout information, that is, partition information and / or ID information indicating a partition structure will be described.

[0106] Information for describing the layout of a sub-image may be signaled at a sequence level. That is, partition information indicating a sub-image partition structure may be signaled at a sequence level. For example, one or more of the following methods may be used: a method of signaling the partition information in a sequence parameter set (SPS), a method of repeatedly signaling for each image group or image belonging to a sequence, and a method of signaling a syntax element (e.g., a flag) indicating whether the partition structure is applied equally to all images belonging to the corresponding sequence. For example, a syntax element indicating that the partition structure is applied equally to all images belonging to the corresponding sequence may be signaled, and accordingly, the partition information may be signaled in a sequence parameter set. In this case, the video decoding device decodes the partition information from the sequence parameter set, and partitions all images belonging to the sequence by the same partition structure.

[0107] The ID information of the sub-image within the image may be signaled in the same manner as the partition information. For example, a syntax element indicating that the ID mapping is applied equally to all images belonging to the corresponding sequence may be signaled, and accordingly, the ID information may be signaled in the sequence parameter set. In this case, the video decoding apparatus decodes the ID information and maps the same ID to the co-located sub-image within the image.

[0108] At this time, a different sub-image layout may be used for each image group belonging to the sequence. Each image group includes one or more images, and images belonging to the same image group have the same layout. In this case, each piece of information used to describe the layout of the sub-image may be signaled at the image group level. For example, one or more of the following methods may be used: a method of signaling about a partition structure in a picture parameter set (PPS), a method of signaling information about a partition structure for each image belonging to an image group, and a method of signaling a syntax element (e.g., a flag) indicating whether a partition structure applied at the image group level is applied to all images belonging to the corresponding image group in the same manner.

[0109] For example, a syntax element indicating that a partition structure (or ID mapping) is applied equally to all images belonging to the corresponding picture group may be signaled, and the partition information (or ID information) may be signaled in a picture parameter set. In this case, the video decoding apparatus partitions all images belonging to the picture group by the same partition structure (or maps the same ID to the co-located sub-images within the images belonging to the picture group). Accordingly, images belonging to different picture groups within a video sequence (i.e., images referring to different picture parameter sets) may have different partition structures. In addition, different IDs may be mapped to co-located sub-images within images belonging to different picture groups.

[0110] When a syntax element included in the SPS indicates that a partition structure or an ID mapping is not commonly applied to images belonging to a corresponding sequence, partition information or ID information may be signaled at a picture group level. For example, when a syntax element at a sequence level indicates that a partition structure is not equally applied to images belonging to a corresponding sequence, partition information is signaled at a picture group level. Similarly, when a syntax element at a sequence level indicates that an ID mapping is not equally applied to a sequence, ID information is signaled at a picture group level.

[0111] When receiving a bit stream containing sub-image layout information (partition information indicating a sub-image partition structure and sub-image ID information) from a video encoding device, the video decoding device can construct a sub-layer by extracting sub-images at the same position or with the same ID using the received layout information.

[0112] In some embodiments of the present invention, when receiving a bitstream having a coded sequence of images, a video decoding device may decode and manage the bitstream independently for each sublayer. For example, each sublayer may be decoded in an independent decoding order of the sublayer, and decoded sub-images may be managed for each sublayer. To this end, a different decoded picture buffer (DPB) may be used for each sublayer.

[0113] Sub-images may be managed by using image IDs and sub-image IDs. Sub-images included in the same image may have the same image ID, while sub-images belonging to different sub-layers may have different sub-image IDs. By using image IDs and sub-image IDs, each sub-layer may be configured to have an independent reference structure between sub-images and may be independently managed by a separate DPB or a separate display buffer. Sub-images may be individually added to at least one of a DPB managed on a per-sub-layer basis or a DPB managed on a per-image basis or deleted from at least one of a DPB managed on a per-sub-layer basis or a DPB managed on a per-image basis, and may be output for display. The video decoding device may simultaneously display sub-images constituting one image by using image IDs, or may display each sub-image individually by using sub-image IDs or information (e.g., sub-image positions) that may be derived from sub-image IDs.

[0114] Sublayers can have independent reference structures. Correspondingly, the reference structure can vary between sublayers.

[0115] Fig.12 is an exemplary view showing a reference structure of each sublayer.

[0116] For example, Fig.12As shown, for two sub-layers, the display order and / or decoding order of random access point (RAP) sub-images may be the same, while for each sub-layer, the decoding order and / or display order of sub-images other than the random access point sub-images may be different. When displaying an image composed of a plurality of sub-images belonging to different sub-layers, the display of the image may be delayed in consideration of the reference structure between the sub-images of each sub-layer and / or the encoding / decoding order of the sub-images. For example, when simultaneously displaying all sub-images included in each image, starting from an image having a specific image ID, the video decoding device may delay the display until all sub-images having the same image ID as the random access point sub-image, which is closest to the image having the specific image ID in the display order, have been decoded.

[0117] Fig.13 is another exemplary view showing a reference structure of each sub-layer.

[0118] like Fig.13 As shown, for each sub-layer, the decoding order and / or display order of the random access point (RAP) sub-picture and other sub-pictures may vary. Fig.13 In the example, the interval between random access point sub-images of sub-layer #1 is three sub-images, while the interval between random access point sub-images of sub-layer #2 is five sub-images. In this example, when displaying an image composed of a plurality of sub-images belonging to different sub-layers, the reference structure between sub-images of each sub-layer and / or the encoding / decoding order of the sub-images are considered.

[0119] As an example, when all sub-images included in each image are displayed simultaneously, starting from an image having a specific image ID, the video decoding apparatus may delay display until all sub-images having the same image ID as a random access point sub-image that is closest in display order to the image having the specific image ID have been decoded. Fig.13 In the example, according to the display order, the decoded sub-image No. 2 of sub-layer #1 shares the same picture ID as the decoded sub-image No. 4 of sub-layer #2. The two sub-layers have different decoding orders according to their reference structures. Therefore, the display of the random access point pictures starting from the corresponding picture ID should be delayed until the decoded sub-image No. 4 of sub-layer #2 has been decoded. In this way, when multiple sub-layers are decoded in different reference structures, the display can be delayed until a picture has been completely decoded.

[0120] As another example, the picture ID to start displaying can be derived by utilizing the interval between random access point sub-pictures of each sub-layer. Fig.13In the example, the spacing between random access point sub-images for sub-layer #1 is three sub-images, while the spacing for sub-layer #2 is five sub-images. Therefore, for every 15 sub-images, two sub-layers can have random access point sub-images with the same picture ID. In some embodiments, if the two sub-layers have different reference structures, the picture-level random access points can be restricted to where the sub-images of both sub-layers become random access points at the same time. Information about the spacing between the random access sub-images of the two layers can be derived from the spacing between them.

[0121] In other embodiments of the present invention, the bitstream transmitted by the video encoding device can be partitioned into a plurality of sub-bitstreams corresponding to the sub-layers respectively. That is, the sub-bitstream corresponding to a specific sub-layer can be extracted from the bitstream transmitted by the video encoding device. The video decoding device can decode the sub-bitstream corresponding to the specific sub-layer. In this case, some syntax elements in the high-level syntax structure (e.g., SPS or PPS) of the sub-bitstream can be created or modified by utilizing the high-level syntax elements in the bitstream. For example, since the sub-bitstream is data created by encoding a sub-image, the syntax element for the image size to be included in the sub-bitstream should be modified to indicate the size of the sub-image. The size of the sub-image can be calculated by utilizing the sub-image layout information included in the bitstream transmitted by the video encoding device. The syntax element of the image size to be included in the sub-bitstream is modified to the calculated size of the sub-image.

[0122] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions described in one or more examples can be implemented as hardware, software, firmware or any combination thereof. The functional components described in this specification are marked as units to more particularly emphasize their implementation independence.

[0123] On the other hand, the various methods or functions described in the present invention can be implemented as instructions stored in a non-volatile recording medium, which can be read and executed by one or more processors. Non-volatile recording media include, for example, all types of recording devices that store data in a form that can be read by a computer system. For example, non-volatile recording media include storage media such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSD).

[0124] Although exemplary embodiments have been described for illustrative purposes, it will be appreciated by those skilled in the art that various modifications and variations are possible without departing from the spirit and scope of the embodiments. For the sake of brevity and clarity, exemplary embodiments have been described. Accordingly, it will be appreciated by those of ordinary skill that the scope of the embodiments is not limited to the embodiments explicitly described above, but is included in the claims and their equivalents.

Claims

1. An apparatus for processing a bitstream, the bitstream being generated by encoding a sequence of images, each image being partitioned into a plurality of sub-images, the apparatus comprising at least one processor, the at least one processor being configured to: Partition information indicating a partition structure in which pictures belonging to the sequence are partitioned into sub-pictures is decoded from a bitstream at a sequence level, wherein The partition structure defined by the partition information is the same for all images in the sequence; Decoding the ID information of the sub-image to map the ID to each sub-image; as well as reconstructing a block within at least one sub-image corresponding to the mapped ID, The ID information is structured to allow different IDs to be mapped to co-located sub-images having the same position in different images belonging to the sequence, The partition information includes information for identifying a basic unit located at the upper left or lower right of each sub-image.

2. The apparatus for processing a bit stream according to claim 1, wherein: The at least one processor is further configured to decode information about a basic unit size of each sub-image, The partition information includes the number of basic units in the horizontal direction and the vertical direction of each sub-image.

3. The apparatus for processing a bit stream according to claim 1, wherein: The partition information is included in a sequence parameter set referenced by the sequence.

4. The apparatus for processing a bit stream according to claim 3, wherein: ID mapping includes: Decoding a syntax element indicating whether ID mapping is applied at the sequence level; If the syntax element indicates that ID mapping is applied on a sequence level, decoding ID information from a sequence parameter set and mapping the same ID to a co-located sub-picture within a picture belonging to the sequence; and If the syntax element indicates that ID mapping is not applied on sequence level, the ID information is decoded from the picture parameter set and the same ID is mapped to the co-located sub-picture within the picture referencing the picture parameter set in the sequence.

5. The apparatus for processing a bit stream according to claim 1, wherein: The at least one processor is further configured to: extracting from the bitstream a sub-bitstream corresponding to a sub-image at the same position or having the same ID within an image in the sequence; and The high-level syntax of the sub-bitstream is modified by utilizing the high-level syntax included in the bitstream.

6. An apparatus for encoding a sequence of images, each image being partitioned into a plurality of sub-images, the apparatus comprising at least one processor, the at least one processor being configured to: Partition information indicating a partition structure in which images belonging to a sequence are partitioned into sub-images is encoded into a bitstream at the sequence level. The partition structure defined by the partition information is the same for all images in the sequence; Encode the ID information of the sub-image; as well as encoding a block within at least one sub-image corresponding to the mapped ID, The ID information is structured to allow different IDs to be mapped to co-located sub-images having the same position in different images belonging to the sequence, The partition information includes information for identifying a basic unit located at the upper left or lower right of each sub-image.

7. The apparatus for encoding an image sequence according to claim 6, wherein: The at least one processor is further configured to encode information about a basic unit size of each sub-image, The partition information includes the number of basic units in the horizontal direction and the vertical direction of each sub-image.

8. The apparatus for encoding an image sequence according to claim 6, wherein: The partition information is included in a sequence parameter set referenced by the sequence.

9. The apparatus for encoding an image sequence according to claim 6, wherein: At least one processor is configured to encode the ID information by the following process, the process comprising: Encodes a syntax element indicating whether ID mapping is applied at the sequence level; If the syntax element indicates that ID mapping is applied at the sequence level, encoding the ID information into the sequence parameter set; and If the syntax element indicates that ID mapping is not applied on sequence level, the ID information is encoded in the picture parameter set.

10. An apparatus for providing a bitstream comprising video data to a video decoding apparatus, the apparatus comprising at least one processor, the at least one processor being configured to: A bitstream is generated by encoding a sequence of images, each image being partitioned into a plurality of sub-images; Sending a bitstream to a video decoding device, in, The at least one processor is configured to generate a bitstream by the following process, the process comprising: encoding partition information indicating a partition structure in which images belonging to the sequence are partitioned into sub-images into a bitstream at a sequence level, wherein the partition structure defined by the partition information is the same for all images in the sequence; Encoding the ID information of the sub-image; and encoding a block within at least one sub-image corresponding to the mapped ID, The ID information is structured to allow different IDs to be mapped to co-located sub-images having the same position in different images belonging to the sequence, The partition information includes information for identifying a basic unit located at the upper left or lower right of each sub-image.

11. The apparatus for providing a bit stream comprising video data to a video decoding apparatus according to claim 10, wherein: The at least one processor is configured to encode the ID information by the following process, the process comprising: Encodes a syntax element indicating whether ID mapping is applied at the sequence level; If the syntax element indicates that ID mapping is applied at the sequence level, the ID information is encoded into the sequence parameter set; If the syntax element indicates that ID mapping is not applied at the sequence level, the ID information is encoded in the picture parameter set.

Images