Method and Apparatus for Encoding and Decoding Video

Abstract

The present invention relates to the encoding and decoding of images, and more specifically, to a method for dividing each picture into independently displayable subpictures and encoding and decoding each subpicture.

Description

Technology Field

[0001] The present invention relates to the encoding and decoding of images, and more specifically to the division of each picture into independently displayable subpictures and the encoding and decoding of each subpicture. Background Technology

[0002] Because video data contains a large amount of data compared to audio or still image data, storing or transmitting it as is without compression processing requires significant hardware resources, including memory.

[0003] Therefore, typically, when storing or transmitting video data, an encoder is used to compress the video data for storage or transmission, and a decoder receives the compressed video data, decompresses it, and plays it. Such video compression technologies include H.264 / AVC, as well as HEVC (High Efficiency Video Coding), which improves coding efficiency by about 40% compared to H.264 / AVC.

[0004] However, as video size, resolution, and frame rates are gradually increasing, and the amount of data that needs to be encoded is also growing accordingly, a new compression technology is required that offers better encoding efficiency and higher image quality improvement effects than existing compression technologies.

[0005] In addition, due to the emergence of various applications such as 360 video, there is a demand for technology that not only displays the entire area of ​​a decoded picture but also displays only a part of the picture. The problem to be solved

[0006] The present disclosure aims to provide a technique for dividing each picture into subpictures that can be displayed independently of each other, and a technique for encoding and decoding each subpicture. means of solving the problem

[0007] One aspect of the present disclosure provides a video decoding method for decoding a bitstream encoded with a sequence of pictures divided into a plurality of subpictures, each containing a plurality of coding tree blocks. The method comprises: decoding layout information of the subpictures divided from the picture included in the sequence from the bitstream; determining a target block by dividing a coding tree block to be decoded into a tree structure within one of the subpictures identified by the layout information; decoding prediction information for predicting the target block and information regarding residual signals of the target block from the bitstream; generating a prediction block by predicting pixels within the target block based on the prediction information; generating a residual block for the target block using information regarding the residual signals; and restoring the target block by adding the prediction block and the residual block.

[0008] Another aspect of the present disclosure provides an image decoding device for decoding a bitstream encoded with a sequence of pictures divided into a plurality of subpictures, each containing a plurality of coding tree blocks. The device comprises: a decoding unit that decodes layout information of the subpictures divided from the picture included in the sequence from the bitstream, determines a target block by dividing a coding tree block to be decoded into a tree structure within one of the subpictures identified by the layout information, and decodes prediction information for predicting the target block and information regarding residual signals of the target block from the bitstream; a prediction unit that generates a prediction block by predicting pixels within the target block based on the prediction information; a residual recovery unit that generates a residual block for the target block using information regarding the residual signals; and an adder that recovers the target block by adding the prediction block and the residual block.

[0009] Here, the sub-pictures constituting the picture are units that can be displayed independently of each other. Brief explanation of the drawing

[0010] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the technologies of the present disclosure. Figure 2 is a diagram illustrating a method for dividing blocks using a QTBTTT structure. Figure 3 is a diagram showing multiple intra prediction modes. FIG. 4 is an exemplary block diagram of an image decoding device capable of implementing the technologies of the present disclosure. Figure 5 is an example diagram showing the structure of a bitstream. Figure 6 is an example diagram showing the structure of a NAL unit including a third parameter set. Figure 7 is an example diagram illustrating the layout of the sub-pictures that make up each picture. Figure 8 is an example diagram illustrating the layout of the sub-pictures that make up each picture. Figure 9 is an example diagram illustrating a method for processing overlapping subpictures. Specific details for implementing the invention

[0011] Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that in assigning identification symbols to the components of each drawing, the same components are assigned the same symbol whenever possible, even if they are shown in different drawings. Furthermore, in describing the present invention, if it is determined that a detailed description of related known components or functions could obscure the essence of the invention, such detailed description is omitted.

[0012] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the technologies of the present disclosure. Hereinafter, the image encoding device and its sub-components will be described with reference to FIG. 1.

[0013] The video encoding device may be configured to include a block division unit (110), a prediction unit (120), a subtractor (130), a conversion unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse conversion unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0014] Each component of the video encoding device may be implemented in hardware or software, or as a combination of hardware and software. Additionally, the function of each component may be implemented in software, and a microprocessor may be implemented to execute the software function corresponding to each component.

[0015] A single image (video) consists of multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles or / and slices. Here, one or more tiles can be defined as a tile group. Each tile or / slice is divided into one or more Coding Tree Units (CTUs). And each CTU is divided into one or more Coding Units (CUs) by a tree structure. Information applicable to each CU is encoded as the CU syntax, and information applicable to all CUs included in a single CTU is encoded as the CTU syntax. Additionally, information applicable to all blocks within a single slice is encoded as the slice header syntax, and information applicable to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or picture header. Furthermore, information commonly referenced by a sequence composed of multiple pictures is encoded in a Sequence Parameter Set (SPS). Additionally, information commonly applied to a single tile or tile group may be encoded as the syntax of a tile or tile group header.

[0016] The block division unit (110) determines the size of the Coding Tree Unit (CTU). Information regarding the size of the CTU (CTU size) is encoded as a syntax of SPS or PPS and transmitted to an image decoder.

[0017] The block division unit (110) divides each picture constituting the image into multiple Coding Tree Units (CTUs) having a predetermined size, and then recursively divides the CTUs using a tree structure. The leaf nodes in the tree structure become the coding units (CUs) which are the basic units of coding.

[0018] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of equal size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that combines two or more of these QT, BT, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT combined may be referred to as an MTT (Multiple-Type Tree).

[0019] FIG. 2 shows a QTBTTT splitting tree structure. As seen in FIG. 2, the CTU can first be split into a QT structure. Quadtree splitting can be repeated until the size of the splitting block reaches the minimum block size of the leaf node allowed in QT (MinQTSize). A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder (155) and signaled to the image decoder. If the leaf node of the QT is not larger than the maximum block size of the root node allowed in BT (MaxBTSize), it can be further split into one or more of the BT structure or TT structure. In the BT structure and / or TT structure, multiple splitting directions may exist. For example, there may be two directions in which the block of the corresponding node is split horizontally and vertically. As shown in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether the nodes have been split, a flag indicating the splitting direction (vertical or horizontal) and / or the splitting type (binary or ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. Alternatively, a CU splitting flag (split_cu_flag) indicating whether the node is split may be encoded prior to encoding a first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer. If the value of the CU splitting flag (split_cu_flag) indicates that the node has not been split, the block of the node becomes a leaf node in the splitting tree structure and becomes a coding unit (CU), which is the basic unit of encoding.When the CU split flag (split_cu_flag) value indicates splitting, the video encoding device starts encoding from the first flag in the manner described above.

[0020] When QTBT is used as another example of a tree structure, there may be two types: a type that divides the block of the corresponding node horizontally into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that divides it vertically (i.e., symmetric vertical splitting). A splitting flag (split_flag) indicating whether each node of the BT structure is split into a block of a lower layer and splitting type information indicating the type of splitting are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may also be an additional type that divides the block of the corresponding node into two blocks of an asymmetric shape. The asymmetric shape may include a shape that divides the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that divides the block of the corresponding node diagonally.

[0021] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from a CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of QTBTTT) is referred to as the 'current block'. Depending on the adoption of QTBTTT partitioning, the shape of the current block may be not only square but also rectangular.

[0022] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0023] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture containing the current block. Multiple intra prediction modes exist depending on the prediction direction. For example, as shown in FIG. 3a, multiple intra prediction modes may include two non-directional modes, including a planar mode and a DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently for each prediction mode.

[0024] The intra prediction unit (122) can determine the intra prediction mode to use for encoding the current block. In some examples, the intra prediction unit (122) may encode the current block using several intra prediction modes and select an appropriate intra prediction mode to use from the tested modes. For example, the intra prediction unit (122) may calculate rate-distortion values ​​using rate-distortion analysis of several tested intra prediction modes and select an intra prediction mode that has the best rate-distortion features among the tested modes.

[0025] The intra prediction unit (122) selects one intra prediction mode among a plurality of intra prediction modes and predicts the current block using a calculation formula and surrounding pixels (reference pixels) determined according to the selected intra prediction mode. Information regarding the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0026] The inter prediction unit (124) generates a prediction block for the current block through a motion compensation process. The inter prediction unit (124) searches for the block most similar to the current block within a reference picture that is encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, it generates a motion vector corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture. Generally, motion estimation is performed on the luminance component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chroma component. Motion information, including information on the reference picture used to predict the current block and information on the motion vector, is encoded by the entropy encoding unit (155) and transmitted to the image decoding device. The inter prediction unit (124) may also perform interpolation on the reference picture or reference block to increase the accuracy of the prediction. That is, subpixels between two consecutive integer pixels are interpolated by applying filter coefficients to multiple consecutive integer pixels that include those two integer pixels. By performing a search for the block most similar to the current block on the interpolated reference picture, the motion vector can be expressed with precision in fractional units rather than in integer pixel units. The precision or resolution of the motion vector can be set differently for each unit of the target area to be encoded, such as slices, tiles, CTUs, CUs, etc.

[0027] The subtractor (130) generates a residual block by subtracting the prediction block generated by the intra prediction unit (122) or the inter prediction unit (124) from the current block.

[0028] The transformation unit (140) divides the residual block into one or more sub-blocks and applies a transformation to one or more sub-blocks to transform the residual values ​​of the transformation blocks from the pixel domain to the frequency domain. In the frequency domain, the transformed blocks are referred to as coefficient blocks or transform blocks containing one or more transformation coefficient values. A two-dimensional transformation kernel may be used for the transformation, and a one-dimensional transformation kernel may be used for the horizontal direction transformation and the vertical direction, respectively. The transformation kernel may be based on the Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), etc.

[0029] The conversion unit (140) can convert residual signals within the residual block using the entire size of the residual block as the conversion unit. Alternatively, the residual block may be divided into multiple sub-blocks, and the residual signals within the sub-blocks may be converted using the sub-blocks as the conversion unit.

[0030] Meanwhile, the transformation unit (140) can perform transformations on the residual block individually in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices may be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one pair of transformation functions with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) regarding the selected pair of transformation functions among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoder.

[0031] The quantization unit (145) quantizes the transformation coefficients output from the transformation unit (140) using quantization parameters and outputs the quantized transformation coefficients to the entropy encoding unit (155). The quantization unit (145) may quantize the associated residual block directly without transformation for any block or frame. The quantization unit (145) may apply different quantization coefficients (scaling values) depending on the position of the transformation coefficients within the transformation block. A matrix of quantization coefficients applied to the quantized transformation coefficients arranged in two dimensions can be encoded and signaled to an image decoder.

[0032] The reordering unit (150) can perform reordering of coefficient values ​​for quantized residual values. The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence through coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from DC coefficients to coefficients in the high-frequency range using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra-prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction, or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction, may be used instead of a zig-zag scan. That is, depending on the size of the transformation unit and the intra-prediction mode, the scanning method to be used among a zig-zag scan, a diagonal scan, a vertical scan, and a horizontal scan may be determined.

[0033] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transformation coefficients output from the reordering unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0034] Additionally, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoder can divide the block in the same way as the image encoding unit. Additionally, the entropy encoding unit (155) encodes information regarding a prediction type indicating whether the current block is encoded by intra prediction or by inter prediction, and encodes intra prediction information (i.e., information about the intra prediction mode) or inter prediction information (information about the reference picture and motion vector) according to the prediction type. Additionally, the entropy encoding unit (155) encodes information related to quantization, i.e., information about quantization parameters and information about the quantization matrix.

[0035] The inverse quantization unit (160) inversely quantizes the quantized transformation coefficients output from the quantization unit (145) to generate transformation coefficients. The inverse transformation unit (165) converts the transformation coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0036] The adder (170) restores the current block by adding the restored residual block and the prediction block generated by the prediction unit (120). The pixels within the restored current block are used as reference pixels when intra-predicting the next block in sequence.

[0037] The loop filter unit (180) performs filtering on the restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. caused by block-based prediction and transformation / quantization. The loop filter unit (180) may include one or more of a deblocking filter (182), a Sample Adaptive Offset (SAO) filter (184), and an Adaptive Loop Filter (ALF, 186).

[0038] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-unit encoding / decoding, and the SAO filter (184) performs additional filtering on the deblocking filtered image. The SAO filter (184) is a filter used to compensate for the difference between the restored pixel and the original pixel caused by lossy coding, and is performed by adding to each restored pixel with a corresponding offset. The ALF (186) performs filtering on the target pixel by applying filter coefficients to the target pixel and the surrounding pixels of the target pixel. The ALF (186) can divide the pixels included in the image into a predetermined group and determine one filter to be applied to that group to perform filtering differentially for each group. Information regarding the filter coefficients to be used in the ALF can be encoded and signaled to an image decoder.

[0039] The restored blocks filtered through the loop filter section (180) are stored in memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0040] FIG. 4 is an exemplary functional block diagram of an image decoding device capable of implementing the technologies of the present disclosure. Hereinafter, the image decoding device and its sub-components will be described with reference to FIG. 4.

[0041] The image decoding device may be configured to include an entropy decoding unit (410), a reordering unit (415), an inverse quantization unit (420), an inverse transformation unit (430), a prediction unit (440), an adder (450), a loop filter unit (460), and a memory (470).

[0042] Similar to the image encoding device of FIG. 1, each component of the image decoding device may be implemented in hardware or software, or in combination of hardware and software. Additionally, the function of each component may be implemented in software, and a microprocessor may be implemented to execute the function of the software corresponding to each component.

[0043] The entropy decoding unit (410) determines the current block to be decoded by decoding the bitstream generated by the video encoding device and extracting information related to block division, and extracts prediction information and information about residual signals necessary to restore the current block.

[0044] The entropy decoding unit (410) extracts information about the CTU size from the SPS (Sequence Parameter Set) or PPS (Picture Parameter Set) to determine the size of the CTU and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and divides the CTU using the tree structure by extracting division information for the CTU.

[0045] For example, when splitting a CTU using a QTBTTT structure, first, a first flag (QT_split_flag) related to QT splitting is extracted to split each node into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, a second flag (MTT_split_flag) related to MTT splitting and splitting direction (vertical / horizontal) and / or splitting type (binary / ternary) information are extracted to split the corresponding leaf nodes into an MTT structure. Through this, each node below the leaf nodes of QT is recursively split into a BT or TT structure.

[0046] As another example, when splitting a CTU using the QTBTTT structure, a CU splitting flag (split_cu_flag) indicating whether to split the CU is first extracted, and if the block is split, a first flag (QT_split_flag) is extracted. During the splitting process, each node may undergo zero or more iterative MTT splittings after zero or more iterative QT splittings. For example, the CTU may undergo MTT splitting immediately, or conversely, only multiple QT splittings may occur.

[0047] As another example, when splitting a CTU using a QTBT structure, a first flag (QT_split_flag) related to the splitting of QT is extracted to split each node into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and split direction information are extracted.

[0048] Meanwhile, when the entropy decoding unit (410) determines the current block to be decoded through the division of the tree structure, it extracts information regarding the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-predicted, the entropy decoding unit (410) extracts a syntax element for the intra-predicted information (intra-predicted mode) of the current block. If the prediction type information indicates inter-predicted, the entropy decoding unit (410) extracts a syntax element for the inter-predicted information, that is, information indicating a motion vector and a reference picture that the motion vector refers to.

[0049] Additionally, the entropy decoder (410) extracts information regarding the quantized transformation coefficients of the current block as information related to quantization and information regarding the residual signal.

[0050] The reordering unit (415) can change the sequence of one-dimensional quantized transformation coefficients entropy-decoded in the entropy decoding unit (410) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning order performed by the image encoding device.

[0051] The inverse quantization unit (420) inversely quantizes the quantized transformation coefficients using quantization parameters. The inverse quantization unit (420) may apply different quantization coefficients (scaling values) to the quantized transformation coefficients arranged in two dimensions. The inverse quantization unit (420) may perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transformation coefficients.

[0052] The inverse transform unit (430) generates a restored residual block for the current block by inversely transforming the inversely quantized transform coefficients from the frequency domain to the spatial domain and restoring the residual signals. Additionally, when MTS is applied, the inverse transform unit (430) determines a transform function or transform matrix to be applied in the horizontal and vertical directions, respectively, using MTS information (mts_idx) signaled from the video encoding device, and performs an inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.

[0053] The prediction unit (440) may include an intra prediction unit (442) and an inter prediction unit (444). The intra prediction unit (442) is activated when the prediction type of the current block is an intra prediction, and the inter prediction unit (444) is activated when the prediction type of the current block is an inter prediction.

[0054] The intra prediction unit (442) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax element for the intra prediction mode extracted from the entropy decoding unit (410), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0055] The inter prediction unit (444) determines the motion vector of the current block and the reference picture that the motion vector refers to using the syntax element for the intra prediction mode extracted from the entropy decoding unit (410), and predicts the current block using the motion vector and the reference picture.

[0056] The adder (450) restores the current block by adding the residual block output from the inverse transformation unit and the prediction block output from the inter prediction unit or the intra prediction unit. The pixels within the restored current block are used as reference pixels when intra-predicting the block to be decoded later.

[0057] The loop filter section (460) may include at least one of a deblocking filter (462), an SAO filter (464), and an ALF (466). The deblocking filter (462) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-unit decoding. The SAO filter (464) performs filtering by adding a corresponding offset to the restored block after deblocking filtering to compensate for the difference between the restored pixel and the original pixel caused by lossy coding. The ALF (466) performs filtering on a target pixel by applying filter coefficients to the target pixel and the surrounding pixels of the target pixel. The ALF (466) can perform filtering differentially for each group by dividing the pixels included in the image into a predetermined group and determining one filter to be applied to that group. The filter coefficients of the ALF are determined using information about the filter coefficients decoded from the non-stream.

[0058] The restored blocks filtered through the loop filter section (460) are stored in memory (470). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0060] As explained above, the video encoding device transmits a bitstream containing encoding data for the video, and the video decoder decodes the bitstream to restore each picture constituting the video.

[0061] In one aspect of the present disclosure, a bitstream may be composed of a plurality of transmission units, namely network abstraction layer (NAL) units. As illustrated in FIG. 5, a NAL unit may consist of a NAL unit header and data transmitted through the NAL unit. Bits may be added to the end of the data constituting the NAL unit to match the size of the NAL unit to bytes. The NAL unit header includes a NAL unit type to indicate the type of data transmitted through the NAL unit, a temporal layer ID of the NAL unit, etc.

[0062] Depending on the type of data included in the data field, NAL units can be classified into VCL (video coding layer) type NAL units and non-VCL type NAL units. A VCL (video coding layer) type NAL unit is a NAL unit that includes image data of a pixel group encoded in the data field, and typically includes data for a coded slice. This means that the unit in which image data is transmitted is a slice. A non-VCL type NAL unit includes parameters necessary for decoding the data of a pixel group in the data field. NAL units that include high-level syntax, such as an SPS (hereinafter referred to as the "first parameter set") containing parameters shared at the sequence level, or a PPS (hereinafter referred to as the "second parameter set") containing parameters shared at one or more picture levels, correspond to non-VCL type NAL units. Whether the transmitted NAL unit is associated with a first parameter set, a second parameter set, or an encoded slice, etc., is indicated by the NAL unit type included in the NAL unit header.

[0063] Additionally, a NAL unit may be further defined to transmit a third parameter set including parameters that are commonly applied to a picture or a group of pixels smaller than a picture, e.g., one or more slices. In the present disclosure, a bitstream representing a sequence includes one or more NAL units that transmit the third parameter set.

[0064] Figure 6 is an example diagram showing the structure of a NAL unit including a third parameter set.

[0065] The third parameter set is included in the data field of the NAL unit. The third parameter set includes at least one of its own ID information and parameter type information. Additionally, it includes parameters corresponding to the parameter type information.

[0066] Each third parameter set is associated with one of a plurality of parameter types depending on which coding tool the parameters it transmits are associated with. As shown in (A) of FIG. 6, information indicating the parameter type may be included in the third parameter set. In the structure of FIG. 6 (A), the NAL unit type may indicate whether the corresponding NAL unit is associated with the third parameter set through a specific index value. The types of parameters included in the third parameter set are identified by the parameter type information included in the third parameter set. Alternatively, as shown in (B) of FIG. 6, the parameter type may be indicated by the NAL unit type included in the NAL unit header. In this case, the NAL unit types are further subdivided according to the types of parameters included in the third parameter set. For example, in the structure of FIG. 6 (B), the NAL unit type may have different index values ​​depending on the types of parameters transmitted through the third parameter set. Depending on the value of the NAL unit type, whether the data included in the NAL unit is a third parameter set and the parameter type of the third parameter set are identified. As another example, as shown in (C) of FIG. 6, the data field of the NAL unit may contain only parameter type information without ID information. In this case, the ID of the third parameter set is assigned according to the encoding or decoding order. ID values ​​may be assigned sequentially for each parameter type, or ID values ​​may be assigned sequentially according to the decoding order regardless of the parameter type.

[0067] The parameter type of the third parameter set may include, for example, a loop filtering type, a scaling list type, a prediction information type, etc. If the parameter type information is a loop filtering type, the third parameter set includes information on one or more sets of filter coefficients used for loop filtering. Here, the filter coefficients belonging to each set may be expressed by the absolute value and the sign of the filter coefficients. If the parameter type information is a scaling list type, the third parameter set includes scaling values ​​used for the quantization of each transformation coefficient, that is, the coefficients of the quantization matrix. Additionally, if the parameter type information is a prediction information type, the third parameter set includes a set of filter coefficients used when generating a prediction signal in a specific prediction mode.

[0068] The video decoding device decodes NAL units associated with the third parameter set and stores the third parameter set in memory. An ID and parameter type for the third parameter set are determined based on the decoding of the NAL units. m (where m is an integer greater than or equal to 1) can be stored in memory for each parameter type, and m may vary depending on the parameter type. If m parameter sets are already stored in memory, the video decoding device deletes the existing parameter sets and stores the newly decoded third parameter set. For example, a parameter set that has the same ID and parameter type as the newly decoded third parameter set may be deleted from memory. Alternatively, the oldest stored third parameter set may be deleted. Or, ID and parameter type information for the third parameter set to be deleted may be extracted from the bitstream, and the third parameter set corresponding to the extracted information may be deleted from memory.

[0070] With the emergence of various applications such as 360 video, there is a demand for technology that displays not only the entire area of ​​a decoded picture but also only a portion of the picture. To support this technical requirement, another aspect of the present disclosure discloses a method for dividing pictures into a plurality of subpictures and encoding and decoding each subpicture. The present disclosure enables independent encoding or independent transmission for the subpictures constituting each picture. Furthermore, it enables independent extraction or decoding of data corresponding to each subpicture from the entire bitstream. Moreover, it enables each subpicture divided from the picture to be displayed independently of one another.

[0071] Figure 7 is an example diagram illustrating the layout of the sub-pictures that make up each picture.

[0072] A group of coding units (CUs) may be defined to represent the layout of subpictures divided from a picture. The coding unit group may be a CTU, a slice, a tile, or a grid of a predetermined size. The coding unit group may be classified into a first coding unit group, which is the basic unit constituting the subpicture, and a second coding unit group composed of multiple first coding unit groups. For example, the first coding unit group may be a CTU, and the second coding unit group may be a slice corresponding to a transmission unit. A subpicture may be composed of one or more second coding unit groups. For convenience of explanation, the first coding unit group is assumed to be a CTU and the second coding unit group is assumed to be a slice; however, it is obvious that the present invention is not necessarily limited thereto. For example, the first coding unit group may be a grid of a predetermined size, and the second coding unit group may be a slice or a tile.

[0073] Layout information is expressed as the number of subpictures within the picture, the ID of each subpicture, and the position and size of each subpicture within the picture. Here, the position and size of each subpicture can be expressed as information capable of identifying the CTUs constituting each subpicture.

[0074] The CTUs constituting each subpicture can be identified by the first and last CTUs constituting the subpicture in raster scan order. For a rectangular subpicture, the first and last CTUs in raster scan order refer to the CTU located at the top-left and bottom-right within each subpicture, respectively. Therefore, information for identifying the CTUs constituting the subpicture may include identification information for identifying the location of the top-left CTU (e.g., coordinate information of the top-left CTU) and identification information for identifying the CTU located at the bottom-right (e.g., coordinate information of the bottom-right CTU). Alternatively, information for identifying the CTUs constituting the subpicture may be expressed by identification information for identifying the location of the top-left CTU, the number of CTUs in the horizontal direction of the subpicture, and the number of CTUs in the vertical direction.

[0075] Figure 8 is another example diagram illustrating the layout of the sub-pictures that make up each picture.

[0076] In the example of FIG. 8, the first coding unit group, which is the basic unit constituting the subpicture, is a grid that groups multiple pixels. The subpicture may be a group of grids.

[0077] Layout information of subpictures can be represented by a grid. As layout information of subpictures within a picture, the size of the grid is defined first. The grid size can be defined by the width and height of the grid, or by the number of grids in the horizontal and vertical directions of the picture.

[0078] Layout information of subpictures includes the number of subpictures within the picture, the ID of each subpicture, and identification information for identifying the grids constituting each subpicture. For example, the identification information includes the identification information of the first grid and the identification information of the last grid within the subpicture in raster scan order. In a rectangular subpicture, the first grid corresponds to the top-left grid within the subpicture, and the last grid corresponds to the bottom-right grid within the subpicture. Therefore, the identification information includes the identification information of the top-left grid and the bottom-right grid within the subpicture. Alternatively, the identification information may include the identification information of the top-left grid within the subpicture and information on the number of grids in the horizontal and vertical directions. As one implementation example, the identification information may be the location of the grid. In another implementation example, the identification information may be represented as the ID or address of the grid. Here, the IDs or addresses for the grids within the picture may be assigned in ascending order starting from 0 according to a specific scan order, e.g., raster scan order.

[0079] The video encoding device encodes division information, i.e., layout information, for dividing each picture into subpictures. The layout information may be included in the aforementioned first parameter set or second parameter set. The video decoding device extracts the layout information included in the first parameter set or the second parameter set to identify the subpictures constituting each picture.

[0080] The video encoding device encodes a subpicture ID for each slice, which is a transmission unit. The video decoding device can determine which subpicture a given slice belongs to by extracting the subpicture ID on a slice-by-slice basis. Then, each subpicture is restored by decoding the blocks within one or more slices constituting the subpicture through the aforementioned decoding process.

[0081] Subpictures must be capable of independent transmission and encoding / decoding, and also capable of being displayed independently of one another. Accordingly, the image encoding device and image decoding device of the present disclosure may perform a process that does not allow reference to restored pixels that extend beyond the subpicture boundaries during the encoding or decoding process of a subpicture. Here, the boundaries of the subpictures may be identified by layout information.

[0082] In one embodiment, the prediction unit (120) of the image encoding device and the prediction unit (440) of the image decoding device predict pixels within the current block from previously restored pixels based on prediction information (inter prediction information or intra prediction information) of the current block. When a pixel location determined based on the prediction information falls outside the boundary of the current subpicture, the prediction unit (120, 440) predicts pixels within the current block using a replacement pixel instead of the previously restored pixel at the determined location, even if a previously restored pixel exists at that determined location. Here, the replacement pixel may have a predefined fixed pixel value, or it may be a pixel at a predefined location within the current subpicture, for example, a pixel within the current subpicture that touches the boundary of the subpicture.

[0083] The process of using replacement pixels instead of restored pixels that extend beyond the boundaries of the current subpicture can be involved in various prediction processes. For example, if restored reference pixels around the current block used for intra prediction of the current block extend beyond the boundaries of the subpicture, those reference pixels may be replaced with predefined fixed pixel values. As another example, if at least some of the integer pixels used to interpolate subpixels in inter prediction extend beyond the boundaries of the current subpicture, integer pixels within the subpicture adjacent to those boundaries may be used instead of the integer pixels extending beyond the boundaries.

[0084] In another embodiment, the loop filter unit (180) of the image encoding device and the loop filter unit (460) of the image decoding device may perform loop filtering on a target pixel by applying filter coefficients to the target pixel to be filtered and to the surrounding pixels of the target pixel within the restored current block. The loop filter unit (180, 460) performs loop filtering on the target pixel using a replacement pixel instead of the surrounding pixels that are outside the boundary of the current subpicture when at least one of the surrounding pixels is outside the boundary of the current subpicture. The replacement pixel may have a predefined fixed pixel value, or it may be a pixel at a predefined location within the current subpicture, for example, a pixel that touches the boundary of the subpicture within the current subpicture.

[0085] Filter coefficients to be used for loop filtering can be determined from the aforementioned third parameter set. The image encoding device signals ID information of the third parameter set associated with the loop filtering type for each coding unit group, for example, for each slice. The image decoding device decodes the ID information of the third parameter set associated with loop filtering from the header of the coding unit group, for example, the slice header, and selects the third parameter set corresponding to the decoded ID information from among the third parameter sets stored in memory.

[0086] ID information of multiple third parameter sets related to loop filtering may be signaled on a slice basis. To this end, information regarding the number of IDs of the third parameter sets included in the slice is signaled first, and ID information of the third parameter sets corresponding to that number is signaled. The number of ID information signaled may vary depending on the color component of the pixels subject to filtering, that is, whether the color component is lumina or chroma. For example, in the case of a lumina component, the number of ID information is signaled first, and ID information of the third parameter sets corresponding to that number is signaled. On the other hand, in the case of a chroma component, a single third parameter set of the loop filtering type can always be used. Therefore, only one ID information may be signaled without signaling information regarding the number of ID information.

[0087] Filter coefficients to be applied to blocks within a slice can be determined at the CTU level using a third parameter set selected at the slice level. All blocks included within a single CTU share the same filter coefficients.

[0088] As described above, each third parameter set may include one or more sets of filter coefficients. A set of filter coefficients includes multiple filter coefficients. The number of filter coefficients is determined by the number of pixels used to filter a single pixel. For example, if a total of n pixels (the pixel to be filtered and pixels surrounding that pixel) are used to filter a single pixel, a single filter set includes n filter coefficients corresponding to the n pixels.

[0089] The video encoding device signals filter information for loop filtering the corresponding CTU on a CTU-by-CTU basis. The video decoding device uses the signaled filter information to determine a set of filter coefficients to be applied to the corresponding CTU on a CTU-by-CTU basis.

[0090] In one embodiment of selecting a third parameter set in slice units, filter index information indicating one filter coefficient set among the filter coefficient sets included in the one third parameter set in CTU units may be signaled.

[0091] In another embodiment where multiple third parameter sets are determined at the slice level, parameter ID information indicating which of the multiple third parameter sets to use may be signaled at the CTU level. Additionally, filter index information indicating one filter coefficient set among the filter coefficient sets included in the third parameter set corresponding to the parameter ID information may be additionally signaled. Alternatively, the choice of which filter coefficient set to use among the filter coefficient sets included in the third parameter set corresponding to the parameter ID information may be derived based on the characteristics of the target pixel to be filtered within the CTU. For example, characteristics such as the directionality or activity of the target pixel may be calculated using the target pixel and surrounding pixels, and a filter coefficient set to be applied to each target pixel may be selected according to the calculated characteristics. Here, pixel characteristics such as the directionality or activity of the target pixel may be calculated through a gradient operation using the target pixel and its surrounding pixels, for example, pixels within a certain area including the target pixel. In this alternative embodiment, signaling for filter index information indicating which filter coefficient set to use among the filter coefficient sets included in the third parameter set corresponding to the parameter ID information is not required. The image encoding device and the image decoding device can calculate the characteristics of the target filter and select a filter set according to the calculated characteristics.

[0092] Meanwhile, in addition to the filter coefficient sets included in the third parameter sets, multiple predefined filter coefficient sets may also be used. Here, the multiple predefined filter coefficient sets may be filter coefficient sets determined for one or more CTUs decoded immediately before the current CTU in the decoding sequence, or filter coefficient sets determined for surrounding CTUs at a predefined location adjacent to the current CTU (e.g., CTUs located above and / or to the left of the current CTU). Alternatively, the multiple predefined filter coefficient sets may be filter coefficient sets fixedly pre-set for all CTUs. Hereinafter, the pre-set filter coefficient sets used in the previously decoded CTU are referred to as the 'filter set reference list'. First, information indicating whether the filter set reference list is used for the current CTU is signaled.

[0093] If the filter set reference list is not used, one or more of the parameter ID information and filter index information among the third parameter sets may be signaled for the filter coefficient set to be used in the current CTU, and the image decoder may derive the filter coefficient set to be applied to the current CTU or each pixel within the current CTU using the signaled information as described above.

[0094] When a filter set reference list is used, a filter set to be applied to the current CTU is selected from among the filter sets included in the preset filter set reference list. The selection may be performed by filter index information signaled from the image encoding device to the image decoding device, or, as described above, by calculating the characteristics of each pixel within the current CTU and deriving each pixel according to the calculated characteristics.

[0095] The subpictures restored through the decoding process described above can be displayed independently of each other. Each subpicture can be stored in a different storage space of a single memory or can be stored individually in multiple memories.

[0096] When multiple subframes are stored in the same memory, the subpictures can be stored with a certain amount of space between them.

[0097] A memory storage structure at the subframe level can be signaled from a video encoding device to a video decoder at the picture or subpicture level, or it may be derived by the index, coordinates, mutual reference relationships, etc. of the subpictures. As one example, multiple subpictures that are not in a reference relationship may be stored in different memories. As another example, multiple subpictures having the same location, the same size, the same subpicture index, or the same memory index may be stored in the same memory.

[0098] Multiple restored subpictures stored in different memories may be stored in the same memory prior to display. Multiple restored subframes stored discontinuously in the same memory may be stored continuously prior to display.

[0099] Meanwhile, as shown in FIG. 9, each picture may be divided into multiple subpictures such that there exists an overlapping area between the subpictures. The image decoder may select a pixel from one of the different subpictures constituting the overlapping area and store it in memory or display it. The selection of which subpicture pixel to choose may be signaled from the image encoding device to the image decoder. Alternatively, the image decoder may derive pixel values ​​within the overlapping area to be stored or displayed by applying an arithmetic expression, such as an average or a weighted average, to the multiple subpictures constituting the overlapping area.

[0101] It should be understood that the exemplary embodiments described above may be implemented in many different ways. The functions described in one or more examples may be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "...unit" to particularly emphasize their implementation independence.

[0102] Meanwhile, the various functions or methods described in this disclosure may be implemented as instructions stored in a non-transient recording medium that can be read and executed by one or more processors. A non-transient recording medium includes, for example, any type of recording device in which data is stored in a form readable by a computer system. For example, a non-transient recording medium includes storage media such as an EPROM (erasable programmable read-only memory), a flash drive, an optical drive, a magnetic hard drive, and a solid-state drive (SSD).

[0103] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment.

Claims

Claim 1 A video decoding method for decoding a bitstream encoded with a sequence of pictures divided into multiple subpictures, each containing multiple coding tree blocks, comprising: a step of decoding layout information of the subpictures divided from the picture included in the sequence from the bitstream; a step of determining a target block by dividing a coding tree block to be decoded into a tree structure within one of the subpictures identified by the layout information; a step of decoding prediction information for predicting the target block and information regarding residual signals of the target block from the bitstream; a step of generating a prediction block by predicting pixels within the target block based on the prediction information; a step of generating a residual block for the target block using information regarding the residual signals; and a step of restoring the target block by adding the prediction block and the residual block. An image decoding method comprising the step of performing loop filtering on a target pixel by applying filter coefficients to a target pixel to be filtered and to surrounding pixels of the target pixel within the restored target block, wherein one or more ID information indicating a parameter set related to the loop filtering is decoded from the header of a slice in which the target block is located, and one or more sets of filter coefficients for loop filtering of blocks within the slice are restored from the parameter set corresponding to the ID information decoded from the header of the slice. Claim 2 ◈Claim 2 was abandoned upon payment of the registration fee.◈ An image decoding method according to Claim 1, characterized in that the sub-pictures within the picture are units capable of being displayed independently of each other. Claim 3 ◈Claim 3 was abandoned upon payment of the registration fee.◈ An image decoding method according to Claim 1, wherein the layout information comprises the number of subpictures, identification information for identifying the location of the first coding tree block in raster scan order within each subpicture, and information regarding the size of the subpicture. Claim 4 ◈Claim 4 was abandoned upon payment of the registration fee.◈ An image decoding method according to Claim 3, characterized in that the identification information is information for identifying the location of a coding tree block located at the top left within each subpicture. Claim 5 ◈Claim 5 was abandoned upon payment of the registration fee.◈ An image decoding method according to Claim 3, characterized in that the information regarding the size of the subpicture includes the number of coding tree blocks in the horizontal direction and the number of coding tree blocks in the vertical direction. Claim 6 ◈Claim 6 was abandoned upon payment of the registration fee.◈ An image decoding method according to Claim 1, wherein the step of generating the prediction block is characterized by predicting a pixel within the target block using a replacement pixel instead of the previously restored pixel when the previously restored pixel at a location determined based on the prediction information falls outside the boundary of the subpicture where the target block is located. Claim 7 ◈Claim 7 was abandoned upon payment of the registration fee.◈ An image decoding method according to Claim 1, wherein, when at least one of the surrounding pixels extends beyond the boundary of the subpicture in which the target block is located, loop filtering is performed on the target pixel using a replacement pixel instead of the at least one surrounding pixel. Claim 8 ◈Claim 8 was abandoned upon payment of the registration fee.◈ An image decoding method according to Claim 7, wherein the bitstream comprises a first parameter set transmitting parameters commonly applied to the levels of the sequence and a second parameter set transmitting parameters commonly applied to the levels of the pictures, and wherein the layout information is decoded from the first parameter set or the second parameter set. Claim 9 ◈Claim 9 was abandoned upon payment of the registration fee.◈ A method for image decoding according to claim 8, wherein the bitstream comprises one or more third parameter sets that transmit parameters commonly applied to a pixel group having a size smaller than or equal to the picture, and each of the third parameter sets comprises ID information, type information indicating the type of parameters transmitted through the third parameter set among a plurality of parameter types, and parameters corresponding to the type information, wherein the plurality of parameter types include at least a parameter type related to the loop filtering. Claim 10 ◈Claim 10 was abandoned upon payment of the registration fee.◈ An image decoding method according to claim 8, characterized in that each of the subpictures comprises one or more slices containing a plurality of coding tree blocks. Claim 11 delete Claim 12 ◈Claim 12 was abandoned upon payment of the registration fee.◈ An image decoding method according to Claim 1, wherein the filter coefficients for loop filtering are determined from among one or more sets of filter coefficients in units of a coding tree block, and loop filtering for a pixel within the restored target block is performed using filter coefficients corresponding to the coding tree block where the target block is located. Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 A video encoding method for encoding a sequence of pictures divided into multiple subpictures, each containing multiple coding tree blocks, into a bitstream, comprising: a step of encoding layout information of the subpictures divided from the picture included in the sequence into the bitstream; a step of determining a target block by dividing a coding tree block to be encoded within one of the subpictures identified by the layout information into a tree structure; a step of encoding prediction information for predicting the target block into the bitstream; a step of generating a prediction block based on the prediction information; a step of generating a residual block for the target block based on the target block and the prediction block; a step of encoding the residual block into the bitstream; and a step of restoring the target block by adding the prediction block and the residual block. A method for decoding images, comprising the step of performing loop filtering on a target pixel by applying filter coefficients to a target pixel to be filtered and to surrounding pixels of the target pixel within the restored target block, wherein one or more ID information indicating a parameter set related to the loop filtering is encoded in the header of a slice in which the target block is located, and one or more sets of filter coefficients for loop filtering of blocks within the slice are included in the parameter set corresponding to the ID information encoded in the header of the slice. Claim 19 A method for transmitting a bitstream comprising a sequence of pictures divided into multiple subpictures, each containing multiple coding tree blocks, to a video encoding device, comprising: a step of encoding the sequence to generate a bitstream; and a step of transmitting the bitstream to a video decoding device, wherein the step of generating the bitstream comprises: a step of encoding layout information of the subpictures divided from the picture included in the sequence into the bitstream; a step of determining a target block by dividing a coding tree block to be encoded within one of the subpictures identified by the layout information into a tree structure; a step of encoding prediction information for predicting the target block into the bitstream; a step of generating a prediction block based on the prediction information; a step of generating a residual block for the target block based on the target block and the prediction block; a step of encoding the residual block into the bitstream; and a step of restoring the target block by adding the prediction block and the residual block. A method comprising the step of performing loop filtering on a target pixel by applying filter coefficients to a target pixel to be filtered and to surrounding pixels of the target pixel within the restored target block, wherein one or more ID information indicating a parameter set related to the loop filtering is encoded in the header of a slice in which the target block is located, and one or more sets of filter coefficients for loop filtering of blocks within the slice are included in the parameter set corresponding to the ID information encoded in the header of the slice.

Images