Image encoding and decoding based on reference pictures with different resolutions

By using reference pictures with different resolutions in video decoding and applying resolution correction filtering, the problem of inefficiency in video encoding and decoding is solved, and the encoding/decoding efficiency of inter-frame prediction is improved.

CN115066900BActive Publication Date: 2024-12-03HYUNDAI MOTOR CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202180008157.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-01-06
Filing Date
2021-01-06
Publication Date
2024-12-03
Estimated Expiration
2041-01-06

AI Technical Summary

Technical Problem

In video encoding and decoding, it is difficult for the prior art to effectively process reference pictures of different resolutions, resulting in low encoding/decoding efficiency, especially during inter-frame prediction, the decoding efficiency of motion vectors is not high.

Method used

By using reference pictures with different resolutions in the video decoding device, resolution correction filtering is applied when performing inter prediction, so that the resolution of the reference picture matches the resolution of the current block, thereby generating an appropriate prediction signal and improving encoding/decoding efficiency.

Benefits of technology

The efficiency of video encoding and decoding is improved, especially during inter-frame prediction, and more efficient motion vector decoding is achieved by taking into account the resolution of the reference picture.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115066900B_ABST
    Figure CN115066900B_ABST
Patent Text Reader

Abstract

The present embodiment relates to image encoding and decoding using reference pictures with different resolutions, and performs inter-frame prediction or intra-frame prediction on a current image by considering the resolution of the reference picture to improve encoding / decoding efficiency. In addition, an image encoding / decoding method for decoding a motion vector of a current image by referring to a motion vector of a reference picture with a different resolution from the current image during inter-frame prediction is provided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to video encoding and decoding, and more particularly, to a video encoding and decoding method for improving encoding / decoding efficiency by referring to reference pictures with different resolutions in consideration of different resolutions before reference. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] Since the volume of video data is larger than that of voice data or still image data, storing or transmitting the video data without compression processing requires a large amount of hardware resources including memory.

[0004] Therefore, when storing or transmitting video data, an encoder is usually used to compress the video data for storage or transmission. Then, a decoder receives the compressed video data and decompresses and reproduces the video data. Compression technologies used for such videos include H.264 / AVC and High Efficiency Video Coding (HEVC), which improves coding efficiency by about 40% over H.264 / AVC.

[0005] However, as picture size, resolution and frame rate gradually increase, the amount of data to be encoded also increases accordingly. Therefore, a new compression technology with better encoding efficiency and higher image quality than existing compression technologies is needed.

[0006] In video encoding / decoding, the current picture may be encoded / decoded with reference to a previously decoded picture in order to improve encoding / decoding efficiency. For example, in the case of video encoding / decoding for multiple layers or multiple views with different resolutions, the current picture and the reference picture may have different resolutions. In this case, a method for encoding / decoding the current picture taking into account the resolution of the reference picture is required. Summary of the invention

[0007] Technical issues

[0008] In the present invention, in video encoding and decoding using reference pictures with different resolutions, inter-frame prediction or intra-frame prediction is performed on a current picture in consideration of the resolution of the reference picture, so as to improve encoding / decoding efficiency. An object of the present invention is to provide a video encoding / decoding method for decoding a motion vector for a current picture by referring to a motion vector of a reference picture having a different resolution from that of the current picture during inter-frame prediction.

[0009] Technical Solution

[0010] According to an embodiment of the present disclosure, a video decoding method for a current block in a current block included in a current picture in a high-level layer is performed by a video decoding device based on a reference picture included in a low-level layer and having a different resolution, the video decoding method comprising: obtaining a prediction mode for the current block; obtaining a decoded residual signal and decoding information of the current block, wherein when the prediction mode is inter-frame prediction, the decoding information includes a reference picture index and a motion vector for the reference picture, and when the prediction mode is intra-frame prediction, the decoding information includes the reference picture and a reference position in the reference picture; when the prediction mode is the inter-frame prediction, generating a prediction signal for the current block based on the decoding information; and generating a reconstructed block by adding the prediction signal to the residual signal, wherein, when inter-frame prediction is used, a filter for correcting different resolutions is applied to the reference block included in the reference picture so that the different resolutions match the resolution of the current block.

[0011] According to another embodiment of the present disclosure, a video encoding method for a current block included in a high-level layer is performed by a video encoding device based on a reference picture included in a low-level layer and having a different resolution, the video encoding method comprising: generating a prediction mode for the current block; obtaining encoding information about the current block, wherein when the prediction mode is inter-frame prediction, the encoding information includes a reference picture index and a motion vector for the reference picture, and when the prediction mode is intra-frame prediction, the encoding information includes the reference picture and a reference position in the reference picture; when the prediction mode is the inter-frame prediction, generating a prediction signal for the current block based on the encoding information; and generating a residual signal by subtracting the prediction signal from the current block, wherein, when inter-frame prediction is used, a filter for correcting different resolutions is applied to the reference block included in the reference picture so that the different resolutions match the resolution of the current block.

[0012] Beneficial Effects

[0013] As described above, according to the present embodiment, encoding / decoding efficiency can be improved by providing a method of performing inter prediction or intra prediction on a current picture in consideration of the resolution of a reference picture in video encoding and decoding using reference pictures having different resolutions.

[0014] Furthermore, according to the present embodiment, by providing a method of decoding a motion vector with reference to a motion vector of a reference picture having a different resolution from a current picture in inter prediction, encoding / decoding efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1is an exemplary block diagram of a video encoding device capable of implementing the techniques of the present disclosure.

[0016] Figure 2 A block partition structure using a QTBTTT structure is exemplarily shown.

[0017] FIG. 3 is a diagram illustrating a plurality of intra prediction modes including a wide-angle intra prediction mode.

[0018] Figure 4 Neighboring blocks surrounding the current block are exemplarily shown.

[0019] Figure 5 is an exemplary block diagram of a video decoding device capable of implementing the techniques of the present disclosure.

[0020] Figure 6 is an exemplary diagram conceptually illustrating a decoding process for multiple layers with different resolutions according to an embodiment of the present disclosure.

[0021] Figure 7 The present invention is a schematic flowchart of a video decoding method using reference pictures with different resolutions according to an embodiment of the present disclosure.

[0022] Figure 8 is an exemplary diagram illustrating determining an intra prediction mode using reference pictures with different resolutions according to an embodiment of the present disclosure.

[0023] Fig. 9 is a conceptual exemplary diagram illustrating intra / inter hybrid prediction using reference pictures with different resolutions according to an embodiment of the present disclosure.

[0024] Fig.10 is a conceptual exemplary diagram illustrating inter-component referencing using reference pictures having different resolutions according to an embodiment of the present disclosure.

[0025] Fig.11 An example of a hierarchical division structure of a current picture and a motion vector reference picture according to an embodiment of the present disclosure is shown.

[0026] Fig.12 is a flowchart illustrating a method of deriving a motion vector and generating a prediction block using reference pictures with different resolutions according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that when reference numerals are added to constituent elements in the various drawings, the same reference numerals represent the same elements even though the elements are shown in different drawings. In addition, in the following description of the present disclosure, detailed descriptions of known functions and configurations incorporated herein will be omitted to avoid blurring the subject matter of the present disclosure.

[0028] Figure 1 is an exemplary block diagram of a video encoding device capable of implementing the technology of the present disclosure. Figure 1 A video encoding apparatus and elements of the apparatus are described.

[0029] The video encoding device includes a block divider 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filter unit 180, and a memory 190.

[0030] Each element of the video encoding device may be implemented in hardware or software or a combination of hardware and software. The functions of each element may be implemented as software, and a microprocessor may be implemented to execute the software functions corresponding to each element.

[0031] A video consists of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed on each region. For example, a picture is divided into one or more tiles or / and slices. Here, one or more tiles may be defined as a tile group. Each tile or slice is divided into one or more coding tree units (CTUs). Each CTU is divided into one or more coding units (CUs) through a tree structure. The information applied to each CU is encoded as the syntax of the CU, and the information applied to the CU included in a common CTU is encoded as the syntax of the CTU. In addition, the information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and the information applied to all blocks constituting one or more pictures is encoded in the picture parameter set (PPS) or the picture header. In addition, the information commonly referred to by a sequence consisting of multiple pictures is encoded in the sequence parameter set (SPS). In addition, the information commonly applied to a tile or tile group can be encoded as the syntax of the tile or tile group header. The syntax included in the SPS, PPS, slice header, and tile or tile group header can be referred to as high-level syntax.

[0032] The block divider 110 determines the size of a coding tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as a syntax of an SPS or a PPS and transmitted to a video decoding apparatus.

[0033] The block divider 110 divides each picture constituting a video into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. In the tree structure, a leaf node serves as a coding unit (CU), which is a basic unit of encoding.

[0034] The tree structure may be a quadtree (QT) (wherein a node (or parent node) is divided into four child nodes (or child nodes) of the same size), a binary tree (BT) (wherein a node is divided into two child nodes), a ternary tree (TT) (wherein a node is divided into three child nodes at a ratio of 1:2:1), or a structure formed by a combination of two or more of a QT structure, a BT structure, and a TT structure. For example, a quadtree plus a binary tree (QTBT) structure may be used, or a quadtree plus a binary tree ternary tree (QTBTTT) structure may be used. Here, BTTT may be collectively referred to as a multi-type tree (MTT).

[0035] Figure 2 The QTBTTT partition tree structure is exemplarily shown.

[0036] like Figure 2 As shown, the CTU can be initially divided into a QT structure. The QT division can be repeated until the size of the divided block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. The first flag (QT_split_flag) indicating whether each node of the QT structure is divided into four nodes of the lower layer is encoded by the entropy encoder 155 and sent to the video decoding device. When the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it can be further divided into one or more of the BT structure and the TT structure. The BT structure and / or the TT structure can have multiple division directions. For example, there can be two directions, namely, the direction in which the blocks of the node are divided horizontally and the direction in which the blocks are divided vertically. As Figure 2 As shown, when the MTT splitting starts, a second flag (mtt_split_flag) indicating whether the node is split, a flag indicating the splitting direction (vertical or horizontal) in the case of splitting, and / or a flag indicating the splitting type (binary or ternary) are encoded by the entropy encoder 155 and sent to the video decoding device with a signal.

[0037] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is divided into 4 nodes of the lower layer, a CU split flag (split_cu_flag) indicating whether the node is divided may be encoded. When the value of the CU split flag (split_cu_flag) indicates that the split is not performed, the block of the node becomes a leaf node in the partition tree structure and serves the coding unit (CU) which is the basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that the split is performed, the video encoding device starts encoding the flags in the above manner, starting with the first flag.

[0038] When QTBT is used as another example of a tree structure, there may be two types of division, namely, a type of dividing a block horizontally into two blocks of the same size (ie, symmetrical horizontal division) and a type of dividing a block vertically into two blocks of the same size (ie, symmetrical vertical division). A division flag (split_flag) indicating whether each node of the BT structure is divided into blocks of the lower layer and division type information indicating the division type are encoded by the entropy encoder 155 and sent to the video decoding device. There may be an additional type of dividing the block of a node into two asymmetric blocks. The asymmetric division type may include a type of dividing a block into two rectangular blocks at a size ratio of 1:3, or a type of diagonally dividing the block of a node.

[0039] Depending on the QTBT or QTBTTT partitioning of the CTU, the CU may have different sizes. In the following, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block". Due to the QTBTTT partitioning, the shape of the current block may be a square or a rectangle.

[0040] The predictor 120 predicts the current block to generate a predicted block. The predictor 120 includes an intra predictor 122 and an inter predictor 124.

[0041] In general, each of the current blocks in a picture may be predictively encoded. In general, prediction of the current block may be performed using intra-prediction techniques (using data from the picture containing the current block) or inter-prediction techniques (using data from a picture encoded before the picture containing the current block). Inter-prediction includes both unidirectional prediction and bidirectional prediction.

[0042] The intra-frame predictor 122 predicts pixels in the current block using pixels (reference pixels) located around the current block in the current picture including the current block. There are multiple intra-frame prediction modes according to the prediction direction. For example, Figure 3a As shown, the plurality of intra prediction modes may include two non-directional modes (including a planar mode and a DC mode) and 65 directional modes. According to each prediction mode, neighboring pixels and calculation formulas to be used are differently defined.

[0043] For efficient directional prediction of the rectangular current block, the Figure 3b The directional modes indicated by the dotted arrows in FIG. 5 (intra-prediction modes #67 to #80 and #-1 to #-14). These may be referred to as "wide-angle intra-prediction modes". Figure 3bThe arrows in indicate the corresponding reference samples used for prediction, not the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. The wide-angle intra prediction mode is a mode that predicts a specific directional pattern in the opposite direction without additional bit transmission when the current block is rectangular. In this case, in the wide-angle intra prediction mode, some wide-angle intra prediction modes available for the current block can be determined by the ratio of the width to the height of the rectangular current block. For example, when the current block has a rectangular shape with a height less than the width, a wide-angle intra prediction mode (intra prediction modes #67 to #80) with an angle less than 45 degrees is available, and when the current block has a rectangular shape with a width greater than the height, a wide-angle intra prediction mode (intra prediction modes #-1 to #-14) with an angle greater than -135 degrees is available.

[0044] The intra-predictor 122 may determine an intra-prediction mode to be used to encode the current block. In some examples, the intra-predictor 122 may encode the current block using several intra-prediction modes and select an appropriate intra-prediction mode to use from a test mode. For example, the intra-predictor 122 may calculate a rate-distortion value using a rate-distortion analysis of several tested intra-prediction modes and may select an intra-prediction mode having the best rate-distortion characteristics among the tested modes.

[0045] The intra predictor 122 selects an intra prediction mode from a plurality of intra prediction modes and predicts the current block using adjacent pixels (reference pixels) and an equation determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoder 155 and transmitted to the video decoding apparatus.

[0046] The inter-frame predictor 124 generates a prediction block for the current block by motion compensation. The inter-frame predictor 124 searches for a block most similar to the current block in a reference picture that has been encoded and decoded earlier than the current picture, and generates a prediction block for the current block using the searched block. Then, the inter-frame predictor generates a motion vector corresponding to the displacement between the current block in the current picture and the prediction block in the reference picture. Typically, motion estimation is performed on the luminance component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chrominance component. Motion information including information about the reference picture and information about the motion vector used to predict the current block is encoded by the entropy encoder 155 and sent to the video decoding device.

[0047] The inter-frame predictor 124 may perform interpolation on a reference picture or a reference block in order to increase prediction accuracy. That is, by applying a filter coefficient to a plurality of consecutive integer samples including two integer samples, a subsample between two consecutive integer samples is interpolated. When the operation of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be represented at an accuracy level of a fractional sampling unit instead of an accuracy level of an integer sampling unit. The accuracy or resolution of the motion vector may be set differently for each target area to be encoded (e.g., each unit such as a slice, a tile, a CTU, or a CU). When this adaptive motion vector resolution is applied, information about the motion vector resolution to be applied to each target area should be signaled for each target area. For example, when the target area is a CU, information about the motion vector resolution applied to each CU is signaled. The information about the motion vector resolution may be information indicating the accuracy of the motion vector difference to be described later.

[0048] The inter-frame predictor 124 may perform inter-frame prediction using bidirectional prediction. In bidirectional prediction, the inter-frame predictor 124 uses two reference pictures and two motion vectors representing the block position most similar to the current block in the corresponding reference picture. The inter-frame predictor 124 selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, searches for a block similar to the current block in the corresponding reference picture, and generates a first reference block and a second reference block. Then, a prediction block of the current block is generated by averaging or weighting the first reference block and the second reference block. Then, it transmits motion information to the encoder 150, the motion information including information about two reference pictures and two motion vectors used to predict the current block. Here, RefPicList0 may consist of pictures before the current picture in the display order among the reconstructed pictures, and RefPicList1 may consist of pictures after the current picture in the display order among the reconstructed pictures. However, the embodiment is not limited thereto. A pre-reconstructed picture subsequent to the current picture in the display order may be further included in RefPicList0, and conversely, a pre-reconstructed picture preceding the current picture may be further included in RefPicList1.

[0049] Different methods can be used to minimize the number of bits required to encode motion information.

[0050] For example, when the reference picture and motion vector of the current block are the same as those of the adjacent block, the motion information about the current block can be transmitted to the decoding device by encoding the information for identifying the adjacent block. This method is called "merge mode".

[0051] In the merge mode, the inter predictor 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as “merge candidates”) from the neighboring blocks of the current block.

[0052] like Figure 4 As shown in , all or part of the left block L, the upper block A, the upper right block AR, the lower left block BL, and the upper left block AL adjacent to the current block in the current picture can be used as the adjacent blocks for deriving the merge candidate. In addition, blocks located in a reference picture other than the current picture in which the current block is located (which may be the same as or different from the reference picture used to predict the current block) can be used as merge candidates. For example, a co-located block at the same position as the current block or a block adjacent to the co-located block in the reference picture can be used as a merge candidate.

[0053] The inter-frame predictor 124 uses such neighboring blocks to configure a merge list including a predetermined number of merge candidates. The inter-frame predictor 124 selects a merge candidate to be used as motion information about the current block from among the merge candidates included in the merge list, and generates merge index information for identifying the selected candidate. The generated merge index information is encoded by the encoder 155 and sent to the decoding device.

[0054] Another method of encoding motion information is the AMVP mode.

[0055] In the AMVP mode, the inter-frame predictor 124 derives a motion vector candidate for prediction of a motion vector of a current block by using neighboring blocks of the current block. Figure 2 All or part of the left block L, the upper block A, the upper right block AR, the lower left block BL, and the upper left block AL adjacent to the current block in the current picture in the , may be used as the adjacent block for deriving the predicted motion vector candidate. In addition, a block located in a reference picture other than the current picture including the current block (which may be the same as or different from the reference picture used to predict the current block) may be used as the adjacent block for deriving the predicted motion vector candidate. For example, a co-located block at the same position as the current block or a block adjacent to the co-located block in the reference picture may be used.

[0056] The inter predictor 124 derives predicted motion vector candidates using motion vectors of neighboring blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, a motion vector difference is calculated by subtracting the predicted motion vector from the motion vector of the current block.

[0057] The predicted motion vector can be obtained by applying a predefined function (e.g., a function for calculating a median, an average, etc.) to the predicted motion vector candidate. In this case, the video decoding device also knows the predefined function. Since the neighboring blocks used to derive the predicted motion vector candidate have been encoded and decoded, the video decoding device also already knows the motion vectors of the neighboring blocks. Thus, the video encoding device does not need to encode the information for identifying the predicted motion vector candidate. Therefore, in this case, the information about the motion vector difference and the information about the reference picture used to predict the current block are encoded.

[0058] The prediction motion vector may be determined by selecting any one of the prediction motion vector candidates. In this case, information for identifying the selected prediction motion vector candidate is further encoded together with information about the motion vector difference and information about the reference picture to be used for predicting the current block.

[0059] The subtractor 130 subtracts the prediction block generated by the intra predictor 122 or the inter predictor 124 from the current block to generate a residual block.

[0060] The transformer 140 converts the residual signal having the pixel value in the spatial domain in the residual block into a transform coefficient in the frequency domain. The transformer 140 may transform the residual signal in the residual block by using the entire size of the residual block as a transform unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transform units to perform the transform. Alternatively, the residual signal may be transformed by classifying the sub-block into two sub-blocks as a transform area and a non-transform area and using only the transform area sub-block as a transform unit. Here, the transform area sub-block may be one of two rectangular blocks having a size ratio of 1:1 based on the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag), direction (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) indicating that only the sub-block has been transformed are encoded by the entropy encoder 155 and signaled to the video decoding device. In addition, the size of the transform area sub-block may have a size ratio of 1:3 based on the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_quad_flag) for distinguishing the corresponding partition is additionally encoded by the entropy encoder 155 and notified to the video decoding device by signal.

[0061] The transformer 140 may transform the residual block separately in the horizontal direction and the vertical direction. For the transformation, different types of transformation functions or transformation matrices may be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation may be defined as a multi-transform set (MTS). The transformer 140 may select a pair of transformation functions with the best transformation efficiency in the MTS and transform the residual block in the horizontal direction and the vertical direction, respectively. Information (mts_idx) about the pair of transformation functions selected in the MTS is encoded by the entropy encoder 155 and notified to the video decoding device with a signal.

[0062] The quantizer 145 quantizes the transform coefficients output from the transformer 140 using the quantization parameter and outputs the quantized transform coefficients to the entropy encoder 155. For some blocks or frames, the quantizer 145 may directly quantize the relevant residual block without transformation. The quantizer 145 may apply different quantization coefficients (scaling values) according to the position of the transform coefficient in the transform block. The matrix of quantized coefficients applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to the video decoding device.

[0063] The rearrangement unit 150 may reorder coefficient values ​​for the quantized residual value.

[0064] The rearrangement unit 150 may change the two-dimensional array of coefficients into a one-dimensional coefficient sequence by coefficient scanning. For example, the rearrangement unit 150 may scan the coefficients from the DC coefficient to the coefficients in the high frequency region using a zigzag scan or a diagonal scan to output a 1-dimensional coefficient sequence. Depending on the size of the converter and the intra-frame prediction mode, vertical scanning (scanning the two-dimensional coefficient array in the column direction) or horizontal scanning (scanning the two-dimensional block-shaped coefficients in the row direction) may be used instead of the zigzag scan. That is, the scanning mode to be used may be determined among the zigzag scan, the diagonal scan, the vertical scan, and the horizontal scan according to the size of the converter and the intra-frame prediction mode.

[0065] The entropy encoder 155 encodes the one-dimensional quantized transform coefficient output from the rearrangement unit 150 using various encoding techniques such as context-based adaptive binary arithmetic coding (CABAC) and exponential Golomb to generate a bitstream.

[0066] The entropy encoder 155 encodes information associated with block partitioning (such as CTU size, CU partitioning mark, QT partitioning mark, MTT partitioning type and MTT partitioning direction) so that the video decoding device can partition the block in the same manner as in the video encoding device. In addition, the entropy encoder 155 encodes information about the prediction type indicating whether the current block is encoded by intra-frame prediction or inter-frame prediction, and encodes intra-frame prediction information (i.e., information about intra-frame prediction mode) or inter-frame prediction information (merge index for merge mode, information about reference picture index and motion vector difference for AMVP mode) according to the prediction type. The entropy encoder 155 also encodes information related to quantization (i.e., information about quantization parameters and information about quantization matrices).

[0067] The inverse quantizer 160 inversely quantizes the quantized transform coefficient output from the quantizer 145 to generate a transform coefficient. The inverse transformer 165 transforms the transform coefficient output from the inverse quantizer 160 from the frequency domain to the spatial domain and reconstructs a residual block.

[0068] The adder 170 adds the reconstructed residual block and the prediction block generated by the predictor 120 to reconstruct the current block. Samples in the reconstructed current block are used as reference samples in performing intra prediction of the next block.

[0069] The loop filter unit 180 filters the reconstructed samples to reduce block artifacts, ringing artifacts, and blurring artifacts due to block-based prediction and transform / quantization. The loop filter unit 180 may include at least a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.

[0070] The deblocking filter 180 filters the boundaries between the reconstructed blocks to remove the block artifacts caused by block-by-block encoding / decoding, and the SAO filter 184 and the ALF 186 perform additional filtering on the deblocking filtered image. The SAO filter 184 and the ALF 186 are filters for compensating the difference between the reconstructed pixels and the original pixels caused by lossy encoding. The SAO filter 184 improves the coding efficiency and the subjective image quality by applying an offset in units of CTU. On the other hand, the ALF 186 performs block-by-block filtering and compensates for the distortion by applying different filters according to the edges and the degree of change of the corresponding blocks. Information about the filter coefficients to be used for the ALF can be encoded and notified to the video decoding device with a signal.

[0071] The reconstructed blocks filtered by the deblocking filter 182, the SAO filter 184, and the ALF 186 are stored in the memory 190. Once all blocks in one picture are reconstructed, the reconstructed picture may be used as a reference picture for inter prediction of blocks in a picture to be encoded next.

[0072] Figure 5 is an exemplary functional block diagram of a video decoding device capable of implementing the technology of the present disclosure. Figure 5 A video decoding apparatus and elements of the apparatus are described.

[0073] The video decoding device may include an entropy decoder 510 , a rearrangement unit 515 , an inverse quantizer 520 , an inverse transformer 530 , a predictor 540 , an adder 550 , a loop filter unit 560 , and a memory 570 .

[0074] and Figure 1 Similar to the video encoding device of the present invention, each element of the video decoding device can be implemented in hardware, software or a combination of hardware and software. In addition, the function of each element can be implemented in software, and the microprocessor can be implemented to execute the function of the software corresponding to each element.

[0075] The entropy decoder 510 determines a current block to be decoded by decoding a bit stream generated by a video encoding device and extracting information related to block partitioning, and extracts prediction information required to reconstruct the current block and information about a residual signal, etc.

[0076] The entropy decoder 510 extracts information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the decoder determines the CTU as the top layer, i.e., the root node of the tree structure, and extracts the partition information of the CTU to partition the CTU using the tree structure.

[0077] For example, when the CTU is divided using the QTBTTT structure, the first flag (QT_split_flag) related to the division of the QT is extracted to divide each node into four nodes of the sublayer. For the node corresponding to the leaf node of the QT, the second flag (MTT_split_flag) and the information of the division direction (vertical / horizontal) and / or the division type (binary / ternary) related to the division of the MTT are extracted to divide the corresponding leaf node in the MTT structure. Therefore, each node below the leaf node of the QT is recursively divided in the BT or TT structure.

[0078] As another example, when a CTU is split using a QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU is split may be extracted. When the corresponding block is split, a first flag (QT_split_flag) may be extracted. In a split operation, after zero or more recursive QT splits, zero or more recursive MTT splits may occur for each node. For example, a CTU may directly undergo MTT splitting without undergoing QT splitting, or may only undergo multiple QT splits.

[0079] As another example, when the CTU is divided using the QTBT structure, the first flag (QT_split_flag) related to the QT division is extracted, and each node is divided into four nodes of the lower layer. Then, the split flag (split_flag) indicating whether the node corresponding to the leaf node of the QT is further divided in the BT and the split direction information are extracted.

[0080] Once the current block to be decoded is determined by dividing in a tree structure, the entropy decoder 510 extracts information about a prediction type indicating whether the current block is intra-prediction or inter-prediction. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts syntax elements for inter-prediction information, that is, information indicating a motion vector and a reference picture referenced by the motion vector.

[0081] The entropy decoder 510 also extracts information about the quantized transform coefficient of the current block as information related to quantization and information about the residual signal.

[0082] The rearrangement unit 515 may change a sequence of one-dimensional quantized transform coefficients entropy-decoded by the entropy decoder 510 into a two-dimensional coefficient array (ie, block) in the reverse order of coefficient scanning performed by the video encoding apparatus.

[0083] The inverse quantizer 520 inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantizer 520 may apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantizer 520 may perform inverse quantization by applying a matrix (scaling values) of quantization coefficients from the video encoding device to the two-dimensional array of quantized transform coefficients.

[0084] The inverse transformer 530 inversely transforms the inversely quantized transform coefficient from the frequency domain to the spatial domain to reconstruct a residual signal, thereby generating a reconstructed residual block for the current block.

[0085] In addition, when the inverse transformer 530 inverse transforms only a partial area (sub-block) of the transform block, the inverse transformer 530 extracts a flag (cu_sbt_flag) indicating that only the sub-blocks of the transform block have been transformed, sub-block direction (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or sub-block position information (cu_sbt_pos_flag), inversely transforms the transform coefficients of the sub-block from the frequency domain to the spatial domain to reconstruct the residual signal, and fills the area that has not been inverse transformed into the residual signal with "0" to generate a final residual block for the current block.

[0086] In addition, when MTS is applied, the inverse transformer 530 uses the MTS information (mts_idx) notified by the video encoding device signal to determine the transformation function or transformation matrix to be applied in the horizontal and vertical directions, respectively, and uses the determined transformation function to inversely transform the transformation coefficients in the transformation block in the horizontal and vertical directions.

[0087] The predictor 540 may include an intra predictor 542 and an inter predictor 544. The intra predictor 542 is activated when the prediction type of the current block is intra prediction, and the inter predictor 544 is activated when the prediction type of the current block is inter prediction.

[0088] The intra predictor 542 determines an intra prediction mode of a current block among a plurality of intra prediction modes based on a syntax element of the intra prediction mode extracted from the entropy decoder 510 , and predicts the current block using reference samples surrounding the current block according to the intra prediction mode.

[0089] The inter predictor 544 determines a motion vector of a current block and a reference picture referred to by the motion vector using the syntax elements of the intra prediction mode extracted from the entropy decoder 510 , and predicts the current block based on the motion vector and the reference picture.

[0090] The adder 550 reconstructs the current block by adding the residual block output from the inverse transformer to the prediction block output from the inter predictor or the intra predictor. When a block to be decoded later is intra predicted, pixels in the reconstructed current block are used as reference pixels.

[0091] The loop filter unit 560 is a loop filter and may include a deblocking filter 562, an SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundary between the reconstructed blocks in order to remove the block artifacts caused by block-by-block decoding. The SAO filter 564 and the ALF 566 perform additional filtering on the reconstructed blocks after the deblocking filtering to compensate for the difference between the reconstructed pixels and the original pixels caused by the lossy encoding. The filter coefficients of the ALF are determined using information about the filter coefficients decoded from the bitstream.

[0092] The reconstructed blocks filtered by the deblocking filter 562, the SAO filter 564, and the ALF 566 are stored in the memory 570. When all blocks in one picture are reconstructed, the reconstructed picture is used as a reference picture for inter prediction of blocks in a picture to be encoded later.

[0093] The present embodiment relates to video encoding and decoding as described above. More specifically, in video encoding and decoding using reference pictures with different resolutions, a current picture is encoded / decoded in consideration of the resolution of the reference picture so as to improve encoding / decoding efficiency. In addition, a video encoding / decoding method is provided for decoding a motion vector for a current picture during inter-frame prediction by referring to a motion vector of a reference picture having a different resolution from the current picture.

[0094] In the following description according to the present disclosure, it is assumed that Figure 5 The memories 190 and 570 of the video encoding / decoding apparatus shown in EMBODIMENT 3 include a decoded picture buffer (DPB) and a decoded buffer (DB).

[0095] Figure 6 is an exemplary diagram conceptually illustrating a decoding process for multiple layers with different resolutions according to an embodiment of the present disclosure.

[0096] exist Figure 6 In an example, a high-level layer and a low-level layer (or sequence) are obtained by encoding the same image at different resolutions, and video decoding may be performed with reference to such resolution features, the same picture sequence number (POC) information, etc. during decoding. In addition, the video decoding device may decode the high-level layer using the decoded information of the current picture based on the pre-stored decoded information of the low-level layer and the decoded residual signal of the current picture. Here, the decoded information of the low-level layer may include at least one of the following: (i) decoded information in terms of syntax, such as block partition information, prediction mode, prediction direction, transform kernel, motion vector, reference picture index, filter information (e.g., information about loop filters, interpolation filters, etc.), and weight prediction parameters (e.g., weights for weighted averaging of bidirectional reference blocks), (ii) pixel information of a decoded block having other features (e.g., different resolutions) matching the position of the current picture for decoding, and (iii) a residual signal for a reference picture. The position matching the current picture may be a picture of the low-level layer of the same POC or a picture of the low-level layer referenced by the current picture (i.e., a reference picture).

[0097] At the same time, in order to obtain low-level decoding information and a residual signal, the video decoding device may use at least one of the following methods: a method using information obtained by parsing a bit stream sent from a video encoding device and a method using pixel information of a reconstructed image generated by parsing a bit stream sent from a video encoding device and then decoding the parsed bit stream.

[0098] Hereinafter, in the description according to the present disclosure, it is assumed that a current picture includes a current block for decoding and is included in a high-level layer. It is assumed that a reference picture includes a reference block for decoding a current block or a reference position to be referenced and is included in a low-level layer.

[0099] Figure 7 The present invention is a schematic flowchart of a video decoding method using reference images with different resolutions according to an embodiment of the present disclosure.

[0100] Figure 7 The example of shows a method for performing inter-frame prediction based on reference pictures with different resolutions during a block-based decoding process. After determining the constraints of the reference picture index and the motion vector (S700), the video decoding device generates information about the reference picture index (S702) and generates information about the motion vector (S704). Here, the reference picture index and the motion vector can be based on the low-level decoding information as described above.

[0101] After generating the first reference picture based on the reference picture index, the video decoding apparatus generates a first prediction signal using the first reference picture (S706). In the process of generating the first prediction signal, a motion vector may be used.

[0102] When the resolution of the current block used for decoding is different from the resolution of the reference picture, the video decoding device may refer to the reference picture after applying a resolution correction filter to the reference picture. That is, the video decoding device generates a second reference picture (S708) by applying a resolution correction filter to a first reference picture having a resolution different from that of the current block, and then generates a second prediction signal (S710) using the second reference picture. In the process of generating the second prediction signal, a motion vector taking into account the resolution difference may be used. A method of generating a motion vector taking into account the resolution difference will be described later. At the same time, the second reference picture may be stored in a decoded picture buffer (DPB).

[0103] The video decoding device selects one of the first prediction signal and the second prediction signal as the prediction signal, and generates a reconstructed block by adding the residual signal about the current block to the prediction signal (S712). Here, when images with different resolutions are cropped and thus the first prediction signal is generated, the first prediction signal can be referenced without adjusting the resolution, so the video decoding device can select the first prediction signal as the prediction signal.

[0104] Hereinafter, constraints for reference picture index and motion vector will be described. When referring to reference pictures with different resolutions, when pictures of the same POC of different layers are reference pictures, transmission of the reference picture index or motion vector may be omitted. When the reference picture index is omitted, the video decoding device may refer to a low-level layer of the same POC for a high-level layer. In addition, when the motion vector is omitted, the video decoding device may consider the resolutions of the high-level layer picture and the low-level layer picture in the reference relationship and the corresponding positional relationship therebetween, and refer to the low-level layer picture without information about the motion vector of the high-level layer.

[0105] Alternatively, the video decoding device may use the reference picture index and motion vector of the low-level layer picture of the same POC as the prediction value. First, when the reference picture index of the low-level layer picture is used as the prediction value, the video decoding device can determine the reference picture index of the high-level layer picture by adding the difference of the reference picture index to the reference picture index. Here, the reference picture index of the high-level layer picture may be limited to the reference picture index of the low-level layer. That is, when decoding the current block included in the high-level layer, the prediction value of the reference picture index relative to the reference block included in the corresponding low-level layer is used as it is, and the transmission of the difference of the index can be omitted. When the motion vector of the low-level layer picture is used as the prediction value, the prediction value of the motion vector can be determined in consideration of the resolutions of the high-level layer picture and the low-level layer picture and the corresponding positional relationship between them.

[0106] Meanwhile, the motion vector of the reference picture for the current block may be limited to a specific value including 0. Here, the fact that the motion vector for the reference picture is 0 indicates that the same position as that of the current block is referenced in the second reference picture obtained by applying the resolution correction filter to the first reference picture.

[0107] As described above, when the reference picture index and the motion vector are restricted, transmission of the reference picture index and the motion vector may be omitted according to an agreement between the video encoding device and the video decoding device, and the video decoding device may decode the current block using the agreed values.

[0108] Meanwhile, even if transmission of a motion vector is omitted, based on decoder motion vector refinement (DMVR) performed by a video decoding apparatus, reference picture indices and motion vectors of a low-level layer considering resolution correction may be restored to the most suitable motion information for a high-level layer.

[0109] Here, DMVR may be performed using reconstructed samples of previously decoded blocks around the current block. Typically, when the decoding order is from top left to bottom right, the surrounding reconstructed samples consist of J (J is a natural number) sample lines on the left and top sides. The video decoding apparatus may calculate an error between the sample for DMVR and a reference picture of a filtered low-level layer to determine a position with a minimum error in the reference picture, and determine a reference position of the current block using the determined position.

[0110] In another embodiment of the present disclosure, the video decoding apparatus may calculate a weighted average of motion vectors of samples for DMVR and determine the weighted average for the motion vector of the current block, as described above.

[0111] Figure 8 is an exemplary diagram illustrating determining an intra prediction mode using reference pictures with different resolutions according to an embodiment of the present disclosure.

[0112] When decoding a reference picture of a low-level layer in an intra-frame prediction mode, the video decoding device according to the present embodiment may use the intra-frame prediction mode information pre-stored for the reference picture of the low-level layer to determine the intra-frame prediction mode of the current block. The video decoding device may obtain information about the low-level layer picture that will refer to the prediction mode, obtain information about the reference position (the position of the block to be referenced by the current block) from the reference picture of the low-level layer, and then perform intra-frame prediction on the current block using the intra-frame prediction mode information of the block corresponding to the reference position to generate a prediction signal. Here, the reference picture and the reference position may be based on the low-level decoding information as described above. The video decoding device may generate a reconstructed block by adding the prediction signal to the residual signal of the current block.

[0113] Meanwhile, according to the agreement between the video encoding / decoding devices, the reference picture and the reference position of the low-level layer may be limited to specific values. The reference picture in the low-level layer relative to the current picture of the video encoding / decoding device may be limited to a picture of a specific index. In addition, the reference position may be limited to a specific value including 0.

[0114] The fact that the reference position information is 0 indicates that when determining the position of a block whose prediction mode will be referenced in consideration of the resolutions of the reference picture and the current picture to be decoded and the corresponding positional relationship between them, the reference is made to the same position as the current block. Restricting the reference picture index indicates that only pictures of a low-level layer having the same POC as the POC of a high-level layer or having a specific temporal positional relationship are used for reference of the intra-frame prediction mode. When the restricted reference picture index is a fixed value or the video decoding device can determine the reference picture index according to a method agreed upon between video encoding / decoding devices, the transmission of the reference position information of the intra-frame prediction mode, the reference picture index, the intra-frame prediction mode, etc. is omitted, and the video decoding device can decode the current block using the agreed value.

[0115] Fig. 9 is a conceptual exemplary diagram illustrating intra / inter hybrid prediction using reference pictures with different resolutions according to an embodiment of the present disclosure.

[0116] The video decoding device according to the present embodiment can decode the current block based on intra / inter hybrid prediction. The video decoding device can generate a third reference signal by weighted summing a first reference signal obtained by performing resolution correction filtering on a reference block of a low-level layer and a second reference signal predicted intra-frame using decoded reference pixels around the current block, and perform decoding with reference to at least one of these signals. Here, the weights of the first reference signal and the second reference signal can be determined using at least one of a method of receiving weights from a video encoding device and using the weights, a method of receiving and using an index of a weight list, and a method of using a fixed value according to an agreement between video encoding / decoding devices. Fig. 9 a and b) in the example.

[0117] In another embodiment of the present invention, when the prediction mode for the first reference signal is an intra-frame prediction mode, the transmission of the intra-frame prediction mode for the second reference signal can be omitted according to an agreement between video encoding / decoding devices, and the prediction mode of the first reference signal can be used to predict the second reference signal.

[0118] In another embodiment of the present invention, when a reference picture with a different resolution is used, the reference picture index and the motion vector may be omitted. In addition, the reference picture of the current picture used for the video encoding / decoding device may be limited to a picture of a specific index. For example, when the relationship between the current picture and the reference picture is a layer relationship, and the current picture may refer to a low-level layer as a high-level layer, such as in Figure 6 In the example of , the video decoding device may refer to the low-level layer picture of the same POC for the high-level layer picture. In addition, the reference picture index of the high-level layer picture may be limited to the same POC picture of the low-level layer.

[0119] In another embodiment of the present disclosure, the motion vector of the reference block of the current block may be limited to a specific value including 0. The fact that the motion vector of the reference block is 0 indicates that in the second reference picture obtained by applying the resolution correction filter to the first reference picture, the same position as that of the current block is referenced.

[0120] As described above, when the reference picture index and the motion vector are restricted, the transmission of the reference picture index and the motion vector is omitted according to the agreement between the video encoding / decoding devices, and the video decoding device can decode the current block using the agreed value. That is, the video decoding device can determine the decoded block located at the same position in the low-level layer as the first reference signal, predict the second reference signal using the decoded reference pixels around the current block, perform weighted summation on the first reference signal and the second reference signal to generate a third reference signal, and perform decoding with reference to at least one of these signals.

[0121] At the same time, the weight for intra / inter hybrid prediction can be sent from the video encoding device to the decoding device in the form of any one of the index of the list, the weight value and the weighted prediction difference value. The video decoding device can determine the weight of the intra / inter hybrid prediction of the current block by using one of the index, the weight value and the weighted prediction difference value, and generate a prediction signal for the current block. The weight can be sent in units of basic decoding blocks, decoding block groups, tiles, slices, subframes, subframe groups, frames, frame groups, or sequences, or in the form of supplementary enhancement information (SEI) messages, and the video decoding device can apply the weight sent in the decoding unit belonging to the corresponding transmission unit according to the transmission unit.

[0122] although Fig. 9 The example shows a case where the resolution of the low-level layer is lower than that of the high-level layer, but the same method can be used in another embodiment of the present disclosure even when the resolution of the low-level layer is higher than that of the high-level layer.

[0123] Fig.10 is a conceptual exemplary diagram illustrating inter-component referencing using reference pictures having different resolutions according to an embodiment of the present disclosure.

[0124] In this embodiment, when a block at a reference position (i.e., a reference block) is decoded according to an inter-component reference, the video decoding apparatus may perform decoding using an inter-component weight and an offset based on the reference block during an inter-component reference for a current block. Here, decoding according to an inter-component reference means decoding a chrominance component of the current block using a luminance component of the current block, an inter-component weight, and an offset, as in Fig.10 The example and equation 1 are shown.

[0125] [Equation 1]

[0126] pred C (i, j) = α·rec L (i, j)+β.

[0127] Here, rec L (i, j) represents the decoded reconstructed pixel value in the (i, j) luma sample in the current block, and pred C (i, j) represents the predicted value in the resampled (i, j) chrominance sample in the current block. In addition, α and β indicate the weights and offsets between the components based on the reference block. In this embodiment, Fig.10 As shown, when a reference block is decoded according to an inter-component reference and a current block is decoded using inter-component reference decoding, the video decoding apparatus may use the pixel values ​​of the luminance and chrominance components of the reference block to calculate the weights and offsets between the components of the current block. Alternatively, the weights and offsets between the components may be calculated using only some pixels of the luminance and chrominance component blocks of the reference block or using pixel values ​​filtered by resolution correction.

[0128] In another embodiment of the present invention, when using reference pictures with different resolutions, the reference pictures and reference positions of the lower-level layers may be limited to specific values ​​according to an agreement between video encoding / decoding devices. In addition, the reference pictures of the current pictures of the video encoding / decoding devices may be limited to pictures of specific indexes. For example, Figure 6 In the example of , when the relationship between the current picture and the reference picture is a layer relationship and the current picture can refer to the low-level layer as a high-level layer, the video decoding device can refer to the low-level layer picture of the same POC for the high-level layer picture. In addition, the reference picture index of the high-level layer picture can be limited to the same POC picture of the low-level layer.

[0129] In another embodiment of the present disclosure, the reference position of the current block may be limited to a specific value including 0. The fact that the reference position information is 0 indicates that when determining the position of a block whose prediction mode will be referenced in consideration of resolutions of a reference picture and a current picture to be decoded and their correspondence, the same position as that of the current block is referenced.

[0130] As described above, when the reference picture index and the reference position are restricted, the transmission of the reference picture index and the reference position information is omitted according to the agreement between the video encoding / decoding devices, and the video decoding device can decode the current block using the agreed value. That is, when predicting a decoded block located at a common position in a low-level layer by inter-component reference, the video decoding device can use some or all of the pixel values ​​in the luminance block and the chrominance block of the decoded block located at the common position in the low-level layer to predict the weights between the components, and use the weights to perform inter-component prediction of the current block.

[0131] although Fig.10 The example shows a case where the resolution of the low-level layer is lower than that of the high-level layer, but in another embodiment of the present disclosure, the same method can be applied even when the resolution of the low-level layer is higher than that of the high-level layer.

[0132] In another embodiment of the present invention, when using a reference picture with a different resolution, if matrix weighted intra prediction (MIP) is applied to a block at a reference position (i.e., a reference block), the video decoding device can use the MIP mode used by the reference block as the MIP mode of the current block to decode the current block. Here, since the MIP mode for the current block does not need to be transmitted, the encoding efficiency can be improved.

[0133] In another embodiment of the present invention, when using reference pictures with different resolutions, the reference pictures and reference positions of the lower-level layers may be limited to specific values ​​according to an agreement between video encoding / decoding devices. In addition, the reference pictures of the current pictures of the video encoding / decoding devices may be limited to pictures of specific indexes. For example, Figure 6 In the example of , when the relationship between the current picture and the reference picture is a layer relationship and the current picture can refer to the low-level layer as a high-level layer, the video decoding device can refer to the low-level layer picture of the same POC for the high-level layer picture. In addition, the reference picture index of the high-level layer picture can be limited to the same POC picture of the low-level layer.

[0134] In another embodiment of the present disclosure, the reference position of the current block may be limited to a specific value including 0. The fact that the reference position information is 0 indicates that when determining the position of a block whose prediction mode will be referenced in consideration of resolutions of a reference picture and a current picture to be decoded and their correspondence, the same position as that of the current block is referenced.

[0135] As described above, when the reference picture index and the reference position are restricted, transmission of the reference picture index and the reference position information is omitted according to an agreement between video encoding / decoding devices, and the video decoding device can decode the current block using the agreed values.

[0136] Although the present embodiment shows the case where the resolution of the low-level layer is lower than that of the high-level layer, in another embodiment of the present disclosure, the same method can be applied even when the resolution of the low-level layer is higher than that of the high-level layer.

[0137] When a video encoding apparatus that performs encoding on a plurality of layers having different resolutions encodes a high-level layer, all or some of the video decoding methods using reference pictures having different resolutions as described above may be used.

[0138] Hereinafter, in the description according to the present disclosure, a current picture includes a current block for decoding, and a reference picture includes a reference block to be referred to for decoding the current block.

[0139] When motion vector prediction is performed from a reference picture having a resolution different from that of the current picture, the encoding efficiency of the motion vector may be reduced due to the difference in ratio between the motion vector required for the current block and the predicted value of the motion vector based on the reference picture. Fig.11 and Fig.12 A method of predicting a motion vector of a current block from a region of a reference picture having a different resolution from the current block by a video decoding apparatus while considering a resolution difference is described.

[0140] As described above, a picture may be hierarchically divided into structures such as sub-pictures, CTUs, slices, tiles, and coding units. A picture may be divided into a plurality of sub-pictures, and a video decoding device may parse the division information from the high-level information of the SPS. A picture may be divided into a plurality of CTUs, and the division information may be parsed from the high-level information of the PPS.

[0141] In addition, the video decoding device may parse the start address and end address of the sub-picture, the sub-picture resampling scaling value (wS) regarding the picture resolution, the width and height of the reference picture, etc. from high-level information such as the SPS. SP and hS SP ). In addition, a table matching motion vector offset values ​​with motion vector offset indices can be parsed from the high-level information of the SPS. The motion vector offset represents the difference between motion vectors.

[0142] The video decoding device may parse the width and height of the reference picture from the high-level information of the PPS, and adjust the scaling value (wS) of the picture according to the width and height of the reference picture. P and hS P ) for resampling. The width and height of the current image can be obtained by multiplying or dividing the width and height of the reference image by wS P and hS P In addition, the number of CTUs, the width and height of the CTU, and the resampling scaling value (wS) of the CTU with respect to the resolution of the sub-picture can be parsed from the high-level information of the PPS. CTU and hS CTU ). This can be done by multiplying the width of the signaled CTU by wS CTU Or divide it by wS CTU The width of the changed CTU can be derived by multiplying the height of the signaled CTU by hS CTU Or divide the signaled CTU height by hS CTU To derive the height of the changed CTU.

[0143] In addition, the video decoding device may parse the number of sub-pictures, sub-picture index, width and height of the sub-picture, wS SP ,hS SP Etc. The width of the sub-picture sent by the signal can be multiplied or divided by wS SP or wS P The width of the changed CTU can be derived by multiplying the height of the signaled sub-picture by hS SP or hS P Or divided by hS SP or hS P To derive the height of the changed CTU.

[0144] Fig.11 An example of a hierarchical division structure of a current picture and a motion vector reference picture according to an embodiment of the present invention is shown.

[0145] exist Fig.11 In the example of , the motion vector reference picture indicates a picture specified by a reference picture index relative to the current block. Fig.11 The example of shows the scaling of the width and height of each CTU including the current block A and the motion vector reference block B, the sub-picture, and the picture. The scaling difference between the CTUs including the current block A and the motion vector reference block B may be the scaling difference (wS) between the current block A and the motion vector reference block B. diff_curr_ref and hS diff_curr_ref ). wS can be derived using the scaling value of the current block A relative to the reference picture and the scaling value of the motion vector reference block B relative to the reference picture. diff_curr_ref and hS diff_curr_ref . You can use wS CTU × SP × P Derives the scaling value of the width of the coding unit relative to the reference picture. hS can be used CTU × SP × P Derives a scaling value for the height of the coding unit relative to the reference picture.

[0146] Fig.12 is a flowchart illustrating a method of deriving a motion vector and generating a prediction block using reference pictures with different resolutions according to an embodiment of the present disclosure.

[0147] The video decoding device may derive a first reference motion vector by parsing an index on a motion vector reference list (S1200). A configuration method mode for a motion vector reference list may be parsed and used. The motion example reference list may consist of blocks at positions close to the same picture as the current block or another picture. In addition, the motion vector reference list may be configured by sequentially updating decoded blocks.

[0148] The video decoding apparatus may derive a second reference motion vector by changing the scaling of the first reference motion vector using the resolution difference between the current block and the motion vector reference block (S1202). The resolution scaling difference between the current block and the motion vector reference block may be derived using the difference between the resolution scaling of the CTU including the current block and the resolution scaling of the CTU including the motion vector reference block. The scaling of the resolution of the current block relative to the reference picture may be determined using wS CTU , wS SP and wS P The scaling of the motion vector reference block relative to the resolution of the reference block can be performed by using one or more wS stored in a decoding buffer (DB). CTU , wS SP and wS P The resolution scaling difference between the current block and the motion vector prediction block can be derived by multiplying or dividing the scaling of the current block relative to the resolution of the reference picture by the scaling of the motion vector prediction block relative to the resolution of the reference picture.

[0149] The video decoding device may derive the second reference motion vector by multiplying or dividing the first reference motion vector by the derived resolution scaling difference. Alternatively, the video decoding device may derive the second reference motion vector by multiplying or dividing the first reference motion vector by the derived resolution scaling difference and adding a motion vector offset to the result. Here, the resolution scaling difference may be used to parse or derive a motion vector offset index. The video decoding device may refer to a motion vector offset table parsed from high-level information to derive a motion vector offset corresponding to the motion vector offset index.

[0150] Based on the prediction mode (eg, merge mode or AMVP), the video decoding apparatus may selectively derive a residual motion vector (S1204). The value of the residual motion vector, the sign index of the residual motion vector, the size index of the residual motion vector, etc. may be parsed from the high-level information.

[0151] In addition, the sign of the residual motion vector and the table of index matching can be parsed from the high-level information or CTU-level information. The video decoding device can derive the sign of the residual motion vector by using the parsed index and the matching table. In addition, the residual motion vector size and the table of index matching can be parsed from the high-level information or CTU-level information. The video decoding device can derive the size of the residual motion vector by using the parsed index and the matching table.

[0152] The magnitude of the derived residual motion vector can be multiplied or divided by wS CTU , wS SP and wS PThe video decoding device may generate a residual motion vector by using a sign of the residual motion vector, a magnitude of the residual motion vector, or a magnitude of the second residual motion vector. When deriving the residual motion vector, the video decoding device adds the residual motion vector to the reference motion vector to derive the motion vector.

[0153] The residual motion vector may be derived without performing the second reference motion vector derivation process according to the prediction mode. The video decoding device may derive the first motion vector by adding the first reference motion vector to the residual motion vector (S1206). In addition, the video decoding device may generate the second motion vector by multiplying or dividing the first motion vector by the resolution scaling parameter (S1208).

[0154] On the other hand, when the residual motion vector is not derived, the video decoding apparatus may designate the second reference motion vector as the motion vector.

[0155] The video decoding device may generate a prediction block using the resolution scaling difference between the current block and the prediction block (S1210). The resolution scaling of the current block and the prediction block may be performed using the wS corresponding to each block. CTU , wS SP and wS P The video decoding device may derive a scaling difference between the current block and the prediction block by multiplying or dividing the resolution scaling of the current block by the resolution scaling of the prediction block. The resolution of the reference block may be resampled using the derived scaling difference so that the resolution is changed to be the same as the resolution of the current block. The resampled reference block may be used as the first prediction block.

[0156] In a prediction mode using two or more pieces of motion vector information, the video decoding device may perform Fig.12 The process shown in is performed multiple times to generate multiple prediction blocks, and then the generated prediction blocks are weighted averaged to generate a second prediction block. The weight of each prediction block used in the weighted average process can be set to the same value, parsed, derived based on time / space distance, or derived based on a scaling difference with the current block.

[0157] When block decoding is complete, the recovered sample values ​​can be stored in the DPB, along with high-level information, motion vectors, picture partitioning structure, wS CTU , wS SP , wS PThe like may be stored in the DB. The memory capacity required to store sample values ​​in the DPB may be equal to N×M×c (M, N, and c are natural numbers). Here, N×M is a common divisor of the width of a CTU, a sub-picture, or a picture, and c is the number of CTUs, sub-pictures, or pictures. In addition, when the number of CTUs, sub-pictures, or pictures currently stored in the DPB is d (d is a natural number), if d is less than or equal to c, the reconstructed CTU, sub-picture, or picture may be stored in the DPB.

[0158] As described above, according to the present embodiment, encoding / decoding efficiency can be improved by providing a method for performing inter prediction or intra prediction on a current picture considering the resolution of a reference picture in video encoding and decoding using reference pictures having different resolutions.

[0159] Furthermore, according to the present embodiment, by providing a method for decoding a motion vector with reference to a motion vector of a reference picture having a different resolution from a current picture in inter-frame prediction, encoding / decoding efficiency can be improved.

[0160] Although it is described that the processing is performed sequentially in each flowchart according to the present embodiment, the present disclosure is not limited thereto. In other words, the flowchart is not limited to a time series order because the order of the processing of the flowchart can be changed or one or more processing can be performed in parallel.

[0161] The various functions or methods described in this embodiment may be implemented as instructions that can be read and executed by one or more processors and stored in a non-transitory recording medium. For example, non-transitory recording media include all kinds of recording devices in which data is stored in a form readable by a computer system. For example, non-transitory recording media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSD).

[0162] Although exemplary embodiments have been described for illustrative purposes, those skilled in the art will recognize that various modifications and variations are possible without departing from the spirit and scope of the embodiments. Exemplary embodiments have been described for the sake of brevity and clarity. Therefore, those of ordinary skill in the art will understand that the scope of the embodiments is not limited to the embodiments explicitly described above, but includes the claims and their equivalents.

[0163] (Marking Description)

[0164] 120: Predictor

[0165] 122: Intra-frame predictor 124: Inter-frame predictor

[0166] 190: Memory

[0167] 540: Predictor

[0168] 542: Intra-frame predictor 544: Inter-frame predictor

[0169] 570: Memory.

[0170] CROSS-REFERENCE TO RELATED APPLICATIONS

[0171] This patent application claims the priority of Patent Application No. 10-2020-0001674 filed in South Korea on January 6, 2020, Patent Application No. 10-2020-0001675 filed in South Korea on January 6, 2020, and Patent Application No. 10-2021-0001214 filed in South Korea on January 6, 2021, the entire contents of which are incorporated by reference into this patent application.

Claims

1. A video decoding method, the video decoding method being performed by a video decoding device based on a reference picture included in a low-level layer and having a different resolution for a current block in a current picture included in a high-level layer, the video decoding method include: Obtaining a prediction mode of the current block; obtaining a decoded residual signal and decoded information of the current block, wherein when the prediction mode is inter-frame prediction, the decoded information includes a reference picture index and a motion vector for the reference picture, and when the prediction mode is intra-frame prediction, the decoded information includes the reference picture and a reference position in the reference picture; generating a prediction signal for the current block based on the decoded information; and by adding the prediction signal to the residual signal, generating a reconstructed block, Wherein, when the prediction mode is the inter-frame prediction, generating the prediction signal comprises: applying a filter for correcting the different resolutions to a reference block included in the reference picture so that the different resolutions match a resolution of the current block; When the prediction mode is the intra prediction, generating the prediction signal includes: setting the same position as the current block in the reference picture as the reference position, determining an intra prediction mode for the current block using the intra prediction mode information of the block at the reference position of the reference picture, and Based on the determined intra prediction mode of the current block, a prediction signal for the current block is generated using pre-reconstructed pixels adjacent to the current block in the current picture.

2. The video decoding method according to claim 1, in, The inter-frame prediction includes: generating a first reference picture based on the reference picture index; and A second reference picture is generated by applying the filtering to the first reference picture, and then a prediction signal of the current block is generated using a motion vector in consideration of the different resolutions and the second reference picture.

3. The video decoding method according to claim 2, in, The inter prediction includes, when transmission of the reference picture index is omitted, referring to a picture having the same picture sequence number as the first reference picture included in the lower-level layer.

4. The video decoding method according to claim 2, in, When the transmission of the motion vector is omitted, the inter-frame prediction includes generating a prediction signal of the current block based on resolutions of the current block and the reference picture and a corresponding position relationship between the current block and the reference picture.

5. The video decoding method according to claim 4, in, The inter prediction includes generating a refined motion vector for the current block using a decoder motion vector refinement based on the motion vector taking into account resolution correction, wherein the decoder motion vector refinement is performed using previously decoded samples around the current block.

6. The video decoding method according to claim 5, in, The decoder motion vector refinement calculates an error between the previously decoded samples and the second reference picture, and determines a reference position of the current block based on a position in the second reference picture having a minimum error.

7. The video decoding method according to claim 1, in, The inter prediction includes generating a reference picture index and a motion vector for the current block using the reference picture index and a motion vector of a picture having the same picture sequence number as the current picture among pictures in the low-level layer as prediction values.

8. The video decoding method according to claim 1, in, Decoding the current block based on intra / inter hybrid prediction, The hybrid prediction includes: generating a first reference signal by applying filtering to the reference block; Predicting a second reference signal using decoded reference pixels around the current block; and The prediction signal of the current block is generated by weighted addition of the first reference signal and the second reference signal based on a preset weight.

9. The video decoding method according to claim 1, in, The intra-frame prediction includes: using weights and offsets between components of the block at the reference position, and pixel values ​​of the luminance component of the current block when the block at the reference position is decoded according to the inter-component reference, to generate a prediction signal of the pixel value of the chrominance component of the current block.

10. The video decoding method according to claim 9, in, The intra prediction includes calculating the weights and the offsets between components using pixel values ​​of all or part of the luminance component and the chrominance component of the block at the reference position, or pixel values ​​of the block at the reference position to which a resolution correction filter is applied.

11. A video encoding method, the video encoding method being performed by a video encoding device based on a reference picture included in a low-level layer and having a different resolution for a current block included in a high-level layer, the video encoding method include: generating a prediction mode for the current block; obtaining encoding information about the current block, wherein when the prediction mode is inter-frame prediction, the encoding information includes a reference picture index and a motion vector for the reference picture, and when the prediction mode is intra-frame prediction, the encoding information includes the reference picture and a reference position in the reference picture; generating a prediction signal for the current block based on the encoding information; and generating a residual signal by subtracting the prediction signal from the current block, Wherein, when the prediction mode is the inter-frame prediction, generating the prediction signal comprises: applying a filter for correcting different resolutions to a reference block included in the reference picture so that the different resolutions match a resolution of the current block; When the prediction mode is the intra prediction, generating the prediction signal includes: setting the same position as the current block in the reference picture as the reference position, determining an intra prediction mode for the current block using the intra prediction mode information of the block at the reference position of the reference picture, and Based on the determined intra prediction mode of the current block, a prediction signal for the current block is generated using pre-reconstructed pixels adjacent to the current block in a current picture.

12. The video encoding method according to claim 11, in, The inter-frame prediction includes: generating a first reference picture from the reference picture based on the reference picture index; and A second reference picture is generated by applying the filtering to the first reference picture, and then a prediction signal of the current block is generated using a motion vector in consideration of the different resolutions and the second reference picture.

13. A method for delivering a bitstream associated with video data, the method comprising: include: Generate a bitstream by encoding video data; as well as passing the bitstream to a video decoding device; Wherein, generating the bit stream comprises: generating a prediction mode for a current block in a current picture included in a high-level layer; obtaining encoding information of the current block, wherein, when the prediction mode is inter prediction, the encoding information includes a reference picture index and a motion vector for a reference picture included in a lower-level layer and having a different resolution, and when the prediction mode is intra prediction, the encoding information includes the reference picture and a reference position in the reference picture; Based on the encoding information, generating a prediction signal for the current block; and generating a residual signal by subtracting the prediction signal from the current block; Wherein, when the prediction mode is the inter-frame prediction, generating the prediction signal comprises: applying a filter for correcting the different resolutions to a reference block included in the reference picture so that the different resolutions match a resolution of the current block; Wherein, when the prediction mode is the intra-frame prediction, generating the prediction signal includes: setting the position in the reference picture that is the same as the position of the current block as the reference position, determining an intra prediction mode for the current block using the intra prediction mode information of the block at the reference position of the reference picture, and Based on the determined intra prediction mode of the current block, a prediction signal for the current block is generated using pre-reconstructed pixels adjacent to the current block in the current picture.