Image encoding / decoding method and apparatus

By configuring and selecting motion information prediction candidate groups, based on differential and predicted motion vector accuracy scaling and encoding, the problem of insufficient inter prediction performance in the prior art is solved, and the image encoding performance is improved.

CN118972564APending Publication Date: 2024-11-15INST OF IMAGE TECH INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411279955.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-07-24
Filing Date
2019-07-01
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In the prior art, inter prediction performance is insufficient in image encoding and decoding, and it is difficult to effectively improve the performance and efficiency of the image processing system.

Method used

By determining the motion information encoding mode of the target block, configuring the motion information prediction candidate group, selecting the motion information in the predicted candidate group, scaling and encoding based on the differential motion vector accuracy and the predicted motion vector accuracy, and realizing inter prediction.

Benefits of technology

Effectively configure the motion information prediction candidate group, reducing the number of motion information encoding bits of the target block and improving the encoding performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972564A_ABST
    Figure CN118972564A_ABST
Patent Text Reader

Abstract

An inter prediction method performed by an image decoding apparatus includes: determining a motion vector prediction candidate from among motion vectors of a reference block; determining a motion vector prediction candidate group including the motion vector prediction candidates; deriving a predicted motion vector of the target block from the motion vector prediction candidate group; determining a differential motion vector accuracy among the plurality of accuracy candidates based on the differential motion vector accuracy information; determining a differential motion vector of the target block; scaling the differential motion vector of the target block based on a precision difference between the differential motion vector precision and a motion vector precision of the predicted motion vector; deriving a motion vector of the target block based on the predicted motion vector of the target block and the scaled differential motion vector of the target block; and performing inter prediction by using the motion vector of the target block.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with the application date of July 1, 2019, application number 201980044609.X, and invention name “Image encoding / decoding method and device”. Technical Field

[0002] The present disclosure relates to image encoding and decoding techniques, and more particularly, to an image encoding / decoding method and device in inter-frame prediction. Background Art

[0003] As the Internet and mobile terminals are provided and information and communication technology develops, the use of multimedia data is rapidly increasing. Therefore, in order to perform various services or tasks through image prediction in all kinds of systems, the demand for improving the performance and efficiency of image processing systems has increased significantly, but the results from research and development that can respond to this atmosphere are not enough.

[0004] Therefore, in the conventional image encoding and decoding methods and devices, it is necessary to improve the performance of image processing, especially the performance of image encoding or image decoding. Summary of the invention

[0005] Technical issues

[0006] The purpose of the present disclosure is to provide an inter-frame prediction method.

[0007] Furthermore, the present disclosure provides a method and apparatus for configuring a motion information prediction candidate group for inter prediction.

[0008] Furthermore, the present disclosure provides a method and apparatus for performing inter-frame prediction according to a motion information encoding mode.

[0009] Technical Solutions

[0010] The image encoding / decoding method and device according to the present disclosure can: determine the motion information encoding mode of the target block; configure a motion information prediction candidate group according to the motion information encoding mode; derive the motion information of the target block by selecting one candidate in the motion information prediction candidate group; and perform inter-frame prediction on the target block based on the motion information of the target block.

[0011] According to the contents of the present disclosure, an inter-frame prediction method performed by an image decoding device includes: determining a motion vector prediction candidate from a motion vector of a reference block; determining a motion vector prediction candidate group including the motion vector prediction candidate; deriving a predicted motion vector of a target block from the motion vector prediction candidate group; determining a differential motion vector accuracy among a plurality of accuracy candidates based on differential motion vector accuracy information; determining the differential motion vector of the target block; scaling the differential motion vector of the target block based on an accuracy difference between the differential motion vector accuracy and the motion vector accuracy of the predicted motion vector; deriving a motion vector of the target block based on the predicted motion vector of the target block and the scaled differential motion vector of the target block; and performing inter-frame prediction by using the motion vector of the target block.

[0012] According to the content of the present disclosure, an inter-frame prediction method performed by an image encoding device includes: deriving a motion vector of a target block; performing inter-frame prediction by using the motion vector of the target block; specifying a reference block of the target block; determining a motion vector prediction candidate from the motion vector of the reference block; determining a motion vector prediction candidate group including the motion vector prediction candidate; deriving the predicted motion vector of the target block from the motion vector prediction candidate group; determining a differential motion vector of the target block based on the predicted motion vector of the target block; determining the differential motion vector accuracy among multiple accuracy candidates; scaling the differential motion vector of the target block based on the accuracy difference between the differential motion vector accuracy and the motion vector accuracy of the predicted motion vector; and encoding the motion vector of the target block based on the predicted motion vector of the target block and the scaled differential motion vector of the target block, and encoding differential motion vector accuracy information indicating the accuracy of the differential motion vector.

[0013] According to a non-temporary computer-readable recording medium of the present disclosure, which stores a bit stream generated by an inter-frame prediction method performed by an image encoding device, the method includes: deriving a motion vector of a target block; performing inter-frame prediction by using the motion vector of the target block; specifying a reference block of the target block; determining a motion vector prediction candidate from the motion vector of the reference block; determining a motion vector prediction candidate group including the motion vector prediction candidate; deriving a predicted motion vector of the target block from the motion vector prediction candidate group; determining a differential motion vector of the target block based on the predicted motion vector of the target block; determining a differential motion vector accuracy among multiple accuracy candidates; scaling the differential motion vector of the target block based on an accuracy difference between the differential motion vector accuracy and the motion vector accuracy of the predicted motion vector; and encoding the motion vector of the target block based on the predicted motion vector of the target block and the scaled differential motion vector of the target block, and encoding differential motion vector accuracy information indicating the accuracy of the differential motion vector.

[0014] According to the contents of the present disclosure, an inter-frame prediction method performed by an image decoding device includes: determining a merge candidate list of a current block including a merge candidate; selecting a merge candidate from the merge candidate list; and performing inter-frame prediction based on the motion vector and reference picture of the selected merge candidate, and wherein the merge candidate list includes spatial merge candidates and temporal merge candidates, and when the number of the spatial merge candidates and the temporal merge candidates is less than the maximum number of the merge candidate list, the merge candidate list also includes statistical merge candidates derived from previously decoded blocks belonging to the current picture and not adjacent to the current block, and when the number of the spatial merge candidates, the temporal merge candidates and the statistical merge candidates is less than the maximum number of the merge candidate list, the merge candidate list also includes a combined merge candidate derived from two pre-included merge candidates, and wherein the merge candidate list is determined when merge flag information indicates that the current block is predicted by a merge mode.

[0015] According to the contents of the present disclosure, an inter-frame prediction method performed by an image encoding device includes: determining a merge candidate list of a current block including a merge candidate; selecting a merge candidate from the merge candidate list; and performing inter-frame prediction based on the motion vector and reference picture of the selected merge candidate, and wherein the merge candidate list includes spatial merge candidates and temporal merge candidates, and when the number of the spatial merge candidates and the temporal merge candidates is less than the maximum number of the merge candidate list, the merge candidate list also includes statistical merge candidates derived from previously decoded blocks belonging to the current picture and not adjacent to the current block, and when the number of the spatial merge candidates, the temporal merge candidates and the statistical merge candidates is less than the maximum number of the merge candidate list, the merge candidate list also includes a combined merge candidate derived from two pre-included merge candidates, and wherein the merge candidate list is determined when merge flag information indicates that the current block is predicted by a merge mode.

[0016] A non-temporary computer-readable recording medium according to the present disclosure stores a bitstream generated by an inter-frame prediction method performed by an image encoding device, the method comprising: determining a merge candidate list of a current block including a merge candidate; selecting a merge candidate from the merge candidate list; and performing inter-frame prediction based on a motion vector and a reference picture of the selected merge candidate, wherein the merge candidate list includes a spatial merge candidate and a temporal merge candidate, and when the number of the spatial merge candidate and the temporal merge candidate is less than the maximum number of the merge candidate list, the merge candidate list also includes a statistical merge candidate derived from a previously decoded block belonging to the current picture and not adjacent to the current block, and when the number of the spatial merge candidate, the temporal merge candidate and the statistical merge candidate is less than the maximum number of the merge candidate list, the merge candidate list also includes a combined merge candidate derived from two pre-included merge candidates, and wherein the merge candidate list is determined when merge flag information indicates that the current block is predicted by a merge mode.

[0017] According to the contents of the present disclosure, an inter-frame prediction method performed by an image decoding device includes: deriving a motion information coding mode of a target block; specifying a reference block according to the derived motion information coding mode; configuring a motion information prediction candidate group based on the specified reference block and the motion information coding mode; deriving the motion information of the target block from the motion information prediction candidate group; and performing inter-frame prediction by using the motion information of the target block, wherein, in a case where an affine model is applied to the target block, configuring the motion information prediction candidate group includes: determining whether the affine model is applied to the reference block; in a case where the affine model is determined to be applied to the reference block, including the reference block as a candidate in the motion information prediction candidate group; constructing a combined candidate by combining multiple translation motion information; and including the combined candidate in the motion information prediction candidate group, wherein the combined candidate is included in the motion information prediction candidate group after the reference block to which the affine model is applied is included in the motion information prediction candidate group.

[0018] According to the content of the present disclosure, an inter-frame prediction method performed by an image encoding device includes: deriving motion information of a target block; performing inter-frame prediction by using the motion information of the target block; determining a motion information encoding mode of the target block; specifying a reference block according to the determined motion information encoding mode; configuring a motion information prediction candidate group based on the specified reference block and the motion information encoding mode; and encoding the motion information of the target block based on the motion information prediction candidate group, wherein, in a case where an affine model is applied to the target block, configuring the motion information prediction candidate group includes: determining whether the affine model is applied to the reference block; in a case where the affine model is determined to be applied to the reference block, including the reference block as a candidate in the motion information prediction candidate group; constructing a combined candidate by combining multiple translation motion information; and including the combined candidate in the motion information prediction candidate group, wherein the combined candidate is included in the motion information prediction candidate group after the reference block to which the affine model is applied is included in the motion information prediction candidate group.

[0019] According to a non-temporary computer-readable recording medium of the present disclosure, a bit stream generated by an inter-frame prediction method performed by an image encoding device is stored, the method comprising: deriving motion information of a target block; performing inter-frame prediction by using the motion information of the target block; determining a motion information encoding mode of the target block; specifying a reference block according to the determined motion information encoding mode; configuring a motion information prediction candidate group based on the specified reference block and the motion information encoding mode; and encoding the motion information of the target block based on the motion information prediction candidate group, wherein, in a case where an affine model is applied to the target block, the configuring the motion information prediction candidate group comprises: determining whether the affine model is applied to the reference block; in a case where the affine model is determined to be applied to the reference block, including the reference block as a candidate in the motion information prediction candidate group; constructing a combined candidate by combining a plurality of translation motion information; and including the combined candidate in the motion information prediction candidate group, wherein the combined candidate is included in the motion information prediction candidate group after the reference block to which the affine model is applied is included in the motion information prediction candidate group.

[0020] Beneficial Effects

[0021] When inter prediction according to the present disclosure is used, since a motion information prediction candidate group can be efficiently configured to result in a reduction in bits representing motion information of a target block, encoding performance can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a conceptual diagram about an image encoding and decoding system according to an embodiment of the present disclosure.

[0023] Figure 2 It is a component block diagram of an image encoding device according to an embodiment of the present disclosure.

[0024] Figure 3 It is a component block diagram of an image decoding device according to an embodiment of the present disclosure.

[0025] Figure 4 is an exemplary diagram showing various division shapes that can be obtained in the block division unit of the present disclosure.

[0026] Figure 5 It is an example of a family number used to describe the genetic traits of members in a family and the number of people in blood relationship.

[0027] Figure 6 are examples of various arrangements of related blocks in horizontal relationship with the target block.

[0028] Figure 7are examples of various arrangements of related blocks in vertical relationship to the target block.

[0029] Figure 8 are examples of various arrangements of related blocks in vertical and horizontal relationships with the target block.

[0030] Fig. 9 is an exemplary diagram about block partitioning obtained according to tree type. In this case, p to r represent examples of block partitioning about QT, BT, TT.

[0031] Fig.10 is an exemplary diagram regarding block partitioning obtained through QT, BT, and TT.

[0032] Fig.11 is an exemplary diagram for confirming the correlation between blocks based on the partitioning method and the partitioning setting.

[0033] Fig.12 is an exemplary diagram showing various cases of obtaining a prediction block through inter-frame prediction.

[0034] Fig.13 is an exemplary diagram of configuring a reference picture list according to an embodiment of the present disclosure.

[0035] Fig.14 is a conceptual diagram illustrating a non-translational motion model according to an embodiment of the present disclosure.

[0036] Fig.15 is an exemplary diagram illustrating motion prediction in units of sub-blocks according to an embodiment of the present disclosure.

[0037] Fig.16 is an exemplary diagram regarding arrangement of blocks spatially or temporally adjacent to a basic block according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] Best Practices

[0039] The image encoding / decoding method and device according to the present disclosure can: determine the motion information encoding mode of the target block; configure a motion information prediction candidate group according to the motion information encoding mode; derive the motion information of the target block by selecting one candidate in the motion information prediction candidate group; and perform inter-frame prediction on the target block based on the motion information of the target block.

[0040] Mode for Carrying Out the Invention

[0041] Various changes and modifications may be made to the present invention, and the present invention may be described with reference to different exemplary embodiments, some of which will be described and shown in the accompanying drawings. However, these embodiments are not intended to limit the present invention, but should be understood to include all modifications, equivalents and substitutions that belong to the spirit and technical scope of the present invention. In the accompanying drawings, similar reference numerals always refer to similar elements.

[0042] Although the terms first, second, etc. can be used to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the teaching of the present invention, the first element can be referred to as the second element, and similarly, the second element can also be referred to as the first element. The term "and / or" includes any combination and all combinations of multiple associated listed items.

[0043] It should be understood that when an element is referred to as being "connected to" or "coupled to" another element, the element may be directly connected or coupled to the other element or an intermediate element. In contrast, when an element is referred to as being "directly connected to" or "directly coupled to" another element, there are no intermediate elements.

[0044] The terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. Unless the context clearly indicates otherwise, as used herein, the singular forms "one", "an" and "the" are also intended to include plural forms. It should also be understood that when used in this specification, the terms "including" and / or "having" indicate the presence of stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or other features, integers, steps, operations, elements, groups of parts.

[0045] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. Terms commonly used and defined in dictionaries should be interpreted as having the same contextual meaning as in the relevant art, and unless explicitly defined in the present disclosure, they should not be interpreted as ideal or overly formal.

[0046] Typically, one or more color spaces can be configured according to the color format of the image. One or more pictures of a certain size or one or more pictures of different sizes can be configured according to the color format. In the example, in the YCbCr color configuration, color formats such as 4:4:4, 4:2:2, 4:2:0, monochrome (configured with only Y) and the like can be supported. In the example, for YCbCr 4:2:0, 1 luminance component (Y in this example) and 2 chrominance components (Cb / Cr in this example) can be configured, and in this case, the configuration ratio of the chrominance component to the luminance component can be an aspect ratio of 1:2. In the example, for 4:4:4, it can have the same aspect ratio. When a picture is configured with one or more color spaces as in the above example, the picture can be divided into each color space.

[0047] Since images can be classified into I, P, B, etc. according to image types (e.g., picture type, slice type, tile group type, tile type, brick type, etc.), image type I can represent an image that is self-encoded without using a reference image, image type P can represent an image that is encoded by using a reference image but only allows forward prediction, and image type B can represent an image that is encoded by using a reference image and allows forward / backward prediction, but some of the above types can be combined (P and B can be combined) according to the encoding settings, or image types in other configurations can be supported.

[0048] Various encoded / decoded information generated in the present disclosure may be processed explicitly or implicitly. In this regard, it is understood that explicit processing may generate selection information indicating one candidate from a plurality of candidate groups of information encoded in a sequence, slice, tile group, tile, brick, block, sub-block, etc., to store the selection information in a bitstream, and parse the relevant information in the decoder in the same units as the encoder to reconstruct the relevant information into decoded information. In this case, it is understood that implicit processing processes the encoded / decoded information in the encoder and decoder with the same processing, rules, etc.

[0049] Figure 1 is a conceptual diagram about an image encoding and decoding system according to an embodiment of the present disclosure.

[0050] Reference Figure 1, the image encoding device (105) and the decoding device (100) can be user terminals such as personal computers (PCs), notebooks, personal digital assistants (PDAs), portable multimedia players (PMPs), portable game consoles (PSPs), wireless communication terminals, smart phones or TVs, or server terminals such as application servers, service servers, etc., and the image encoding device (105) and the decoding device (100) can include various devices equipped with communication devices, for example, a communication modem for communicating with various instruments or wired and wireless communication networks, a memory (120, 125) for storing all kinds of programs and data for inter-frame prediction or intra-frame prediction to encode or decode images, or a processor (110, 115) for program operation and control by running the program, etc.

[0051] In addition, the image encoded in the bit stream by the image encoding device (105) can be sent to the image decoding device (100) in real time or non-real time through a wired and wireless communication network such as the Internet, a wireless local area network, a wireless LAN network, a wireless broadband network, or a mobile radio communication network, or through various communication interfaces such as a cable or a universal serial bus, and decoded in the image decoding device (100). And it can be reconstructed into an image and played. In addition, the image encoded in the bit stream by the image encoding device (105) can be sent from the image encoding device (105) to the image decoding device (100) through a computer-readable recording medium.

[0052] The image encoding device and the image decoding device mentioned above may be separate devices, but according to the implementation mode, they may be made into one image encoding / decoding device. In this case, some configurations of the image encoding device may be implemented as technical elements substantially the same as some configurations of the image decoding device, at least including the same structure or at least performing the same function.

[0053] Therefore, in the detailed description of the following technical elements and their operating principles, etc., repeated descriptions of the corresponding technical elements will be omitted. In addition, since the image decoding device corresponds to a computing device that applies the image encoding method performed in the image encoding device to decoding, the image encoding device will be mainly described below.

[0054] The computing device may include: a memory storing a program or software module implementing the image encoding method and / or the image decoding method; and a processor connected to the memory to execute the program. In this case, the image encoding device may be referred to as an encoder and the image decoding device may be referred to as a decoder, respectively.

[0055] Figure 2It is a component block diagram of an image encoding device according to an embodiment of the present disclosure.

[0056] Reference Figure 2 , the image encoding device (20) may include a prediction unit (200), a subtraction unit (205), a transform unit (210), a quantization unit (215), a dequantization unit (220), an inverse transform unit (225), an addition unit (230), a filter unit (235), a coded picture buffer (240) and an entropy coding unit (245).

[0057] The prediction unit (200) can be implemented by using a prediction module, a software module, and can generate a prediction block for a block to be encoded in an intra-frame prediction method or an inter-frame prediction method. The prediction unit (200) can generate a prediction block by predicting a target block to be encoded in a current image. In other words, the prediction unit (200) can generate a prediction block by using a predicted pixel value of each pixel generated by predicting the pixel value of each pixel in the target block to be encoded in the image according to intra-frame prediction or inter-frame prediction. In addition, the prediction unit (200) can cause the encoding unit to encode information about the prediction mode by sending information required for generating the prediction block, such as information about a prediction mode such as an intra-frame prediction mode or an inter-frame prediction mode, to the encoding unit. In this case, the processing unit that performs the prediction and the processing unit that determines the prediction method and specific content can be determined according to the encoding setting. For example, the prediction method, the prediction mode, etc. can be determined in the prediction unit, and the prediction can be performed in the transform unit.

[0058] The intra prediction unit may have a directional prediction mode such as a horizontal mode, a vertical mode, etc. used according to a prediction direction and a non-directional prediction mode such as DC, a plane using a method such as an average of reference pixels, interpolation, etc. The intra prediction mode candidate group may be configured by a directional mode and a non-directional mode and one of various candidates such as 35 prediction modes (33 directional prediction modes + 2 non-directional prediction modes) or 67 prediction modes (65 directional prediction modes + 2 non-directional prediction modes), 131 prediction modes (129 directional prediction modes + 2 non-directional prediction modes), etc., which may be used as a candidate group.

[0059] The intra prediction unit may include a reference pixel construction unit, a reference pixel filtering unit, a reference pixel interpolation unit, a prediction mode determination unit, a prediction block generation unit, and a prediction mode encoding unit. The reference pixel construction unit may configure pixels belonging to a block adjacent to the target block and adjacent to the target block as reference pixels for intra prediction. According to the encoding settings, one of the nearest reference pixel lines may be configured as reference pixels, another adjacent reference pixel line may be configured as a reference pixel, or multiple reference pixel lines may be configured as reference pixels. When some of the reference pixels are unavailable, the reference pixels may be generated by using available reference pixels, and when all reference pixels are unavailable, the reference pixels may be generated by using preset values ​​(e.g., a center value within a range of reference values ​​represented by a bit depth, etc.).

[0060] The reference pixel filtering unit of the intra prediction unit may perform filtering on the reference pixel to reduce residual degradation in the encoding process. In this case, the filter may be a low-pass filter, such as a 3-tap filter [1 / 4, 1 / 2, 1 / 4], a 5-tap filter [2 / 16, 3 / 16, 6 / 16, 3 / 16, 2 / 16], etc. Depending on the encoding information (e.g., the size or shape of the block, the prediction mode, etc.), it may be determined whether filtering is applied, the type of filtering, etc.

[0061] The reference pixel interpolation unit of the intra prediction unit can generate pixels in decimal units in the linear interpolation process of the reference pixels according to the prediction mode, and can determine the interpolation filter to be applied according to the encoding information. In this case, the interpolation filter may include a 4-tap cubic filter, a 4-tap Gaussian filter, a 6-tap Wiener filter, an 8-tap Kalman filter, etc. In general, interpolation is performed separately from the process of performing low-pass filtering, but the filtering process can be performed by integrating the filters applied to the two processes into one filter.

[0062] The prediction mode determination unit of the intra prediction unit may select at least one optimal prediction mode from the prediction mode candidate group by considering the encoding cost. The prediction block generation unit may generate a prediction block by using the corresponding prediction mode. In the prediction mode encoding unit, the optimal prediction mode may be encoded based on the prediction value. In this case, the prediction information may be adaptively encoded according to the case where the prediction value is accurate and the case where the prediction value is inaccurate.

[0063] In the intra prediction unit, the prediction value may be referred to as an MPM (most probable mode), and among all the modes belonging to the prediction mode candidate group, some of the modes may be configured as an MPM candidate group. The MPM candidate group may include a preset prediction mode (e.g., DC, plane, vertical, horizontal, diagonal mode, etc.) or a prediction mode of a spatially adjacent block (e.g., left block, upper block, upper left block, upper right block, lower left block, etc.), etc. In addition, a mode derived from a mode pre-included in the MPM candidate group (for a directional mode, the difference is +1, -1, etc.) may be configured as an MPM candidate group.

[0064] There may be a priority for the prediction mode used to configure the MPM candidate group. The order to be included in the MPM candidate group may be determined according to the priority. If the number of MPM candidates (determined according to the number of prediction mode candidates) is filled according to the priority, the configuration of the MPM candidate group may be terminated. In this case, the priority may be determined in the order of the prediction mode of the spatially adjacent blocks, the preset prediction mode, and the mode derived from the prediction mode pre-included in the MPM candidate group, but other modifications are also possible.

[0065] For example, the candidate group may be included in the order of left block-upper block-lower left block-upper right block-upper left block, etc. among spatially adjacent blocks, may be included in the candidate group in the order of DC mode-plane mode-vertical mode-horizontal mode, etc. among preset prediction modes, and a mode obtained by adding +1, -1, etc. to a pre-included mode may be included in the candidate group to configure a total of 6 modes as the candidate group. Alternatively, the candidate group may be included in the candidate group with a priority of left-upper-DC-plane-lower left-upper right-upper left-(left +1)-(left-1)-(upper +1), etc. to configure a total of 7 modes as the candidate group.

[0066] The subtraction unit (205) may generate a residual block by subtracting the prediction block from the target block. In other words, the subtraction unit (205) may calculate the difference between the pixel value of each pixel in the target block to be encoded and the predicted pixel value of the corresponding pixel in the prediction block generated by the prediction unit to generate a residual signal in the form of a block, i.e., a residual block. In addition, the subtraction unit (205) may generate the residual block in units other than the blocks obtained by the block division unit described later.

[0067] The transform unit (210) may transform the spatial signal into a frequency signal. The signal obtained by the transform process is called a transform coefficient. For example, a residual block having a residual signal received from the subtraction unit may be transformed into a transform block having a transform coefficient, and an input signal is determined according to an encoding setting, and the input signal is not limited to the residual signal.

[0068] The transform unit may transform the residual block through a transform scheme such as, but not limited to, Hadamard transform, discrete sine transform (DST)-based transform, or DCT-based transform. These transform schemes may be changed and modified in various ways.

[0069] At least one of the transform schemes may be supported, and at least one sub-transform scheme of each transform scheme may be supported. The sub-transform scheme may be obtained by modifying a portion of a basis vector in the transform scheme.

[0070] For example, in the case of DCT, one or more of the sub-transform schemes DCT-1 to DCT-8 may be supported, and in the case of DST, one or more of the sub-transform schemes DST-1 to DST-8 may be supported. A portion of the sub-transform schemes may be used to construct a transform scheme candidate group. For example, DCT-2, DCT-8, and DST-7 may be grouped into a candidate group for transform.

[0071] Transformation can be performed in the horizontal direction / vertical direction. For example, a one-dimensional transform can be performed in the horizontal direction by DCT-2, and a one-dimensional transform can be performed in the vertical direction by DST-7. Using a two-dimensional transform, pixel values ​​can be transformed from the spatial domain to the frequency domain.

[0072] A fixed transform scheme may be adopted, or a transform scheme may be adaptively selected according to the encoding setting. In the latter case, the transform scheme may be selected explicitly or implicitly. When the transform scheme is selected explicitly, information about the transform scheme or transform scheme set applied in each of the horizontal direction and the vertical direction may be generated, for example, at the block level. When the transform scheme is selected implicitly, the encoding setting may be defined according to the image type (I / P / B), color component, block size, block shape, block position, intra-frame prediction mode, etc., and a predetermined transform scheme may be selected according to the encoding setting.

[0073] Furthermore, some transforms may be skipped according to the encoding settings. That is, one or more of the horizontal units and the vertical units may be omitted explicitly or implicitly.

[0074] In addition, the transform unit can send information required to generate the transform block to the encoding unit, so that the encoding unit encodes the information, includes the encoded information in a bitstream, and sends the bitstream to the decoder. Therefore, the decoding unit of the decoder can parse the information from the bitstream for inverse transformation.

[0075] The quantization unit (215) may quantize the input signal. The signal obtained from the quantization is called a quantization coefficient. For example, the quantization unit (215) may obtain a quantization block having a quantization coefficient by quantizing a residual block having a residual transform coefficient received from the transform unit, and the input signal may be determined according to an encoding setting, which is not limited to the residual transform coefficient.

[0076] The quantization unit may quantize the transformed residual block through a quantization scheme such as, but not limited to, Dead Zone Uniform Threshold Quantization, a quantization weight matrix, etc. The above quantization scheme may be changed and modified in various ways.

[0077] Quantization can be skipped according to the encoding setting. For example, quantization (and dequantization) can be skipped according to the encoding setting (for example, the quantization parameter is 0, that is, a lossless compression environment). In another example, when the compression performance based on quantization is not exerted according to the characteristics of the image, the quantization process can be omitted. Quantization can be skipped in the entire area or a partial area (M / 2×N / 2, M×N / 2, or M / 2×N) of the quantization block (M×N), and the quantization skip selection information can be set explicitly or implicitly.

[0078] The quantization unit may send information required to generate the quantized blocks to the encoding unit so that the encoding unit encodes the information, includes the encoded information on the bitstream, and sends the bitstream to the decoder. Thus, the decoding unit of the decoder may parse the information from the bitstream for dequantization.

[0079] Although the above example is described under the assumption that the residual block is transformed and quantized by the transform unit and the quantization unit, the residual block having the transform coefficient may be generated by transforming the residual signal, and it may not be quantized. The residual block may be subjected to quantization only without being transformed. In addition, the residual block may be subjected to both transformation and quantization. These operations may be determined according to the encoding setting.

[0080] The dequantization unit (220) dequantizes the residual block quantized by the quantization unit (215). That is, the dequantization unit (220) generates a residual block having frequency coefficients by dequantizing the quantized frequency coefficient sequence.

[0081] The inverse transform unit (225) inversely transforms the residual block dequantized by the dequantization unit (220). That is, the inverse transform unit (225) inversely transforms the frequency coefficients of the dequantized residual block to generate a residual block having pixel values, that is, a reconstructed residual block. The inverse transform unit (225) can perform the inverse transform by inversely executing the transform scheme used by the transform unit (210).

[0082] The adding unit (230) reconstructs the target block by adding the prediction block predicted by the prediction unit (200) and the residual block restored by the inverse transform unit (225). The reconstructed target block is stored in the coded picture buffer (240) as a reference picture (or reference block) to be used as a reference picture when encoding the next block, another block or another picture of the target block later.

[0083] The filter unit (235) may include one or more post-processing filters such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF). The deblocking filter may eliminate block distortion occurring at the boundary between blocks in the reconstructed picture. The ALF may perform filtering based on a value obtained by comparing a reconstructed image after filtering the block by the deblocking filter with the original image. The SAO may reconstruct the offset difference between the original image and the residual block to which the deblocking filter is applied at a pixel level. These post-processing filters may be applied to the reconstructed picture or block.

[0084] The coded picture buffer (240) may store the block or picture reconstructed by the filter unit (235). The reconstructed block or reconstructed picture stored in the coded picture buffer (240) may be provided to the prediction unit (200) that performs intra prediction or inter prediction.

[0085] The entropy coding unit (245) scans the generated quantized frequency coefficient sequence in various scanning methods to generate a quantized coefficient sequence, encodes the quantized coefficient sequence by entropy coding, and outputs the entropy coded coefficient sequence. The scanning pattern can be configured as one of various patterns such as a zigzag, a diagonal line, and a raster. In addition, encoded data including the encoding information received from each component can be generated and the encoded data can be output as a bit stream.

[0086] Figure 3 is a block diagram showing an image decoding device according to an embodiment of the present disclosure.

[0087] Reference Figure 3 , the image decoding device (30) can be configured to include an entropy decoder (305), a prediction unit (310), a dequantization unit (315), an inverse transform unit (320), an addition unit / subtraction unit (325), a filter (330) and a decoded picture buffer (335).

[0088] Furthermore, the prediction unit (310) may be configured to include an intra prediction module and an inter prediction module.

[0089] When the image bitstream is received from the image encoding device 20 , the image bitstream may be transmitted to the entropy decoder 305 .

[0090] The entropy decoder (305) may decode the bitstream into decoded data including quantized coefficients and decoding information to be transmitted to each component.

[0091] The prediction unit (310) may generate a prediction block based on data received from the entropy decoder (305). Based on the reference images stored in the decoded picture buffer (335), a reference picture list may be generated using a default configuration scheme.

[0092] The intra prediction unit may include a reference pixel construction unit, a reference pixel filtering unit, a reference pixel interpolation unit, a prediction block generation unit, and a prediction mode decoding unit, and some may perform the same process as the encoder, and some may inversely perform a derivation process.

[0093] The dequantization unit (315) may dequantize the quantized transform coefficients provided in the bitstream and decoded by the entropy decoder (305).

[0094] The inverse transform unit (320) may generate a residual block by applying an inverse DCT, an inverse integer transform, or a similar inverse transform technique to the transform coefficients.

[0095] The dequantization unit (315) and the inverse transform unit (320) may perform the processes of the transform unit (210) and the quantization unit (215) of the above-mentioned image encoding device 20 in reverse, and may be implemented in various ways. For example, the dequantization unit (315) and the inverse transform unit (320) may use the same processes and inverse transforms shared with the transform unit (210) and the quantization unit (215), and may perform transform and quantization in reverse using information about transform and quantization processes (e.g., transform size, transform shape, quantization type, etc.) received from the image encoding device 20.

[0096] The residual block that has been dequantized and inversely transformed may be added to the prediction block derived by the prediction unit (310), thereby producing a reconstructed image block. The addition may be performed by an addition unit / subtraction unit (325).

[0097] Regarding the filters (330), when necessary, a deblocking filter may be applied to remove blocking artifacts in the reconstructed image blocks. To improve the video quality before and after the decoding process, other loop filters may be additionally used.

[0098] The reconstructed and filtered image blocks may be stored in a decoded picture buffer (335).

[0099] Although not shown in the drawings, the image encoding / decoding apparatus may further include a block dividing unit.

[0100] The block partition unit can be divided into blocks of various units and sizes. The basic coding unit (or maximum coding unit, coding tree unit CTU) can refer to the basic (or starting) unit for prediction, transformation, quantization, etc. in the image coding process. In this case, the basic coding unit can be composed of a luminance basic coding block (maximum coding block or CTB) and two basic chrominance coding blocks according to the color format (YCbCr in this example), and the size of each block can be determined according to the color format. The coding block (CB) can be obtained according to the partitioning process. The CB can be understood as a unit that is no longer further subdivided due to certain restrictions, and can be set as a starting unit for division into sub-units. In the present disclosure, the block conceptually includes various shapes such as triangles, circles, etc., without being limited to squares.

[0101] Although the following description is given in the case of one color component, it is also applicable to other color components with some modifications in proportion to the ratio according to the color format (for example, in the case of YCbCr 4:2:0, the aspect ratio of the luminance component to the chrominance component is 2:1). In addition, although block division according to other color components (for example, according to the block division result of Y in Cb / Cr) is possible, it should be understood that independent block division of each color component is also possible. In addition, although a common block division configuration can be used (considering proportionality to the length ratio), it is also necessary to consider and understand that a separate block division configuration is used according to the color component.

[0102] In the block segmentation unit, a block may be represented as M×N, and the maximum and minimum values ​​of each block may be obtained within the range. For example, if the maximum and minimum values ​​of the block are 256×256 and 4×4, respectively, a block of size 2m×2n (in this example, m and n are integers from 2 to 8), a block of size 2m×2n (in this example, m and n are integers from 2 to 128), or a block of size m×m (in this example, m and n are integers from 4 to 256) may be obtained. In this article, m and n may be equal or different, and one or more ranges such as maximum and minimum values ​​of the support block may be generated.

[0103] For example, information about the maximum size and minimum size of a block may be generated, and in some partition configurations, information about the maximum size and minimum size of a block may be generated. In the former case, the information may be range information about the maximum size and minimum size that can be generated in an image, and in the latter case, the information may be information about the maximum size and minimum size that can be generated according to some partition configurations. The partition configuration may be defined by image type (I / P / B), color component (YCbCr, etc.), block type (coding / prediction / transformation / quantization), partition type (index or type), and partition scheme (quadtree (QT), binary tree (BT), and ternary tree (TT) as tree methods, and SI2, SI3, and SI4 as type methods).

[0104] In addition, there may be constraints on the aspect ratio (block shape) that can be used for blocks, and at this point, a boundary value may be set. Only blocks that are less than or equal to / less than a boundary value k may be supported, where k may be defined based on the aspect ratio A / B (A is the longer or equal value between width and height, and B is another value). k may be a real number equal to or greater than 1, such as 1.5, 2, 3, 4, etc. As in the above example, constraints on the shape of a block in an image may be supported, or one or more constraints may be supported based on a partitioning configuration.

[0105] In summary, whether block partitioning is supported can be determined based on the above ranges and constraints and the partitioning configuration described later. For example, when the candidate (child block) split from the block (parent block) meets the supporting block condition, the partitioning can be supported, otherwise, the partitioning may not be supported.

[0106] The block segmentation unit can be configured with respect to each component of the image encoding device and the image decoding device, and the size and shape of the block can be determined in the process. Different blocks can be configured according to the components. The block may include a prediction block for a prediction unit, a transform block for a transform unit, and a quantization block for a quantization unit. However, the present disclosure is not limited to this, and the block unit may be further defined for other components. Although the shape of each of the input and output in each component is described as a rectangle in the present disclosure, the input and output of some components may have any other shape (e.g., a triangle).

[0107] The size and shape of the initial (or starting) block in the block partition unit can be determined from the higher unit. The initial block can be divided into smaller blocks. Once the optimal size and shape are determined based on the block partition, the block can be determined as the initial block with respect to the lower unit. The higher unit can be a coding block, and the lower unit can be a prediction block or a transform block, but the present disclosure is not limited to this. On the contrary, various modified examples are possible. Once the initial block of the lower unit is determined as in the above example, the partitioning process can be performed to detect the optimal size and shape of the block like the higher unit.

[0108] In summary, the block segmentation unit can segment the basic coding block (or the maximum coding block) into at least one coding block, and the coding block can be segmented into at least one prediction block / transform block / quantization block. In addition, the prediction block can be segmented into at least one transform block / quantization block, and the transform block can be segmented into at least one quantization block. Some blocks may have a dependency relationship with other blocks (i.e., defined by a higher unit and a lower unit), or may have an independent relationship with other blocks. For example, the prediction block may be a higher unit above the transform block, or may be a unit independent of the transform block. Various relationships can be established according to the type of block.

[0109] Depending on the encoding setting, it may be determined whether to combine the higher unit and the lower unit. Combination between units means that a block of the higher unit is subjected to the encoding process of the lower unit (e.g., in a prediction unit, a transform unit, an inverse transform unit, etc.) without being divided into the lower unit. That is, this may mean that the partitioning process is shared between multiple units, and the partitioning information is generated in one of the units (e.g., the higher unit).

[0110] For example, a coding block may be subjected to prediction, transformation, and inverse transformation (when a coding block is combined with a prediction block or a transform block).

[0111] For example, (when a coding block is combined with a prediction block), the coding block may be subjected to prediction, and a transform block equal to or smaller in size than the coding block may be subjected to transform and inverse transform.

[0112] For example, (when a coding block is combined with a transform block), a prediction block that is equal to or smaller in size than the coding block may be subjected to prediction, and the coding block may be subjected to transform and inverse transform.

[0113] For example, a prediction block that is equal to or smaller in size than a coding block (when a prediction block is combined with a transform block) may be subjected to prediction, transform, and inverse transform.

[0114] For example, (when there is no block combination), a prediction block equal to or smaller in size than a coding block may be subject to prediction, and a transform block equal to or smaller in size than a coding block may be subject to transform and inverse transform.

[0115] Although various cases of coding blocks, prediction blocks, and transform blocks have been described in the above examples, the present disclosure is not limited thereto.

[0116] For the combination between units, a fixed configuration can be supported in the image, or an adaptive configuration can be supported by taking into account various coding factors. The coding factors include image type, color component, coding mode (intra / inter), partition configuration, block size / shape / position, aspect ratio, prediction related information (e.g., intra prediction mode, inter prediction mode, etc.), transform related information (e.g., transform scheme selection information, etc.), quantization related information (e.g., quantization area selection information and quantization transform coefficient coding information), etc.

[0117] When a block of optimal size and shape has been detected as described above, mode information (e.g., partition information) of the block may be generated. The mode information may be included in a bitstream together with information (e.g., prediction-related information and transform-related information) generated from a component to which the block belongs and transmitted to a decoder, and may be parsed by the decoder at the same unit level for use in a video decoding process.

[0118] The partitioning scheme will now be described. Although it is assumed that the initial block is shaped like a square for ease of description, the present disclosure is not limited thereto, and the description may be applied in the same or similar manner to a case where the initial block is rectangular.

[0119] The block segmentation unit may support various types of partitioning. For example, tree-based partitioning or index-based partitioning may be supported, and other methods may also be supported. In tree-based partitioning, the partitioning type may be determined based on various types of information (e.g., information indicating whether to perform partitioning, tree type, partitioning direction, etc.), while in index-based partitioning, the partitioning type may be determined using specific index information.

[0120] Figure 4 is an exemplary diagram showing various types of partitions that can be obtained in the block segmentation unit of the present disclosure. In this example, it is assumed that Figure 4 The partition types shown in are obtained by one partition operation (or process), which should not be understood as limiting the present disclosure. The partition types can also be obtained in multiple partition operations. In addition, Figure 4 Other partitioning types not shown are also available.

[0121] (Tree-based partitioning)

[0122] In the tree-based partitioning of the present disclosure, QT, BT and TT can be supported. If one tree method is supported, this can be called single-tree partitioning, and if two or more tree methods are supported, this can be called multi-tree partitioning.

[0123] In QT, a block is partitioned into two partitions (n) in each of the horizontal and vertical directions, while in BT, a block is partitioned into two partitions (b to g) in the horizontal or vertical direction. In TT, a block is partitioned into three partitions (h to m) in the horizontal or vertical direction.

[0124] In QT, a block can be divided into four partitions (o and p) by limiting the division direction to one of the horizontal direction and the vertical direction. In addition, in BT, it is possible to support only dividing a block into partitions of equal size (b and c), only dividing a block into partitions of different sizes (d to g), or both types of division. In addition, in TT, it is possible to support dividing a block into partitions (h, j, k, and m) concentrated only in a specific direction (1:1:2 or 2:1:1 in the direction of left->right or up->down), dividing a block into partitions (i and l) concentrated in the center (1:2:1), or both types of division. In addition, it is possible to support dividing a block into four partitions in each of the horizontal and vertical directions (i.e., a total of 16 partitions) (q).

[0125] In the tree method, it is possible to support only horizontally splitting a block into z partitions (b, d, e, h, i, j, o), only vertically splitting a block into z partitions (c, f, g, k, l, m, p), or both types of partitions. In this article, z can be an integer equal to or greater than 2, for example, 2, 3 or 4.

[0126] In the present disclosure, it is assumed that partition type n is supported as QT, partition types b and c are supported as BT, and partition types i and l are supported as TT.

[0127] Depending on the encoding settings, one or more of the tree partitioning schemes may be supported. For example, QT, QT / BT, or QT / BT / TT may be supported.

[0128] In the above example, the basic tree partitioning scheme is QT, and BT and TT are included as additional partitioning schemes depending on whether other trees are supported. However, various modifications may be made. Information indicating whether other trees are supported (bt_enabled_flag, tt_enabled_flag, and bt_tt_enabled_flag, where 0 indicates non-support and 1 indicates support) may be implicitly determined according to encoding settings or explicitly determined in units such as sequences, pictures, slices, tile groups, tiles, or bricks.

[0129] The split information may include information indicating whether splitting is performed (tree_part_flag or qt_part_flag, bt_part_flag, tt_part_flag, and bt_tt_part_flag, which may have a value of 0 or 1, where 0 indicates no splitting and 1 indicates splitting). In addition, according to the splitting scheme (BT and TT), information about the splitting direction (dir_part_flag or bt_dir_part_flag, tt_dir_part_flag, and bt_tt_dir_part_flag, which have a value of 0 or 1, where 0 indicates <width / horizontal> and 1 indicates <height / vertical>) may be added. This may be information that may be generated when splitting is performed.

[0130] When supporting multi-tree partitioning, various partitioning information may be configured. For ease of description, the following description gives an example of how to configure partitioning information at one depth level (that is, recursive partitioning may be performed by setting one or more supported partitioning depths).

[0131] In Example 1, information indicating whether the division is performed is checked. If the division is not performed, the division ends.

[0132] If partitioning is performed, selection information about the partitioning type is checked (e.g., tree_idx. 0 for QT, 1 for BT, 2 for TT). The partitioning direction information is additionally checked according to the selected partitioning type, and the process proceeds to the next step (if additional partitioning is possible due to reasons such as when the partitioning depth has not yet reached the maximum value, the process starts again from the beginning, and if additional partitioning is not possible, the partitioning process ends).

[0133] In Example 2, information indicating whether the division is performed with a specific tree scheme (QT) is checked, and the process proceeds to the next step. If the division is not performed with the tree scheme (QT), information indicating whether the division is performed with another tree scheme (BT) is checked. In this case, if the division is not performed with the tree scheme, information indicating whether the division is performed with a third tree scheme (TT) is checked. If the division is not performed with the third tree scheme (TT), the division process ends.

[0134] If the division is performed with the tree scheme (QT), the process proceeds to the next step. In addition, the division is performed with the second tree scheme (BT), the division direction information is checked, and the process proceeds to the next step. If the division is performed with the third tree scheme (TT), the division direction information is checked, and the process proceeds to the next step.

[0135] In Example 3, information indicating whether the partitioning is performed in the tree scheme (QT) is checked. If the partitioning is not performed in the tree scheme (QT), information indicating whether the partitioning is performed in other tree schemes (BT and TT) is checked. If the partitioning is not performed, the partitioning process ends.

[0136] If the partitioning is performed in the tree scheme (QT), the process proceeds to the next step. In addition, the partitioning is performed in other tree schemes (BT and TT), the partitioning direction information is checked, and the process proceeds to the next step.

[0137] Although in the above examples, the tree partitioning schemes are prioritized (Example 2 and Example 3) or are not assigned priorities (Example 1), various modified examples are also available. In addition, in the above examples, the partitioning in the current step is independent of the partitioning result of the previous step. However, the partitioning in the current step can depend on the partitioning result of the previous step.

[0138] In Examples 1 to 3, if some tree partitioning schemes (QT) are performed in the previous step and thus the process proceeds to the current step, the same tree partitioning schemes (QT) may also be supported in the current step.

[0139] On the other hand, if a certain tree partitioning scheme (QT) is not performed in the previous step and another tree partitioning scheme (BT or TT) is performed, other tree partitioning schemes (BT and TT) except the certain tree partitioning scheme (QT) may be supported in the current step and subsequent steps.

[0140] In the above case, the tree configuration supported for block partitioning can be adaptive, so the aforementioned partitioning information can also be configured differently. (The example to be described later is assumed to be Example 3). That is, if the partitioning is not performed with a certain tree scheme (QT) in the previous step, the partitioning process can be performed without considering the tree scheme (QT) in the current step. In addition, the partitioning information related to a certain tree scheme (for example, information indicating whether to perform the partitioning, information about the partitioning direction, etc.) can be removed. <qt>In the information indicating whether to perform the division).

[0141] The above example relates to adaptive partition information configuration for a case where block partitioning is allowed (e.g., the block size is within the range between the maximum value and the minimum value, the partition depth of each tree scheme has not reached the maximum depth (allowed depth), etc.). Even when block partitioning is restricted (e.g., the block size does not exist in the range between the maximum value and the minimum value, the partition depth of each tree scheme has reached the maximum depth, etc.), the partition information can be adaptively configured.

[0142] As already mentioned, in the present disclosure, tree-based partitioning may be performed recursively. For example, if the partition flag of a coding block having a partition depth k is set to 0, coding block encoding is performed in the coding block having a partition depth k. If the partition flag of a coding block having a partition depth k is set to 1, coding block encoding is performed in N sub-coding blocks having a partition depth k+1 according to a partitioning scheme (where N is an integer equal to or greater than 2, such as 2, 3, and 4).

[0143] In the above process, the sub coding block may be set as the coding block (k+1) and divided into the sub coding block (k+2). The hierarchical division scheme may be determined according to division settings such as a division range and an allowed division depth.

[0144] In this case, a bitstream structure representing the split information may be selected from one or more scanning methods. For example, the bitstream of the split information may be configured based on the order of split depths or based on whether splitting is performed.

[0145] For example, in the case of splitting depth order, splitting information is obtained at the current depth level based on the initial block, and then splitting information is obtained at the next depth level. In the case of whether to perform splitting, additional splitting information is first obtained in the block split from the initial block, and other additional scanning methods can be considered.

[0146] Regardless of the tree type (or all trees), the maximum block size and the minimum block size may have a common setting, or may have separate settings for each tree, or may have a common setting for two or more trees. In this case, the maximum block size may be set to be equal to or less than the maximum coding block. If the maximum block size according to the predetermined first tree is different from the maximum coding block, the split is implicitly performed using the predetermined second tree method until the maximum block size of the first tree is reached.

[0147] Furthermore, regardless of the tree type, a common partition depth may be supported, a separate partition depth may be supported per tree, or a common partition depth may be supported for two or more trees. Alternatively, a partition depth may be supported for some trees, but not for other trees.

[0148] Explicit syntax elements for setting information may be supported, and some setting information may be determined implicitly.

[0149] (Partitioning based on index)

[0150] In the index-based partitioning of the present disclosure, a constant partition index (CSI) scheme and a variable partition index (VSI) scheme may be supported.

[0151] In the CSI scheme, k sub-blocks may be obtained by division in a predetermined direction, and k may be an integer equal to or greater than 2, such as 2, 3, or 4. Specifically, regardless of the size and shape of the block, the size and shape of the sub-block may be determined based on k. The predetermined direction may be one of the horizontal direction, the vertical direction, and the diagonal direction (upper left->lower right direction or lower left->upper right direction), or a combination of two or more of the horizontal direction, the vertical direction, and the diagonal direction (upper left->lower right direction or lower left->upper right direction).

[0152] In the index-based CSI partitioning scheme of the present disclosure, z candidates may be obtained by partitioning in the horizontal direction or the vertical direction. In this case, z may be an integer equal to or greater than 2, such as 2, 3, or 4, one of the width and height of the sub-block may be the same, and the other of the width and height of the sub-block may be the same or different. The length ratio of the width or height of the sub-block is A1:A2:...:AZ, and each of A1 to AZ may be an integer equal to or greater than 1, such as 1, 2, or 3.

[0153] In addition, candidates may be obtained by dividing into x partitions and y partitions in the horizontal direction and the vertical direction, respectively. Each of x and y may be an integer equal to or greater than 1, such as 1, 2, 3, or 4. However, candidates in which both x and y are 1 may be limited (because a already exists). Figure 4 The case where sub-blocks have the same width ratio or height ratio is shown, but candidates having different width ratios or height ratios may also be included.

[0154] In addition, the candidate may be split into w partitions in one of the diagonal directions: upper left->lower right and lower left->upper right. Herein, w may be an integer equal to or greater than 2, such as 2 or 3.

[0155] Reference Figure 4 , according to the length ratio of each sub-block, the partition type can be classified into a symmetric partition type (b) and an asymmetric partition type (d and e). In addition, the partition type can be classified into a partition type (k and m) centered in a specific direction and a center partition type (k). The partition type can be defined by various coding factors including the sub-block shape and the sub-block length ratio, and the supported partition types can be determined implicitly or explicitly according to the coding settings. Therefore, the candidate group can be determined based on the partition types supported in the index-based partitioning scheme.

[0156] In the VSI scheme, when the width w or height h of each sub-block is fixed, one or more sub-blocks can be obtained by division in a predetermined direction. In this article, each of w and h can be an integer equal to or greater than 1, for example, 1, 2, 4 or 8. Specifically, the number of sub-blocks can be determined based on the size and shape of the block and the w value or h value.

[0157] In the index-based VSI partitioning scheme of the present disclosure, the candidate may be partitioned into sub-blocks, one of the width and length of each sub-block being fixed. Alternatively, the candidate may be partitioned into sub-blocks, both the width and length of each sub-block being fixed. Since the width or height of the sub-block is fixed, equal partitioning in the horizontal direction or the vertical direction may be allowed. However, the present disclosure is not limited thereto.

[0158] In the case where the size of the block before division is M×N, if the width w of each sub-block is fixed, the height h of each sub-block is fixed, or both the width w and the height h of each sub-block are fixed, the number of sub-blocks obtained can be (M*N) / w, (M*N) / h, or (M*N) / w / h.

[0159] According to encoding settings, only one or both of the CSI scheme and the VSI scheme may be supported, and information about the supported scheme may be determined implicitly or explicitly.

[0160] The present disclosure will be described in the case where a CSI scheme is supported.

[0161] According to encoding settings, in the index-based partitioning scheme, a candidate group may be constructed to include two or more candidates.

[0162] For example, a candidate group such as {a, b, c}, {a, b, c, n}, or {a to g and n} may be formed. The candidate group may include examples of block types that are predicted to appear multiple times based on general statistical features, such as a block that is partitioned into two partitions in the horizontal direction or in the vertical direction, or in each of the horizontal and vertical directions.

[0163] Alternatively, a candidate group such as {a, b}, {a, o} or {a, b, o} or a candidate group such as {a, c}, {a, p} or {a, c, p} may be constructed. The candidate group may be an example including candidates each of which is divided into two partitions and four partitions in the horizontal direction and the vertical direction, respectively. This may be an example in which a block type predicted to be mainly divided in a specific direction is configured as a candidate group.

[0164] Alternatively, a candidate group such as {a, o, p} or {a, n, q} may be constructed. This may be an example of constructing a candidate group to include a block type predicted to be divided into many partitions smaller than the block before division.

[0165] Alternatively, a candidate group such as {a, r, s} may be constructed, and it may be an example of determining the best partitioning result that can be obtained in a rectangular shape from a block before being segmented by other methods (tree method), and constructing a non-rectangular shape as a candidate group.

[0166] As noted from the above examples, various candidate group configurations may be available, and taking into account various encoding factors, one or more candidate group configurations may be supported.

[0167] Once the candidate set is fully constructed, various partition information configurations may be available.

[0168] For example, regarding a candidate group including the undivided candidate a and the divided candidates b to s, index selection information may be generated.

[0169] Alternatively, information indicating whether the split is performed (information indicating whether the split type is a) may be generated. If the split is performed (if the split type is not a), index selection information on a candidate group including the split candidates b to s may be generated.

[0170] The split information may be configured in many other ways other than those described above. In addition to the information indicating whether to perform splitting, binary bits may be allocated to the index of each candidate in the candidate group in various ways such as fixed-length binarization, variable-length binarization, etc. If the number of candidates is 2, 1 bit may be allocated to the index selection information, and if the number of candidates is 3, one or more bits may be allocated to the index selection information.

[0171] Compared to the tree-based partitioning scheme, a partitioning type predicted to occur multiple times may be included in a candidate group in the index-based partitioning scheme.

[0172] Since the number of bits used to represent index information can increase according to the number of supported candidate groups, this scheme may be suitable for single-layer partitioning (e.g., the partition depth is limited to 0) rather than tree-based hierarchical partitioning (recursive partitioning). That is, a single partitioning operation can be supported, and the sub-blocks obtained by index-based partitioning may not be further split.

[0173] This may mean that further division into smaller blocks of the same type is impossible (for example, a coding block obtained by index-based division cannot be further split into coding blocks), and also means that further division into blocks of different types is impossible (for example, division of a coding block into a prediction block and a coding block is impossible). Obviously, the present disclosure is not limited to the above examples, and other modifications are also available.

[0174] Now, a description will be given of determining the block division setting mainly based on the block type among the encoding factors.

[0175] First, the coding blocks can be obtained in the partitioning process. A tree-based partitioning scheme can be used for the partitioning process, and the following can be generated according to the tree type: Figure 4 The partitioning type of the tree may be a (no partitioning), n (QT), b, c (BT), i or l (TT). Depending on the coding settings, various combinations of tree types such as QT / QT+BT / QT+BT+TT may be available.

[0176] The following example is a process of finally partitioning the coding block obtained in the above process into a prediction block and a transform block. It is assumed that prediction, transform, and inverse transform are performed based on the size of each partition.

[0177] In Example 1, prediction may be performed by setting the size of the prediction block equal to the size of the coding block, and transform and inverse transform may be performed by setting the size of the transform block equal to the size of the coding block (or prediction block).

[0178] In Example 2, prediction may be performed by setting the size of the prediction block equal to the size of the coding block. A transform block may be obtained by dividing the coding block (or prediction block), and transform and inverse transform may be performed based on the size of the obtained transform block.

[0179] Here, a tree-based partitioning scheme can be used for partitioning, and according to the tree type, Figure 4 The partitioning type of the tree may be a (no partitioning), n (QT), b, c (BT), i or l (TT). Depending on the coding settings, various combinations of tree types such as QT / QT+BT / QT+BT+TT may be available.

[0180] Here, the partitioning process may be an index-based partitioning scheme. The partitioning type may be obtained according to the index type, for example, Figure 4 a (no segmentation), b, c, or d. Depending on the encoding settings, various candidate groups such as {a, b, c} and {a, b, c, d} can be constructed.

[0181] In Example 3, a prediction block can be obtained by dividing the coding block, and prediction is performed based on the size of the obtained prediction block. For a transform block, its size is set to the size of the coding block, and a transform and an inverse transform can be performed on the transform block. In this example, the prediction block can be independent of the transform block.

[0182] Index-based partitioning schemes can be used for partitioning processing, and the partition type can be obtained according to the index type, for example, Figure 4 a (no segmentation), b to g, n, r, or s. Various candidate groups can be constructed according to the encoding settings, for example, {a, b, c, n}, {a to g, n}, and {a, r, s}.

[0183] In Example 4, a prediction block may be obtained by dividing a coding block, and prediction may be performed based on the size of the obtained prediction block. For a transform block, its size is set to the size of the prediction block, and transform and inverse transform may be performed on the transform block. In this example, the transform block may have a size equal to the size of the obtained prediction block, or vice versa (the size of the transform block is set to the size of the prediction block).

[0184] Tree-based partitioning schemes can be used for partitioning, and partition types can be generated based on tree types, e.g. Figure 4 The tree types are a (no split), b, c (BT), i, l (TT), or n (QT). Depending on the encoding settings, various combinations of tree types such as QT / BT / QT+BT may be available.

[0185] Here, an index-based partitioning scheme can be used for the partitioning process, and the partitioning type can be generated according to the index type, such as Figure 4 a (no segmentation), b, c, n, o, or p. Various candidate groups may be constructed according to encoding settings, such as {a, b}, {a, c}, {a, n}, {a, o}, {a, p}, {a, b, c}, {a, o, p}, {a, b, c, n}, and {a, b, c, n, p}. In addition, in the index-based partitioning scheme, the candidate group may be constructed with the VSI scheme alone or with a combination of the CSI scheme and the VSI scheme.

[0186] In Example 5, a prediction block can be obtained by dividing a coding block, and prediction is performed based on the size of the obtained prediction block. A transform block can also be obtained by dividing a coding block, and transform and inverse transform are performed based on the size of the obtained transform block. In this example, each of the prediction block and the transform block can be obtained by dividing the coding block.

[0187] Here, a tree-based partitioning scheme and an index-based partitioning scheme may be used for the partitioning process, and a candidate group may be constructed in the same manner as Example 4 or in a similar manner.

[0188] In this case, the above example is a situation that may occur depending on whether the process of dividing each block type is shared, which should not be understood as limiting the present disclosure. Various modified examples may also be available. In addition, various encoding factors and block types can be considered to determine the block division setting.

[0189] The encoding factors may include image type (I / P / B), color component (YCbCr), block size / shape / position, block aspect ratio, block type (coding block, prediction block, transform block or quantization block), partition state, encoding mode (intra / inter), prediction-related information (intra prediction mode or inter prediction mode), transform-related information (transform scheme selection information), and quantization-related information (quantization area selection information and quantization transform coefficient encoding information).

[0190] (Description of the relationship between blocks)

[0191] Figure 5 It is an example used to describe the genetic traits of members in one family and the number of families of characters in blood relations. For ease of description, horizontal relationships and vertical relationships according to a specific gender (male) will be described.

[0192] Reference Figure 5 , the target person (primarily) may have a horizontal relationship (a) with the older brother and younger brother, and a vertical relationship (b) with the grandfather, father (ancestor), son, and grandson (descendants). In this case, the characters in the horizontal relationship may have similar genetic factors, such as appearance, physique, personality, etc. Alternatively, some factors may be similar, while some factors may be dissimilar. Whether all or part of the genetic factors are similar can be determined by various environmental factors, etc. (including the mother).

[0193] This description can also be applied to vertical relationships in the same or similar manner. For example, there may be a situation where the target person's genetic factors (appearance, physique, personality) are similar to those of the father. Alternatively, some of the target person's genetic factors (appearance, physique) may be similar to those of the father, but some genetic factors (personality) may be different from those of the father (similar to the mother).

[0194] In another example, the target person may be genetically similar to the grandfather (or grandson), and the grandfather may be genetically similar to the grandson, but the degree of similarity between the characters may be determined based on the relationship between the characters. In other words, the similarity between the grandfather and the target person may be high (two generations of difference), and the similarity between the grandfather and the grandson (four generations of difference) may be low.

[0195] Generally, direct analysis may be the first priority to grasp the characteristics of the target person, but when the target person does not exist, direct analysis is impossible. In this case, as in the example, the characteristics of the target person can be approximately grasped through indirect analysis of people in various relationships. Of course, it may be necessary to analyze people who have high similarity with the target person.

[0196] This example describes the relationship between characters based on various blood relationships, which may be equally or similarly applicable to image compression encoding. In this case, block-based encoding will be used as an example. A predetermined block (target block) may be encoded using / referencing / referenced to information of blocks (related blocks) having various relationships with the target block.

[0197] Here, the information of the relevant block may be data based on a pixel value, data based on pattern information used in the encoding process, or data based on setting information used in the encoding process. For example, it may be a pixel value in the spatial domain or a coefficient value (or quantized coefficient) in the frequency domain of the relevant block. Alternatively, it may be pattern information generated in a process of encoding the relevant block. Alternatively, it may be information about a reference setting used in the encoding process of the relevant block (e.g., a reference candidate group). Here, the data based on a pixel value or the data based on pattern information may be information for configuring the reference setting.

[0198] In the present invention, the relationship between blocks (a target block and a related block described with reference to the target block) can be defined as follows.

[0199] - Horizontal relationship: when the target block has no overlapping area with the related blocks (independent relationship between blocks)

[0200] - Vertical relationship: When the target block is larger than the relevant block and contains the relevant block. Or when the target block is smaller than the relevant block and is included in the relevant block (dependency relationship between blocks)

[0201] Here, in the case of having a horizontal relationship, the related block can be located regardless of the space to which the target block belongs. That is, the related block can belong to the same space as the target block in time, or can belong to a different space from the target block in time.

[0202] Here, in the case of having a vertical relationship, the related block may be located in the space to which the target block belongs. That is, the related block does not belong to a space different in time from the target block, but according to the encoding setting, the related block may have a vertical relationship based on an area corresponding to the target block in a space different in time.

[0203] Figure 6 Examples of various arrangements of related blocks having a horizontal relationship with the target block are shown. Figure 6 , blocks placed in a horizontal relationship with the target block can be classified into blocks belonging to the same space (Curr) in time and blocks belonging to the different space (Diff) in time.

[0204] Here, even if the related block belongs to a color component different from that of the target block (X), it is considered to belong to the same space in time, but some limitations of the horizontal relationship are changed (there is a related block having the same size and position as the target block). Here, blocks belonging to the same space can be classified into blocks adjacent to (or closest to) the target block (UL, U, UR, L, DL) and blocks not adjacent to (or far away from) the target block (F0, F1, F2).

[0205] Among the blocks belonging to the same space, the block adjacent to the target block may be the block closest to the left, top, upper left, upper right, lower left, etc. This is a block that has been encoded by considering the raster scan order (or Z scan. In the case of 2×2, upper left->upper right->lower left->lower right). That is, the position of the adjacent block can be determined according to a predetermined scanning order, and changes such as removing blocks at the above positions or adding blocks at new positions (right, bottom, lower right, etc.) may occur depending on the type of scanning order (reverse Z scan <lower right->lower left->upper right->upper left>, clockwise scan <upper left->upper right->lower right->lower left>, counterclockwise scan <upper left->lower left->lower right->upper right>, etc.).

[0206] In addition, a block that is not adjacent to the target block may be a block that has already been encoded. In this case, it may belong to the same block unit (e.g., the maximum coding block) as the target block, or it may belong to the same segmentation unit (slice, tile, etc.). That is, limited settings may be supported, such as setting a range for an area that is not adjacent but may be included as a related block (based on the target block being within the range of x_offset and y_offset in the horizontal and vertical directions). In the present invention, it is assumed that a block having a horizontal relationship with the target block has already been encoded, but is not limited thereto.

[0207] For encoding of a target block, encoding information / reference settings of related blocks having a horizontal relationship may be used (referenced).

[0208] For example, the pixel values ​​of the relevant blocks can be used to generate the predicted values ​​of the target blocks. In detail, in intra prediction, the predicted values ​​of the target blocks can be obtained by applying methods such as extrapolation, interpolation, averaging, or methods such as block matching or template matching to the pixel values ​​of the relevant blocks. In addition, in inter prediction, the predicted values ​​of the target blocks can be obtained by using methods such as block matching or template matching with the pixel values ​​of the relevant blocks. In this case, in terms of finding the predicted values ​​in the same space, block matching or template matching can be defined as intra prediction (Mode_Intra), or can be defined as inter prediction (Mode_Inter) according to the prediction method, or it can be classified as other coding modes defined separately.

[0209] Here, only the pixel values ​​in the spatial domain are targeted, but all or some coefficient values ​​in the frequency domain of the relevant block may be used as prediction values ​​of the target block (ie, for prediction of frequency components).

[0210] For example, the mode information of the related block can be used to encode the mode information of the target block. Specifically, in intra-frame prediction, the prediction information (directional mode, non-directional mode, motion vector, etc.) of the related block can be used to encode the prediction information (MPM, non-MPM, etc.) of the target block. In addition, in inter-frame prediction, the prediction information (motion vector, reference picture, etc.) of the related block can be used to encode the prediction information of the target block.

[0211] Here, according to the prediction method of intra-frame prediction, not only related blocks belonging to the same space and the same color component in time as the target block can be targeted (prediction mode using extrapolation, interpolation, averaging, etc.), but also related blocks belonging to the same space and different color components in time as the target block can be targeted (prediction mode in which data of different color components are copied).

[0212] Here, in the case of inter prediction, a motion vector and a reference picture are taken as examples of prediction information, but various information such as a motion information encoding mode, a motion prediction direction, and a motion model may be included.

[0213] For example, for the reference setting of the target block, the reference setting of the related block can be used. In detail, in intra prediction, the MPM candidate group of the related block can be used as the MPM candidate group of the target block. In addition, in inter prediction, the motion prediction candidate group of the related block can be used as the motion prediction candidate group of the target block. That is, even if the candidate group is constructed based on the related block, it means that the candidate group of the related block can be used as it is without performing a separate candidate group construction in the target block.

[0214] In the above examples, description has been made under the assumption that the related block is a block having a horizontal relationship with the target block. However, there may be many related blocks in the image, and at least one related block to be used for encoding the target block must be specified. The following case classification is only an example, and it is necessary to understand that various case configurations and definitions are possible without limitation thereto.

[0215] Here, a block belonging to the same space and adjacent to the target block may be designated as a related block (case 1). Alternatively, a block belonging to the same space and not adjacent to the target block may be designated as a related block (case 2). Alternatively, a block belonging to a different space may be designated as a related block (case 3). Alternatively, all or some of the blocks belonging to (case 1) to (case 3) may be designated as related blocks.

[0216] Here, in the case of (case 1), all or some of the upper left block, upper block, upper left block, upper right block, and lower left block (L, U, UL, UR, DL) adjacent to the target block may be designated as related blocks. In the case of (case 2), one or more of the blocks not adjacent to the target block may be designated as related blocks. In the case of (case 3), all or some of the center block, left block, right block, upper block, lower block, upper left block, upper right block, lower left block, and lower right block (C, L, R, U, D, UL, UR, DL, DR) adjacent to the target block and one or more of the blocks not adjacent to the target block may be designated as related blocks.

[0217] In order to specify the relevant block, there are various verification methods. First, a block including coordinates based on a predetermined position of the target block can be specified as a relevant block. First, it is assumed that the target block (m×n) has a range of (a+m-1, b+n-1) based on the upper left coordinates of (a, b).

[0218] (Case 1 or Case 3) The C block is described under the assumption that the target block and position are the same in each picture. Therefore, the description of the block with the same letter in the same image Curr and the different image Diff may be common. However, (in the case of Case 3), the position of the C block may be different from the position of the target block in the picture, and in the example described later (ie, the block belonging to Diff), the pixel position may be changed according to the position of the C block.

[0219] A C block refers to a block including pixels at predetermined positions among the internal pixels of the target block, such as (a, b), (a, b+n-1), (a+m-1, b), (a+m-1, b+n-1), (a+m / 2-1, b+n / 2-1), (a+m / 2+1, b+n / 2-1), (a+m / 2-1, b+n / 2+1), (a+m / 2+1, b+n / 2+1). Also, an L block refers to a block including pixels at predetermined positions among the pixels outside the left boundary of the target block, such as (a-1, b), (a-1, b+n-1); and a U block refers to a block including pixels at predetermined positions among the pixels outside the upper boundary of the target block, such as (a, b-1), (a+m-1, b-1). In addition, a UL block refers to a block including pixels at predetermined positions among pixels outside the upper left boundary of the target block, such as (a-1, b-1); a UR block refers to a block including pixels at predetermined positions among pixels outside the upper right boundary of the target block, such as (a+m, b-1), (a-1, b+n); and a DL block refers to a block including pixels at predetermined positions among pixels outside the lower left boundary of the target block, such as (a-1, b+n). In the case of the right direction, the lower direction, and the lower right direction, since they can be derived from the above description, they are omitted.

[0220] In the above description, an example is given of designating a block including a pixel at one position among pixels existing in a block in each direction as a related block, but two or more related blocks may be designated in all or some directions, and two or more pixel positions may be limited therefor.

[0221] (Case 2) The Fk block (k is 0 to 2) may refer to a block of pixels separated by a predetermined length (off_x, off_y, etc.) in a predetermined direction of horizontal / vertical / diagonal lines, such as (a-off_x, b), (a, b-off_y), (a-off_x, b-off_y). Here, the predetermined length may be an integer of 1 or greater, such as 4, 8, 16, etc., and may be set based on the horizontal length and vertical length of the target block. Alternatively, it may be set based on the horizontal length and vertical length of the maximum coding block, and various examples of modifications thereof are possible. The predetermined length may be set implicitly as in the above example, or the relevant syntax element may be generated as an explicit value.

[0222] As another method of designating the related block, a block having the same / similar mode information as the encoding information of the target block may be designated as the related block. In this case, the mode information does not refer to information to be encoded (or used / predicted, candidate group constructed, etc.) at the current stage, but is information that has already been encoded or has a different attribute / meaning that has been determined. The determined information may be information determined in a previous step in the mode determination process, or information restored in a previous step in the decoding process.

[0223] For example, when the target block performs inter-frame prediction using a non-translational motion model, a block encoded using a non-translational motion model among previously encoded blocks may be designated as a related block. In this case, motion vector information according to the non-translational motion model of the designated related block may be used to construct a candidate group for motion vector prediction according to the non-translational motion model of the target block.

[0224] In the above examples, the motion model information refers to information with different attributes / meanings for checking the sameness / similarity with the target block, and the motion vector according to the motion model (non-translation motion model) of the relevant block can be information for constructing a motion vector prediction candidate group according to the motion model of the target block. In this case, the relevant block may be adjacent to the target block or may not be adjacent to the target block. If there are fewer patterns with the sameness / similarity with the target block, it may be useful when blocks in an area not adjacent to the target block are also designated as relevant blocks and used / referenced.

[0225] The following may be considered to determine a related block used / referenced for encoding a target block.

[0226] The related block can be determined based on the information to be used / referenced for encoding the target block. Here, the information to be used / referenced for encoding the target block is pixel value information for prediction, mode information related to prediction / transformation / quantization / loop filtering / entropy coding, etc., and reference candidate group information related to prediction / transformation / quantization / loop filtering / entropy coding, etc.

[0227] In addition, the relevant block can be determined based on the state information of the target block and the image information to which the target block belongs. Here, the state information of the target block can be defined based on the block size, block shape, horizontal / vertical length ratio of the block, and the position in the unit such as picture / segmentation unit (slice, tile, etc.) / maximum coding block. Here, the image information to which the target block belongs can be defined based on the image type (I / P / B), color component (Y / Cb / Cr), etc.

[0228] In addition, the related block can be determined based on the encoding information of the target block. In detail, the related block can be determined based on whether the related block has information about the sameness / similarity with the target block. Here, the information for the sameness / similarity reference can be mode information related to prediction / transformation / quantization / loop filtering / entropy coding, etc.

[0229] Taking into account all or some of the factors mentioned in the above examples, the category (the aforementioned case), the number and the position of the relevant blocks can be determined. In detail, it can be determined which category is selected, and the number and position of the relevant blocks supported in the selected category can be determined. In this case, the number of blocks supported in each category can be m, n, o, and the number of blocks supported in each category can be 0 or a greater integer, such as 0, 1, 2, 3, 5, etc.

[0230] The relevant block (block position) may be determined in a direction such as left, right, top, bottom, upper left, upper right, lower left, lower right, center, etc. centered on the target block (or a block corresponding to the target block in an image that is temporally different from the image to which the target block belongs). For example, the relevant block may be determined based on the block closest to the direction. Alternatively, the relevant block may be determined in a block that otherwise satisfies the direction and a specific range / condition. It may belong to a maximum coding block different from the maximum coding block to which the target block belongs, or may be a block at a position (e.g., left direction, top direction, upper left direction) having a difference based on the horizontal length or vertical length of the target block.

[0231] In addition, the related blocks may be determined based on the encoding order, and in this case, the encoding order may be defined by various scanning methods such as raster scanning and z scanning. As an example, a predetermined number of blocks may be included as related blocks (based on the close encoding order), and the predetermined number may be an integer greater than or equal to 0, such as 0, 1, 2, 3, etc. That is, the related blocks may be managed in a memory management method such as FIFO (First In First Out) according to the encoding order, and it may be an example of determining (decision to be included in the candidate group) related blocks that may appear in (Case 2).

[0232] When one related block is supported, this may mean that only information of the corresponding block can be used / referenced. In addition, even if multiple related blocks are supported, one piece of information can be derived based on multiple related blocks according to the encoding setting. For example, in inter-frame prediction, three related blocks such as a left block, an upper block, and an upper right block are specified for motion vector prediction to support three motion vectors, but one motion vector can be derived by a method such as the median (or average) of the motion vectors of the three blocks according to the encoding setting to be used as a motion vector prediction value of the target block.

[0233] In the case of the above example, it can be a coding setting that can reduce the occurrence of the best candidate selection information generated by supporting two or more candidates. However, it may be difficult to expect to derive a candidate with a high correlation with the coding information of the target block. Therefore, a method of constructing a candidate group with multiple candidates may be more efficient. Of course, as the number of candidates included in the candidate group increases, the amount of information used to express this may increase, so it is important to construct an efficient candidate group.

[0234] Therefore, related blocks such as the above-mentioned various examples can be supported, but it may be necessary to specify the best related block and construct a candidate group based on the best related block in consideration of general image characteristics, etc. In the present invention, it is assumed that candidates are constructed by two or more information from one or more related blocks.

[0235] The following shows a candidate group construction based on blocks having a horizontal relationship with a target block and encoding / decoding processing.

[0236] A block to be referred to for encoding information of a target block is designated (1). A candidate group is constructed in a predetermined order based on the encoding information of the designated block (2). One candidate in the candidate group is selected based on the encoding information of the target block (3). An image encoding / decoding process is performed based on the selected candidate (4).

[0237] In (1), a related block for constructing a candidate group of encoding information for a target block is specified. In this case, the related block may be a block having a horizontal relationship with the target block. It has been described that: various categories of related blocks as described above may be included; and in addition to the encoding information of the target block, various information such as state information of the target block may be considered to specify the related block.

[0238] In (2), based on the encoding information of the relevant block specified by the above process, the candidate group is constructed according to a predetermined order. Here, information obtained based on the encoding information of one relevant block may be included in the candidate group, or information obtained based on the encoding information of a plurality of relevant blocks may be included in the candidate group. In this case, for the candidate group construction order, a fixed order may be supported, or an adaptive order based on various encoding elements (elements to be considered when specifying relevant blocks, etc.) may be supported.

[0239] In (3), one candidate in the candidate group is selected based on the encoding information of the target block, and in (4), an image encoding / decoding process may be performed based on the selected candidate.

[0240] The flowchart may be a process checked and executed in units of blocks. Here, in the case of some steps (1, 2), it may be a process checked and executed in the initial stage of encoding. Even if the content is not mentioned in (1) to (4) above, since it can be derived through the various embodiments described above, its detailed description will be omitted. Generally, it is difficult to confirm in advance which block has a high correlation with the target block among blocks with a horizontal relationship. A method for pre-confirming what correlation it has with the target block among blocks with a horizontal relationship will be described later. In addition, although (3) describes the case of selecting one candidate in the candidate group, two or more information may also be selected according to the type of encoding information, encoding settings, etc., and this may be a description generally applicable to the present invention.

[0241] The target block and the related block can be one of the units of encoding / prediction / transformation / quantization / loop filtering, etc., and the target block and the related block can be set in the same unit. For example, when the target block is a coding block, the related block can also be a coding block, and modifications set in different units according to the coding setting are also possible.

[0242] Figure 7 Examples of various arrangements of related blocks in vertical relationship to a target block are shown.

[0243] Figure 7 This is a case where recursive tree-based partitioning (QT) is performed, and a description will be given centering on blocks X and A to C. Starting from a basic coding block (CTU.C block) having a partition depth of 0, blocks B (1), X (2), and A (3) are obtained as the partition depth increases. Here, blocks placed in a vertical relationship with a target block may be classified into a higher block (or ancestor block) and a lower block (or descendant block). In this case, the higher block of the target block X may be block B or block C, and the lower block may be block A. Here, the target block and the related block may be set to a higher block and a lower block, respectively, or may be set to a lower block and a higher block, respectively.

[0244] For example, in this example, if the related block has a larger value than the segmentation depth (k) of the target block, the related block may be a child block (k+1) or a grandchild block (k+2), and if the related block has a smaller value, the related block may be a parent block (k-1) or an ancestor block (k-2). That is, in addition to defining the vertical relationship between existing blocks, the detailed relationship between blocks can also be confirmed by the segmentation depth.

[0245] Here, as in the above example, one tree method is supported and comparison can be made by common segmentation depth. When multiple tree methods are supported and more than one segmentation depth according to each tree method is supported, detailed relationships can be examined by considering the number of segmentations and each segmentation depth, rather than simple classification as in the above example.

[0246] For example, if QT is performed once in a 4M×4N block, a 2M×2N block can be obtained at a (QT) partition depth of 1, but when BT is performed twice, a 2M×2N block can be obtained. However, the (BT) partition depth is the same as that which can be obtained at 2. In this case, the 4M×4N block can be a parent block (QT) or an ancestor block (BT) for the 2M×2N block; conversely, the 4M×4N block can be a child block (QT) or a grandchild block (BT) for the 2M×2N block, and the detailed relationship can be determined based on the block partition result.

[0247] In the above examples, the starting unit of the partition is the maximum coding unit (the highest ancestor block, the maximum size that a block can have. Here, it is assumed that the starting unit of the partition is a coding unit or a block. When a block is a unit of prediction or transformation, the block can be understood as a maximum prediction block or a maximum transformation block). It is not possible to have a vertical relationship exceeding the maximum coding unit, but block areas with a vertical relationship can be freely specified according to a coding setting separate from a block partition setting such as a maximum coding unit. In the present invention, it is assumed that the vertical relationship does not deviate from the maximum coding unit. In addition, the relationship between blocks will be described later with a tree-based partitioning method as the center, but it will be mentioned in advance that the same or similar applications are possible for an index-based partitioning method.

[0248] For the encoding of the target block, the encoding information / reference setting of the related block having a vertical relationship can be used. For ease of explanation, it is assumed that the target block is a lower block of the related block. In this case, the higher block is not an independent unit for performing encoding, prediction, transformation, etc., but can be a temporary unit composed of multiple lower blocks. That is, it must be understood that the higher block is a starting unit or intermediate unit of the block partitioning process for obtaining an independent unit for performing encoding (i.e., a coding / prediction / transformation block, etc. that is no longer divided).

[0249] For example, the reference pixels of the relevant block may be used to generate the prediction value of the target block. In detail, in intra prediction, the prediction value of the target block may be obtained by applying a method such as extrapolation, interpolation, averaging, or template matching to the reference pixels of the relevant block. In addition, in inter prediction, the prediction value of the target block may be obtained by using a method such as template matching with the reference pixels of the relevant block.

[0250] Here, the reference pixel of the relevant block is not a pixel located in the relevant block, but refers to a pixel obtained by assuming that the relevant block is a unit for performing intra prediction / inter prediction. That is, it means that the pixel of the block having a horizontal relationship with the relevant block (higher block) (for example, the closest to the left, top, upper left, upper right, lower left direction) is used for intra prediction / inter prediction of the target block (lower block).

[0251] For example, for the reference setting of the target block, the reference setting of the related block can be used. In detail, in intra prediction, the MPM candidate group of the related block can be used as the MPM candidate group of the target block. In addition, in inter prediction, the motion prediction candidate group of the related block can be used as the motion prediction candidate group of the target block. That is, even if the candidate group is constructed based on the related block, this means that the candidate group of the related block can be used as it is without going through the construction of a separate candidate group for the target block.

[0252] In the above examples, the prediction value and reference setting are determined based on the relevant block rather than the target block, so the problem of using information with poor correlation to the target block to perform encoding may occur. However, since the relevant information is not obtained from completely spaced-apart regions, there may be a certain degree of correlation. Since the processing to be performed in each lower block unit is integrated into a common processing in the higher block, complexity can be reduced. In addition, parallel processing of lower blocks belonging to higher blocks is possible.

[0253] In the above examples, description has been made based on the assumption that the related block is a block having a vertical relationship with the target block, but there may be many related blocks in an image, and at least one related block to be used for encoding the target block must be specified.

[0254] A description of support conditions / scopes that blocks having a vertical relationship may have is shown below, and the support conditions / scopes may be determined based on all or some of the factors mentioned in examples to be described later.

[0255] (Case 1) The higher block may be less than or equal to a predetermined first threshold size. Here, the first threshold size may mean the maximum size that the higher block may have. Here, the first threshold size may be represented by width (W), height (H), W×H, W*H, etc., and W and H may be integers of 8, 16, 32, 64 or more. Here, a block having a first threshold size may be set based on the size of a maximum coding block, a maximum prediction block, and a maximum transform block.

[0256] (Case 2) The lower block may be greater than or equal to a predetermined second threshold size. Here, the second threshold size may mean the minimum size that the lower block may have. Here, the second threshold size may be expressed in width (W), height (H), W×H, W*H, etc., and W and H may be integers of 4, 8, 16, 32 or more. However, the second threshold size may be set to be less than or equal to the first threshold size. Here, a block having a second threshold size may be set based on the size of a minimum coding block, a minimum prediction block, and a minimum transform block.

[0257] (Case 3)The minimum size of the lower block can be determined based on the size of the higher block. Here, the minimum size of the lower block (e.g., W%p or H>>q, etc.) can be determined by applying a predetermined division value (p) or a shift operation value (q. right shift operation) to at least one of the width (W) or height (H) of the higher block. Here, the division value can be an integer such as 2, 4, 8, or greater, and the shift operation value q can be an integer such as 1, 2, 3, or greater.

[0258] (Case 4)The maximum size of the higher block can be determined based on the size of the lower block. Here, the maximum size of the higher block (e.g., W*r or H<<s, etc.) can be determined by applying a predetermined multiplication value (r) or a shift operation value (s. left shift operation) to at least one of the width (W) or height (H) of the lower block. Here, the multiplication value can be an integer such as 2, 4, 8, or greater, and the shift operation value s can be an integer such as 1, 2, 3, or greater.

[0259] (Case 5)The minimum size of the lower block can be determined considering the size of the higher block and the division settings. Here, the division settings can be determined by the split type (tree type), split depth (common depth, individual depth for each tree), etc. For example, if QT is supported in the higher block, the size of the block that has been divided m times can be determined as the minimum size of the lower block; if BT (or TT) is supported, the size of the block that has been divided n times can be determined as the minimum size of the lower block; if both QT and BT (or TT) are supported, the size of the block that has been divided once can be determined as the minimum size of the lower block. Here, m to l can be integers such as 1, 2, 3, or greater. The split depth (m) of the tree that is split into smaller blocks (or a larger number) (due to one split operation) can be set to be less than or equal to the split depth (n) of the tree that is not split into smaller blocks. In addition, the split depth (l) when tree splitting is mixed can be set to be greater than or equal to the split depth (m) of the tree that is split into smaller blocks, and can be set to be less than or equal to the split depth (n) of the tree that is not split into smaller blocks.

[0260] Alternatively, the maximum size of the higher block can be determined considering the size of the lower block and the division settings. In this description, it can be derived conversely from the above examples, so the detailed description will be omitted.

[0261] The following can be considered to determine the relevant blocks that are used / referred to for encoding the target block.

[0262] The related blocks can be determined based on the information to be used / referenced for encoding the target block. Here, the information to be used / referenced for encoding the target block can be pixel value information for prediction and reference candidate group information related to prediction / transformation / quantization / loop filtering / entropy coding, etc.

[0263] In addition, the relevant block can be determined based on the state information of the target block and the image information to which the target block belongs. Here, the state information of the target block can be defined based on the block size, block shape, horizontal / vertical length ratio of the block, and the position in the unit such as picture / segmentation unit (slice, tile, etc.) / maximum coding block. Here, the image information to which the target block belongs can be defined based on the image type (I / P / B), color component (Y / Cb / Cr), etc.

[0264] The number, size, position, etc. of the related blocks may be determined by considering all or some of the factors mentioned in the above examples. In detail, it may be determined whether to use / reference information of a block having a vertical relationship for encoding the target block, and (when used / referenced) the position and size of the related block may be determined. Here, the position of the related block may be expressed by predetermined coordinates (e.g., upper left coordinates) within the block, and the size of the related block may be expressed by width (W) and height (H). The related block may be specified by combining these.

[0265] For example, if there is no special range restriction for the lower block, all lower blocks (target blocks) belonging to the relevant block can use / reference the encoding information of the relevant block. Alternatively, if the range for the lower block is limited, if it belongs to the relevant block and is larger than the size of the lower block, the encoding information of the relevant block can be used / referenced. In addition, when two or more relevant blocks are supported, selection information of the relevant block can be additionally generated.

[0266] The following shows a candidate group construction based on blocks having a vertical relationship with a target block and encoding / decoding processes.

[0267] A basic block for specifying a block to be referenced is determined (1). A block to be referenced for encoding information of a target block is specified based on the determined basic block (2). A candidate group is constructed in a predetermined order using the encoding information of the specified block (3). One candidate in the candidate group is selected based on the encoding information of the target block (4). An image encoding / decoding process is performed based on the selected candidate (5).

[0268] In (1), a block (basic block) used as a criterion for constructing a candidate group related to encoding information of the target block is determined from the target block or the first related block. Here, the first related block may be a block having a vertical relationship with the target block (here, a higher block).

[0269] In (2), when the basic block is determined by the above process, a second related block for constructing a candidate group related to the encoding information of the target block is specified. Here, the second related block may be a block having a horizontal relationship with the basic block. In (3), the candidate group is constructed in a predetermined order using the encoding information of the second related block specified by the above process. Since the blocks having a horizontal relationship can be derived not only by the candidate group construction based on the blocks having a horizontal relationship and the encoding / decoding process, but also by the above-mentioned various embodiments, the description of the blocks having a horizontal relationship will be omitted.

[0270] In (4), one candidate in the candidate group is selected based on the encoding information of the target block, and in (5), an image encoding / decoding process may be performed based on the selected candidate.

[0271] In this case, when determining the higher block and the lower block based on the vertical relationship setting between the blocks, since the process of constructing the candidate group based on the higher block is performed only once, the lower block can use / borrow them. That is, the flowchart can be a configuration that can be generated in the block in which encoding / decoding is first performed. If the construction of the candidate group based on the higher block has been completed and the basic block is determined as a related block in a certain order (2, 3), the candidate group that has already been constructed can be simply used / borrowed.

[0272] In the above example, if the candidate group is constructed based on the higher block, the lower block is described as a simple use / borrow configuration, but is not limited thereto.

[0273] For example, even if a candidate group is constructed based on a higher block, some candidates may be fixed regardless of the lower block, while other candidates may be adaptive based on the lower block. That is, this means that some candidates may be deleted / added / changed based on the lower block. In this case, the deletion / addition / change may be performed based on the position and size of the lower block within the higher block.

[0274] That is, even if a basic block is determined as a related block in some steps (2, 3), a candidate group in which all or some modifications to the candidate group that has already been constructed are reflected can be constructed.

[0275] The target block and the related block may be one of the units of encoding / prediction / transformation / quantization / loop filtering, etc., and the target block may be the same unit as the related block or a higher unit. For example, when the target block is a coding block, the related block may be a coding block, and when the target block is a coding block, the related block may be a prediction block or a transformation block.

[0276] Figure 8 Various arrangement examples of related blocks having a vertical relationship and a horizontal relationship with the target block are shown.

[0277] Figure 8 A case where recursive tree-based partitioning (quadtree) is performed is shown, and the description will be centered around blocks X, A to G, and p to t. Starting from a basic coding block (CTU) with a partition depth of 0, as the partition depth increases, blocks q / r / t (1), p / D / E / F / s (2), A / B / C / G / X (3) can be obtained. Here, it can be classified into blocks having a vertical relationship with the target block and blocks having a horizontal relationship with the target block. In this case, the related blocks (higher blocks) having a vertical relationship with the target block X can be blocks p and q (excluding CTU), and the related blocks having a horizontal relationship can be blocks A to G.

[0278] Here, in the case of some related blocks, there are not only blocks having a smaller value than the segmentation depth (k) of the target block, but also blocks having a larger value than the segmentation depth (k) of the target block. It is assumed that the present embodiment targets the above-mentioned block and the target block has the maximum segmentation depth (i.e., no longer segmented).

[0279] In block segmentation, the segmentation result is determined according to the characteristics of the image. In flat parts such as background or areas with small time changes, block segmentation can be minimized. In parts with complex patterns or areas with fast time changes, block segmentation can be performed in large quantities.

[0280] Through the above-mentioned examples of the present invention, it has been mentioned that blocks with many horizontal or vertical relationships can be used / referenced for encoding the target block. Through the examples to be described later, various examples of methods for using them more efficiently by considering the blocks with horizontal relationships and blocks with vertical relationships around the target block will be presented. Therefore, the premise is that the description of the above-mentioned horizontal and vertical relationships can be applied to the content to be described later in the same or similar manner.

[0281] Next, various cases of correlation between blocks placed in a horizontal relationship according to the type (method) of block partitioning will be described. In this case, it is assumed that QT, BT, and TT are supported as partitioning schemes, BT is symmetric partitioning (SBT), and TT is partitioning at a ratio of 1:2:1. In addition, it is assumed that blocks placed in a general horizontal relationship can have high correlation or low correlation (general relationship). In each case, it is assumed that only one of the described tree methods is supported.

[0282] Fig. 9 is an exemplary diagram of block partitioning obtained according to the tree type. Here, p to r represent examples of block partitioning of QT, BT, and TT. It is assumed that block partitioning is performed when the encoding of the block itself is not efficient due to different characteristics of some areas in the block.

[0283] In the case of QT(p), it is split into two horizontally and vertically, respectively, and it can be seen that at least one of the four sub-blocks has different characteristics. However, since 4 sub-blocks are acquired, it is impossible to know which specific sub-block has different characteristics.

[0284] For example, blocks A to D may all have different characteristics, only one of blocks A to D may have different characteristics, and the remaining blocks may have the same characteristics. Block A and block B may have the same characteristics, and block C and block D may have the same characteristics. Block A and block C may have the same characteristics, and block B and block D may have the same characteristics.

[0285] If block A and block B, block C and block D each have the same characteristics and BT is supported, horizontal segmentation can be performed in BT. However, if it is segmented by QT like p, it can be seen that block A and block B, block C and block D have different characteristics from each other. However, in this example, since it is assumed that only QT is supported, the correlation between blocks cannot be accurately determined.

[0286] If only QT is supported and QT is performed on a higher block having a depth difference of 1 partitioned around a target block (one of the sub-blocks), the correlation between the target block and related blocks (blocks in the sub-blocks except the target block) may be high or low.

[0287] BT(q) is split into two in the horizontal direction or the vertical direction, and it can be seen that the two sub-blocks (block E and block F) have different characteristics. If the characteristics between the sub-blocks are the same or similar, it can be derived as explained above under the assumption that it will not be split. If only BT is supported and BT is performed on a higher block with a split depth difference of 1 around the target block, the correlation between the target block and the related block may be low.

[0288] In the case of TT(r), TT(r) is split into three in the horizontal or vertical direction, and it can be seen that at least one of the three sub-blocks has different characteristics. However, since three sub-blocks are obtained, it is impossible to know which specific sub-block has different characteristics.

[0289] For example, blocks G through I may all have different characteristics. Blocks G and H may have the same characteristics, while block I may have different characteristics. Blocks H and I may have the same characteristics, while block G may have different characteristics.

[0290] If blocks G and H have the same characteristics, block I has different characteristics, and BT (asymmetric) is also supported, vertical partitioning (3:1) can be performed in BT, but if it is partitioned by TT like r, it can be seen that blocks G and H and block I have different characteristics. However, since it is assumed that only TT is supported in this example, the correlation between blocks cannot be accurately determined. If only TT is supported and TT is performed on a higher block with a partition depth difference of 1 around the target block, the correlation between the target block and the related block may be high or low.

[0291] It has been described above that in the case of QT and TT in the partitioning scheme, the correlation between blocks may be high or low. Assume that only the partitioning scheme (e.g., QT) is supported, and the encoding information of the remaining sub-blocks except one sub-block (e.g., block D) is known. If the encoding information of the remaining sub-blocks A to C except block D is the same or similar, block D may have different characteristics and can therefore be segmented by QT. In this way, correlation information can also be identified by checking the encoding information of the lower block belonging to the higher block. However, since the possibility of occurrence is low and this may be a complicated situation, the description of this will only mention the possibility, and its detailed description will be omitted. For reference, in the above case, since block D has a low correlation with the block having a horizontal relationship, the encoding information of the block having a vertical relationship can be used / referenced, which refers to the candidate group construction information of the higher block of block D (including blocks of A and D).

[0292] In the above examples, the correlation between sub-blocks when one tree partition is supported for a higher block has been described. That is, when the coding information of a block having a horizontal relationship is used / referenced for encoding a target block, the coding information of the related block can be used / referenced more efficiently by checking the partition state (path) between the target block and the related block. For example, when constructing a candidate group, information about a block determined to have a low correlation can be excluded, or a low priority can be assigned to a block determined to have a low correlation. Alternatively, information and reference settings of blocks having a vertical relationship can be used.

[0293] In this case, the case of supporting one tree splitting may include the case of supporting only one tree method for block splitting, and even if multiple tree splitting is supported, it may include the case of supporting only one tree splitting by the maximum value, minimum value, and maximum splitting depth of the block according to each tree method, and the block splitting setting of the tree not allowed at the previous splitting depth is not supported at the later splitting depth. That is, only QT is supported and QT can be used for splitting, and QT / BT / TT is supported, but only BT is possible at this stage, so BT can be used for splitting.

[0294] The case where the correlation between each block is checked when multiple tree partitions are supported will be described below.

[0295] Fig.10 is an exemplary diagram of the partitioning obtained due to QT, BT, TT. In this example, it is assumed that the maximum coding block is 64×64 and the minimum coding block is 8×8. In addition, it is assumed that the maximum value of the block supporting QT is 64×64, the minimum value is 16×16, the maximum value of the block supporting BT is 32×32, the minimum value is 4, 4 is one of the horizontal length / vertical length of the block, and the maximum partition depth is 3. In this case, it is assumed that the partitioning setting for TT is determined together with BT (they are used bundled). It is assumed that QT and BT (QT, BT) are supported for upper blocks (A to M), asymmetric BT (ABT) is additionally supported for the lower left block (N to P) (QT, BT<or SBT>, ABT), and TT (QT, SBT, ABT, TT) is additionally supported for the lower right block (Q to S).

[0296] (Basic blocks: blocks including B, C, D, and E)

[0297] Blocks B to E may be sub-blocks that may be obtained through QT (division once), and may be sub-blocks that may be obtained through BT (the number of divisions is 3 times, one vertical division + two horizontal divisions, or one horizontal division + two vertical divisions).

[0298] Since the maximum block size supported by QT is 16×16, blocks B to E cannot be obtained through QT, and blocks B to E may be sub-blocks partitioned by BT. In this example, horizontal partitioning (B+C / D+E) is performed in BT, and vertical partitioning is performed in each region (B / C / D / E). Therefore, as described above, blocks B and C, D, and E obtained from the same higher block (parent block, partition depth difference is 1) through BT may have low correlation.

[0299] Furthermore, since blocks B and D or blocks C and E are partitioned, the correlation between blocks B and D or between blocks C and E may be low. This is because if the correlation is high, vertical partitioning is performed only in BT and partitioning may not be performed in each region.

[0300] In the above example, it is mentioned that the correlation between sub-blocks obtained through BT is low, but this is limited to sub-blocks belonging to the same higher block (parent block, segmentation depth difference is 1), but in this example, the correlation between blocks can be checked by extending to the same higher block (ancestor block, segmentation depth difference is 2).

[0301] (Basic blocks: blocks including J, K, L, and M)

[0302] Blocks J to M may be sub-blocks that may be obtained through QT or sub-blocks that may be obtained through BT.

[0303] Since the tree method can be selected as the best segmentation type within the range that QT and BT can be supported at the same time. This is an example of performing QT. Through the above example, it is mentioned that the correlation between sub-blocks obtained by QT may be high or low. However, in this example, since multiple tree segmentations are supported, the correlation between sub-blocks can be determined differently.

[0304] Each of block J and block K or block L and block M may have a low correlation, and each of block J and block L or block K and block M may have a low correlation. This is because if the correlation between blocks adjacent in the horizontal direction and the vertical direction among blocks J to block M is high, even if QT is not performed and BT is performed, a region with a high correlation may not be segmented.

[0305] In the above example, it is mentioned that the correlation between sub-blocks obtained by QT may be high or low, but this is the case where a single tree method is supported. In this example, when multiple tree methods are supported, the correlation between blocks can be checked based on the number of various cases of block segmentation.

[0306] (Basic block: including N, O, P blocks)

[0307] Blocks N to P may be sub-blocks (in a ratio of 2:1:1) obtained through BT (one horizontal partitioning+one horizontal partitioning).

[0308] (When only symmetric BT is supported <sbt>(When) block N and block O may have high correlation or low correlation. In the case of block N, since it is acquired from a higher block by BT, the correlation with the area where block O and block P are bundled may be low. However, it cannot be said that block N has low correlation with block O and block P. Of course, block N may have low correlation with block O and block P. Alternatively, block N may have high correlation with block O and may have low correlation with block P, and block N may have low correlation with block O and may have high correlation with block P.

[0309] In this example, asymmetric BT can be supported <abt>If block N and block O have high correlation, the regions of block N and block O are grouped, and horizontal partitioning of asymmetric BT may be performed at a ratio of 3:1. However, since BT (SBT) is performed twice, the correlation between block N and block O may be low.

[0310] (Basic block: including Q, R, S blocks)

[0311] Blocks Q to S may be sub-blocks obtained through TT (one horizontal partition).

[0312] (When only TT is supported) Block Q and block S may have high correlation or low correlation. In this example, asymmetric BT can be supported. If the correlation between block Q and block R is high, the areas of block Q and block R are grouped, and horizontal segmentation of asymmetric BT can be performed at a ratio of 3:1. However, since TT is performed, the correlation between block Q and block R may be low.

[0313] As in the above example, the correlation between the target block and the related block having a horizontal relationship with the target block can be estimated based on the supported segmentation method and segmentation setting. Please see various cases about the correlation between blocks through the following content.

[0314] Fig.11 Parts (a) to (h) of FIG. 1 are exemplary diagrams for checking correlation between blocks based on a segmentation method and a segmentation setting.

[0315] Fig.11 Parts (a) to (c) of are cases where QT, BT, and TT are performed respectively and only QT, BT, and TT are supported for higher blocks. As described above, in the case of QT and TT, the correlation between adjacent blocks (A and B or A and C) in the horizontal direction or vertical direction may be high or low. This is called a general relationship. Meanwhile, in the case of BT, the correlation between blocks A and B adjacent in the horizontal direction or vertical direction may be low. This is called a special relationship.

[0316] Fig.11 Part (d) is a case where QT is performed, and it may be a case where QT and BT can be supported. In this example, the correlation between adjacent blocks (A and B or A and C) in the horizontal direction or vertical direction may be a special relationship with low correlation. If the correlation is high, BT may have been applied instead of QT.

[0317] Fig.11 Part (e) is a case where BT is performed (one vertical split + one horizontal split), and it may be a case that QT and BT can be supported. In the case of A and B, since they are split from the same higher block by BT, this may be a special relationship with low correlation. In the case of A and C, if the areas under A and C are highly correlated, they can be grouped together and splitting can be performed on them. However, due to the encoding cost, it may be a case of being split into c. Of course, since it may be other cases, A and C may be a general relationship with high or low correlation.

[0318] Fig.11 Part (f) is a case where BT (one vertical split + one vertical split) is performed, and it may be a case where BT and TT can be supported. In the case of A and C, since they are split from the same higher block by BT, this may be a special relationship with low correlation. In the case of A and B, there is a case where A and B are grouped together and split (the part corresponding to 2 in the 1:2:1 area of ​​TT), but additional splitting of the left area occurs due to TT. In this case, since it is difficult to accurately determine the correlation, A and B may have a general relationship with high or low correlation.

[0319] Fig.11 Part (g) is a case where TT (one vertical segmentation) is performed, and it may be a case where BT (or SBT), ABT, and TT may be supported. In this example, the correlation between blocks (A and B or A and C) adjacent in the horizontal direction or vertical direction may be a special relationship with low correlation. If the correlation is high, ABT may have been applied instead of TT.

[0320] Fig.11 Part (h) is the case where QT and BT are performed (two vertical divisions based on BT) and can support QT and BT <1> , and may be able to support additional TT situations <2> In the case of A and B, <1> In the case of , there is no case of splitting the basic block in a state where A and B are bound, so the correlation may be a general relationship of high or low. However, in <2> In the case of , there is a case where the basic block is split in a state where A and B are tied (after BT horizontal split, BT vertical split for the upper side, TT vertical split for the lower side), but it is still split using QT and BT. Therefore, this may be a special relationship with low correlation. In this example, in order to check the relationship between the blocks, the number of cases regarding block splits that can be obtained from the same higher block with a difference of 1 or more (the difference is 2 in this example) can be checked.

[0321] Through the above various examples, it has been confirmed that the correlation between blocks is measured to use / reference the related block having a horizontal relationship with the target block. In this case, the related block belonging to the same space as the target block and adjacent to the target block can be used as the target. In particular, the target block and the related block can be blocks adjacent to each other in the horizontal direction or the vertical direction.

[0322] The correlation between blocks can be grasped / estimated based on various information. For example, the correlation between blocks can be checked based on the state information (size, shape, position, etc.) of the target block and the related blocks.

[0323] Here, as an example of determining the correlation based on the size of the block, if the predetermined length (horizontal length or vertical length) of the related block adjacent to the boundary (horizontal or vertical) contacting the target block is greater than or equal to the predetermined length of the target block, the correlation between the blocks may be very high or slightly low, which is referred to as a general relationship A. If the predetermined length of the related block is less than the predetermined length of the target block, the correlation between the blocks may be slightly high or very low, which is referred to as a general relationship B. In this case, the horizontal length of each block may be compared when contacting the horizontal boundary (blocks on the upper side), and the vertical length of each block may be compared when contacting the vertical boundary (blocks on the left side).

[0324] Here, as an example of determining the correlation based on the shape of the block, when the target block has a rectangular shape, the correlation with the related blocks adjacent to the longer boundary of the horizontal length and the vertical length is the general relationship A, and the correlation with the related blocks adjacent to the shorter boundary can be the general relationship B.

[0325] The above description may be some examples of determining the correlation between blocks based on the state information of the blocks, and various modification examples are possible. The correlation between blocks may be grasped based not only on the state information of the blocks but also on various information.

[0326] The following describes a process of checking the correlation with the related block having a horizontal relationship with the target block.

[0327] (Check the block segmentation settings within the image)

[0328] <1> Various setting information about block segmentation within an image can be checked. Supported range information such as the unit of coding / prediction / transformation (assuming the target block is a coding unit in this example) is checked, such as the maximum block size, the minimum block size, etc. As an example, it can be checked that the maximum coding block is 128×128 and the minimum coding block is 8×8.

[0329] <2> Check the supported partitioning schemes, and check the conditions supported by each partitioning scheme, such as the maximum block size, the minimum block size, and the maximum partitioning depth. For example, the maximum size of a block supporting QT can be 128×128, the minimum size can be 16×16, the maximum size of a block supporting BT and TT can be 128×128 and 64×64 respectively, the minimum size can be 4×4 together, and the maximum partitioning depth can be 4.

[0330] <3> Check the settings to be assigned to the segmentation scheme, such as priority. For example, if segmentation is performed by QT, QT may be supported for its lower blocks (sub-blocks); if segmentation is not performed by QT and segmentation is performed by a different scheme, QT may not be supported for the lower blocks.

[0331] <4> When multiple segmentation schemes are supported, the conditions for prohibiting certain segmentations can be checked to avoid overlapping results according to the segmentation schemes. For example, after executing TT, vertical segmentation of BT can be prohibited for the center area. That is, in order to prevent the overlap of segmentation results that may occur according to each segmentation scheme, the setting information for the prohibited segmentation is checked in advance.

[0332] By inspection <1> to <4> All or some of the above and other setting information attached can be used to check block candidates that can be obtained in the image. This can be referenced to identify block candidates that can be obtained based on the target block and related blocks to be described later.

[0333] (Check the block information)

[0334] State information such as the size, shape, position, etc. of the target block and the related blocks may be checked. Here, it is checked whether the position of the block is located at the boundary or inside a unit such as a picture, a slice, a tile group, a tile, a brick, or a block.

[0335] The block in the unit can be set as the maximum coding block, and the maximum coding block can be a higher block (highest ancestor block) of the target block, but is a unit that is commonly divided in a picture unit rather than in a form obtained according to the characteristics of the image. Therefore, if it belongs to a maximum coding block different from the maximum coding block to which the target block belongs, the correlation between blocks cannot be checked, so it is necessary to check whether it belongs to the boundary.

[0336] In addition, since other units such as pictures and slices are composed of integer multiples of the maximum coding block or have settings that cannot be referenced, the process of checking the correlation can be performed only when the position of the block is not a boundary. In other words, the correlation can be checked only when the position of the block is not a boundary.

[0337] The size, shape and position of the target block and the related blocks can be used to check the segmentation status of each block or obtain the segmentation path of each block therefrom. This will be described in detail later.

[0338] (Check partition status and check common higher blocks)

[0339] The segmentation state of the target block and the related blocks may be checked. Here, the segmentation state may mean a segmentation path from which each block is obtained. A process of checking a higher block of each block may be performed by checking the segmentation state, wherein the higher block may mean a block having a vertical relationship with each block. A process of checking a higher block obtained based on state information such as the size, shape, and position of each block and the segmentation path is performed.

[0340] For example, state information such as the target block position (at the top left) is (32, 32), the width and height are 8×8, the segmentation depth is p, and the segmentation path is (QT / 1-BT / h / 0-BT / v / 1) may be obtained. State information such as the relevant block position is (24, 32), the width and height are 8×8, the segmentation depth is q, and the segmentation path is (QT / 1-BT / h / 0-BT / v / 0) may be checked. Here, the segmentation path may be expressed as a segmentation scheme / segmentation direction (h is horizontal, v is vertical, and omitted if not present) / segmentation position (0 to 3 for QT, 0 to 1 for BT, etc.).

[0341] In this example, state information such as the position of the upper block (parent block, the segmentation depth difference is 1) of the target block is (24, 32), the width and height are 16×8, the segmentation depth is p-1, and the segmentation path is (QT / 1-BT / h / 0) can be obtained. In this example, the upper block of the relevant block (segmentation depth is q-1) can be the same as the upper block of the target block.

[0342] For example, state information such as the target block position is (128, 64), the width and height are 16×32, the segmentation depth is p, and the segmentation path is (QT / 3-QT / 2-BT / v / 1) can be obtained. State information such as the relevant block position is (120, 64), the width and height are 8×32, the segmentation depth is q, and the segmentation path is (QT / 3-QT / 2-BT / v / 0-BT / v / 1) can be obtained.

[0343] In this example, state information such as the position of the higher block (parent block, the segmentation depth difference is 1) of the target block is (112, 64), the width and height are 32×32, the segmentation depth is p-1, and the segmentation path is (QT / 3-QT / 2) can be obtained.

[0344] On the other hand, state information such as the position of the higher block (parent block, the segmentation depth difference is 1) of the relevant block is (112, 64), the width and height are 16×32, the segmentation depth is q-1, and the segmentation path is (QT / 3-QT / 2-BT / v / 0) can be obtained. State information such as the position of the higher block (ancestor block, the segmentation depth difference is 2) of the relevant block is (112, 64), the width and height are 32×32, the segmentation depth is q-2, and the segmentation path is (QT / 3-QT / 2) can be obtained. It can be seen that this is the same higher block as the higher block (parent block) of the target block.

[0345] As in the above example, a process of checking an upper block of each block may be performed based on the division state, and a process of checking a common upper block may be performed.

[0346] In summary, a higher block having a segmentation depth difference of 1 or more with the target block and the related block may be checked. As an example, a higher block having a segmentation depth difference of c with the target block may be the same as a higher block having a segmentation depth difference of d with the related block. In this case, c and d may be integers of 1, 2 or more, and c and d may be the same or may be different.

[0347] Here, since it is difficult to grasp the complexity or correlation, it may not be necessary to check the higher blocks with a large segmentation depth difference. For example, when the higher blocks are common in the largest coding block, it may be difficult to check the correlation between the blocks.

[0348] To this end, there may be a predetermined first threshold (maximum value) regarding c and d, and the first threshold may be an integer of 1, 2 or more. Alternatively, there may be a predetermined second threshold related to the sum of c and d, and the second threshold may be an integer of 2, 3 or more. That is, when the threshold condition is not met, the correlation between the blocks is not checked.

[0349] There may be various methods for checking whether the target block and the related block have the same higher block. For example, it may be possible to check whether the higher blocks are the same by information about a predetermined position of the higher block or information about the width and height of the block. In detail, it may be possible to check whether the higher blocks are the same based on the upper left coordinates of the higher blocks and information about the width and height of the blocks.

[0350] (Check available candidates)

[0351] When a common higher block of the target block and the related block can be obtained, the number of cases of various block segmentation that can be obtained from the corresponding higher block can be checked. This can be checked based on the block segmentation setting and segmentation state of the higher block. Since this has been mentioned through the various examples above, a detailed description will be omitted.

[0352] (Check for dependencies)

[0353] To check the correlation between blocks, the following can be checked in this example: In this example, it is assumed that the maximum value of the segmentation depth difference is 2 for each block.

[0354] <1> If the higher block has a partition depth difference of 1 with the target block and the related block, the partitioning schemes that can be supported are checked.

[0355] When only one partitioning scheme is available, the correlation can be determined according to the partitioning scheme. If the partitioning scheme is QT or TT, the correlation can be set to a general relationship (the correlation may be high or low), and if the partitioning scheme is BT, the correlation can be set to a special relationship.

[0356] If multiple partitioning schemes are available, check whether they are partitioned in a form where the target block and the related block are combined. If so, set the relatedness to a special relationship, and if not, set the relatedness to a general relationship.

[0357] <2> In case the higher block has a partition depth difference of 2 with at least one of the target block and the related block, partition schemes that can be supported are checked.

[0358] If only one partitioning scheme is available, the dependency can be set as a general relationship regardless of the partitioning scheme.

[0359] If multiple partitioning schemes are available, check whether they are partitioned in a form where the target block and the related block are combined. If so, set the relatedness to a special relationship, and if not, set the relatedness to a general relationship.

[0360] The above examples are some cases for checking the correlation between blocks, and are not limited thereto, and various modifications and additions are possible. Information of the related blocks can be used / referenced for encoding the target block by referring to the correlation between the blocks checked through the above process.

[0361] In summary, in order to determine the correlation between the target block and the related blocks, all or some of the processes, such as (checking the block segmentation within the image), (checking the information of the block), (checking the segmentation state and checking the common higher block), (checking the candidates that can be obtained), (checking the correlation), etc., can be used, and the process of determining the correlation can be performed in various orders instead of the order listed above. In addition, it is not limited to those mentioned above, and the correlation can be determined by changing some components or combining other components. Alternatively, the process of determining the correlation of other configurations can be performed, and the information of the related blocks can be used / referenced based on the correlation between the blocks determined by the above process for encoding the target block.

[0362] The correlation determined by the above process may not be an absolute fact about the characteristics between blocks, but may be prediction information for estimating the correlation between blocks in consideration of block segmentation, etc. Therefore, since the correlation may be information referenced for constructing a candidate group of encoding information for a target block, a correlated block with a low correlation may not be included in the candidate group. Considering the possibility that the determined correlation is inaccurate, the priority regarding the candidate group construction may be set to a lower priority, or the candidate group information of a higher block with a vertical relationship may be borrowed. In addition, in the above example, although it is assumed that the correlation between blocks is classified into two types, two, three or more classification categories may be supported.

[0363] Whether to use / reference the correlation between blocks for encoding the target block (construction of the candidate group, etc.) may be explicitly determined in units such as sequences, pictures, slices, tile groups, tiles, bricks, blocks, etc., or may be implicitly determined based on encoding settings. Next, examples of various information constituting the encoding settings will be described.

[0364] Here, whether to refer to the correlation between blocks can be determined based on the information to be used / referenced for encoding the target block. For example, for constructing an intra-frame prediction mode candidate group, the correlation between blocks can be considered; while for constructing a candidate group for motion vector prediction representing a non-translational motion model in inter-frame prediction, the correlation between blocks may not be considered.

[0365] Here, it can be determined whether to refer to the correlation between blocks based on the state information of the target block and the image information to which the target block belongs. Here, the state information of the target block can be defined based on the block size, block shape, horizontal length / vertical length ratio of the block, and the position in the unit such as picture / segmentation unit (slice, tile, etc.) / maximum coding block. Here, the image information to which the target block belongs can be defined based on the image type (I / P / B), color component (Y / Cb / Cr), etc. For example, the correlation between blocks can be referenced only when it has a block size belonging to a predetermined range; and when the block size is outside the predetermined range, the correlation between blocks may not be referenced. In this case, the predetermined range can be defined by a first threshold size (minimum value) and a second threshold size (maximum value), and each threshold size can be expressed as W, H, W×H, W*H based on width (W) and height (H). W and H can be integers of 1 or greater, such as 4, 8, 16.

[0366] Here, whether to refer to the correlation between blocks can be determined according to the category of the related blocks (which can be derived from the description of the positions of the related blocks having a horizontal relationship). For example, the related block may belong to the same space as the target block and may be an adjacent block. Even if the related block belongs to the same space as the target block, in the case of non-adjacent related blocks, the correlation may not be referred to.

[0367] All or some of the factors mentioned in the above examples may be considered to define the encoding settings, and thus whether to refer to the correlation between blocks may be implicitly determined.

[0368] The following shows a candidate group construction based on blocks having a horizontal relationship with a target block and encoding / decoding processing.

[0369] A correlation between a target block and a block in which a reference possibility exists is checked (1). A block to be referred to for encoding information of the target block is specified based on the correlation (2). A candidate group is constructed in a predetermined order using the specified encoding information (3). One candidate in the candidate group is selected based on the encoding information of the target block (4). An image encoding / decoding process is performed based on the selected candidate (5).

[0370] In (1), the correlation between the target block and a block that can be considered as a related block is checked. In (2), a block of a candidate group for constructing encoding information for the target block is specified based on the correlation checked by (1). That is, it may mean that whether to include as a related block is determined based on the checked correlation result. Of course, the contents describing the specifications of the related blocks having a horizontal relationship as described above can be considered together in this example.

[0371] In (3), a candidate group is constructed according to a predetermined order using the encoding information of the related blocks specified by the above process. In this case, an adaptive order considering the related blocks included or not included by (2) can be supported. In (4), one candidate in the candidate group is selected based on the encoding information of the target block. In (5), an image encoding / decoding process can be performed based on the selected candidate.

[0372] In the flowchart, blocks determined to have low correlation based on the correlation may not be included as relevant blocks.

[0373] Another example of candidate group construction based on blocks having a horizontal relationship with a target block and encoding / decoding processes is shown below.

[0374] A block to be referred to for encoding information of a target block is designated (1). A correlation between the target block and the designated block is checked (2). A predetermined order is determined based on the encoding information of the target block and the correlation checked by (2), and a candidate group is constructed accordingly (3). One candidate in the candidate group is selected based on the encoding information of the target block (4). An image encoding / decoding process is performed based on the selected candidate (5).

[0375] In (1), a related block of a candidate group to be used for constructing encoding information for a target block is specified. In (2), a correlation between the target block and the related block is checked. In (3), an order of including the candidates may be determined based on the correlation checked by (2).

[0376] For example, if the correlation is high or low, a predetermined order may be applied. If the correlation is high, an order in which the related blocks have a high priority may be applied, and if the correlation is low, an order in which the related blocks have a low priority may be applied.

[0377] Subsequently, in (3), after the order of constructing the candidate group is determined through the above process, the candidate group can be constructed according to the order. In (4), one candidate in the candidate group is selected based on the encoding information of the target block. In (5), the image encoding / decoding process can be performed based on the selected candidate.

[0378] In the flowchart, the order of including the candidates may be adaptively set based on relevance.

[0379] An example of candidate group construction based on a block having a horizontal relationship or a vertical relationship with a target block and an encoding / decoding process is shown below. Here, it is assumed that a basic block for specifying a block to be referred to is set as a target block.

[0380] Check the correlation between the target block and the blocks in which the reference possibility exists (1).

[0381] (The number of blocks judged to have low correlation with the target block is less than a predetermined number or less)

[0382] A block of encoding information to be referred to for a target block is specified based on the correlation (2A). A candidate group is constructed in a predetermined order using the specified encoding information (3A).

[0383] (The number of blocks judged to have low correlation with the target block exceeds a predetermined number or more)

[0384] A basic block for specifying a block to be referenced is changed to a predetermined higher block (2B). A block to be referenced for encoding information of a target block is specified based on the changed basic block (3B). A candidate group is constructed in a predetermined order using the encoding information of the specified block (4B).

[0385] One of the candidate groups is selected based on the encoding information of the target block (5). Image encoding / decoding processing is performed based on the selected candidate group (6).

[0386] In the flowchart, one of the order of 1-2A-3A-5-6 (P) or the order of 1-2B-3B-4-5-6 (Q) can be determined according to the correlation determination result. Specifically, when there are few blocks determined to have low correlation with the target block, the remaining blocks except the corresponding block are designated as related blocks; when there are many blocks determined to have low correlation with the target block, by changing the basic block constructed by the candidate group to a block higher than the target block, the block having a horizontal relationship with the higher block is designated as the related block.

[0387] In the case of the P order, since some of the flowcharts in the above flowcharts in which the blocks determined to have low correlation are not included in the related blocks are the same, the detailed description is omitted. The Q order may be a configuration based on a combination of blocks having a vertical relationship and a candidate group construction. It may be an example of constructing a candidate group by changing a block unit that is the basis for the candidate group construction when a block adjacent to a target block is composed of blocks having low correlation. In the following description, the redundant description previously described is omitted and the difference is focused on.

[0388] In (2B), the block used as the candidate group standard is changed to the first relevant block. Here, the first relevant block may be a block having a vertical relationship with the target block (here, a higher block).

[0389] In (3B), a second related block for constructing a candidate group of encoding information for the target block is specified. Here, the second related block may be a block having a horizontal relationship with a basic block, which is a higher block. In (4B), the candidate group is constructed in a predetermined order using the encoding information of the second related block specified by the above process.

[0390] Here, the criterion for determining that the correlation with the target block is low is the case of segmentation by the number of blocks in the flowchart, but various criteria to be determined may be set.

[0391] Various relationships between blocks are described through various examples above, and the case of performing encoding / decoding using this is described. When describing the algorithm based on the relationship between the above blocks in various encoding / decoding processes to be described later, it should be understood that even without adding a detailed description, the settings suggested by the above various embodiments can be applied in the same or similar manner.

[0392] (Inter-frame prediction)

[0393] In the image encoding method according to an embodiment of the present disclosure, inter-frame prediction may be configured as follows. Inter-frame prediction in a prediction unit may include a reference picture construction stage, a motion estimation stage, a motion compensation stage, a motion information determination stage, and a motion information encoding stage. In addition, the image encoding device may be configured to include a reference picture construction unit, a motion estimation unit, a motion compensation unit, a motion information determination unit, and a motion information encoding unit that implement the reference picture construction stage, the motion estimation stage, the motion compensation stage, the motion information determination stage, and the motion information encoding stage. Some of the above-mentioned processes may be omitted or other processes may be added, and may be changed in other orders rather than the above-mentioned order.

[0394] In the image decoding method according to the embodiment of the present disclosure, inter-frame prediction can be configured as follows. Inter-frame prediction in the prediction unit may include a motion information decoding stage, a reference picture construction stage, and a motion compensation stage. In addition, the image decoding device may be configured to include a motion information decoding unit, a reference picture construction unit, and a motion compensation unit that implement the motion information decoding stage, the reference picture construction stage, and the motion compensation stage. Some of the above-mentioned processes may be omitted or other processes may be added, and may be changed in other orders rather than the above-mentioned order.

[0395] Since the reference picture construction unit and the motion compensation unit of the image decoding device play the same role as the configuration corresponding to the image encoding device, a detailed description is omitted, and the motion information decoding unit can be performed by reversely using the method used in the motion information encoding unit. In this case, the prediction block generated in the motion compensation unit can be sent to the addition unit.

[0396] Fig.12 is an exemplary diagram showing various cases of obtaining a prediction block through inter-frame prediction.

[0397] Reference Fig.12 , unidirectional prediction can obtain a prediction block (A. forward prediction) from a previously encoded reference picture (T-1, T-2), or can obtain a prediction block (B. backward prediction) from a subsequently encoded reference picture (T+1, T+2). Bidirectional prediction can generate a prediction block (C, D) from multiple previously encoded reference pictures (T-2 to T+2). Generally, image type P can support unidirectional prediction, while image type B can support bidirectional prediction.

[0398] As in this example, a picture referenced for encoding of the current picture may be obtained from a memory. The reference picture list may be configured to include, based on the current picture (T), reference pictures having a temporal order or display order before the current picture and reference pictures having a temporal order or display order after the current picture.

[0399] Inter prediction (E) can be performed in the current image and in images before or after the current image. When inter prediction is performed in the current image, it can be called non-directional prediction. It can be supported in image type I or image type P / B, and the supported image type can be determined according to the encoding setting. Performing inter prediction using the current image is different from performing inter prediction using other images to use temporal correlation because performing inter prediction using the current image generates a prediction block by using spatial correlation, but the prediction method between them (for example, reference image, motion vector, etc.) can be the same.

[0400] In this case, it is assumed that the image types that can perform inter-frame prediction are P and B, but inter-frame prediction can also be applied to various other image types that are added or replaced. For example, a predetermined image type may not support intra-frame prediction, may only support inter-frame prediction, may only support inter-frame prediction in a predetermined direction (backward), and may only support inter-frame prediction in a predetermined direction.

[0401] The reference picture construction unit can configure and manage reference pictures used to encode the current picture through a reference picture list. At least one reference picture list can be configured according to encoding settings (e.g., image type, prediction direction, etc.), and a prediction block can be generated from the reference pictures included in the reference picture list.

[0402] For unidirectional prediction, inter prediction may be performed using at least one reference picture included in reference picture list 0 (L0) or reference picture list 1 (L1). In addition, for bidirectional prediction, inter prediction may be performed using at least one reference picture included in a combined list (LC) generated by combining L0 and L1.

[0403] For example, unidirectional prediction can be classified into forward prediction (Pred_L0) using forward reference picture list (L0) and backward prediction (Pred_L1) using backward reference picture list (L1). Bidirectional prediction (Pred_BI) can use both forward reference picture list (L0) and backward reference picture list (L1).

[0404] Alternatively, performing two or more forward predictions by copying a forward reference picture list (L0) to a backward reference picture list (L1) may be included in the bidirectional prediction. Performing two or more backward predictions by copying a backward reference picture list (L1) to a forward reference picture list (L0) may be included in the bidirectional prediction.

[0405] The prediction direction can be represented by flag information indicating the corresponding direction (for example, inter_pred_idc. It is assumed that the value can be adjusted by predFlagL0, predFlagL1, predFlagBI). predFlagL0 indicates whether to perform forward prediction, and predFlagL1 indicates whether to perform backward prediction. Whether to perform bidirectional prediction can be indicated by predFlagBI or by activating predFlagL0 and predFlagL1 at the same time (for example, when each flag is 1).

[0406] In the present disclosure, the case of unidirectional prediction of forward prediction using a forward reference picture list is mainly described, but it can also be applied to other cases equally or by modification.

[0407] Generally, the following method can be used: the best reference picture for the picture to be encoded is determined in the encoder, and information about the corresponding reference picture is explicitly sent to the decoder. To this end, the reference picture construction unit can manage the picture list referenced for inter-frame prediction of the current picture, and set the rules for reference picture management by considering the limited memory size.

[0408] The transmitted information may be defined as an RPS (reference picture set), and pictures selected from the RPS may be classified as reference pictures and stored in a memory (or DPB), and pictures not selected from the RPS may be classified as non-reference pictures and removed from the memory after a certain period of time. A preset number of pictures (e.g., 14, 15, 16 or more pictures) may be stored in the memory, and the memory size may be set according to the level and image resolution.

[0409] Fig.13 is an exemplary diagram of configuring a reference picture list according to an embodiment of the present disclosure.

[0410] Reference Fig.13 , generally, reference pictures (T-1, T-2) existing before the current picture may be allocated to L0 and managed, and reference pictures (T+1, T+2) existing after the current picture may be allocated to L1 and managed. When the allowed number of reference pictures of L0 is not reached when L0 is configured, reference pictures of L1 may be allocated. Similarly, when the allowed number of reference pictures of L1 is not reached when L1 is configured, reference pictures of L0 may be allocated.

[0411] In addition, the current picture may be included in at least one reference picture list. For example, the current picture may be included in L0 or L1, and L0 may be configured by adding a reference picture (or current picture) whose temporal order is T to a reference picture before the current picture, and L1 may be configured by adding a reference picture whose temporal order is T to a reference picture after the current picture.

[0412] The reference picture list configuration may be determined according to encoding settings.

[0413] The current picture may be managed in a separate memory distinct from the reference picture list without including the current picture in the reference picture list, or may be managed by including the current picture in at least one reference picture list.

[0414] For example, it may be determined by a signal (curr_pic_ref_enabled_flag) indicating whether the current picture is included in the reference picture list. In this case, the signal may be information that is implicitly determined or explicitly generated.

[0415] In detail, when the signal is deactivated (e.g., curr_pic_ref_enabled_flag=0), the current picture may not be included in all reference picture lists as a reference picture, and when the signal is activated (e.g., curr_pic_ref_enabled_flag=1), it may be implicitly determined whether the current picture is included in the predetermined reference picture list (e.g., the current picture may be added only to L0, only to L1, or both), or it may be explicitly determined whether the current picture is included in the predetermined reference picture list by generating a related signal (e.g., curr_pic_ref_from_l0_flag, curr_pic_ref_from_l1_flag). The signal may be supported in units of video, sequence, picture, slice, tile group, tile, brick, etc.

[0416] In this case, you can Fig.13 The current picture is positioned at the first or last place in the reference picture list as in , and the arrangement order in the list can be determined according to encoding settings (e.g., image type information, etc.). For example, in the case of I type, the current picture can be positioned at the first place, and in the case of P / B type, the current picture can be positioned at the last place, and is not limited thereto, and other modified examples are also possible.

[0417] Alternatively, a separate reference picture memory may be supported according to a signal (ibc_enabled_flag) indicating whether block matching (or template matching) is supported in the current picture. In this case, the signal may be information that is implicitly determined or explicitly generated.

[0418] In detail, when the signal is deactivated (e.g., ibc_enabled_flag = 0), this may mean that block matching is not supported in the current picture; when the signal is activated (e.g., ibc_enabled_flag = 1), block matching may be supported in the current picture, and its reference picture memory may be supported. In this example, it is assumed that an additional memory is set, but without setting the additional memory, the current picture may also be set to directly support block matching in the existing memory supported by the current picture.

[0419] The reference picture construction unit may include a reference picture interpolation unit, and whether to perform interpolation processing on pixels in fractional units may be determined according to the interpolation accuracy of inter-frame prediction. For example, when there is interpolation accuracy in integer units, the reference picture interpolation processing may be omitted, and when there is interpolation accuracy in fractional units, the reference picture interpolation processing may be performed.

[0420] The interpolation filter used in the reference picture interpolation process may be determined according to the encoding setting, and one preset interpolation filter {e.g., DCT-IF (interpolation filter based on discrete cosine transform), etc.} may be used or one of a plurality of interpolation filters may be used, and for the former, selection information about the interpolation filter may be implicitly omitted, while for the latter, selection information about the interpolation filter may be included in units of video, sequence, picture, slice, tile group, tile, brick, etc. For the latter, information about the interpolation filter (e.g., filter coefficient information, etc.) may also be information that may be explicitly generated.

[0421] The same type of filter can be used according to the interpolation position (e.g., fractional units such as 1 / 2, 1 / 4, 1 / 8). For example, filter coefficient information can be obtained from a filter formula according to the interpolation position. Alternatively, different types of interpolation filters can be used according to the interpolation position. For example, a 6-tap Wiener filter can be applied to a 1 / 2 unit, an 8-tap Kalman filter can be applied to a 1 / 4 unit, and a linear filter can be applied to a 1 / 8 unit.

[0422] The interpolation precision (maximum precision for performing interpolation) may be determined according to encoding settings, and the interpolation precision may be a precision of integer units or fractional units (e.g., 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, etc.). In this case, the interpolation precision may be determined according to the image type, reference picture settings, supported inter-frame prediction methods, etc.

[0423] For example, the interpolation precision may be set to an integer unit in the image type I, and may be set to a decimal unit in the image type P / B. Alternatively, when the current picture is included in the reference picture, the interpolation precision may be set to one of an integer unit or a decimal unit according to the picture to be referenced. Alternatively, when block matching or template matching is supported for the current picture, the interpolation precision may be set to one of an integer unit or a decimal unit, otherwise the interpolation precision may be set to a decimal unit.

[0424] Alternatively, the interpolation process may be performed by selecting one of a plurality of interpolation precisions, and when the interpolation process according to the adaptive interpolation precision is supported (for example, when adaptive_ref_resolution_enabled_flag is 0, a preset interpolation precision is used; when adaptive_ref_resolution_enabled_flag is 1, one of a plurality of interpolation precisions is used), the precision selection information (for example, ref_resolution_idx) may be generated.

[0425] Settings and information related to interpolation precision (e.g., whether adaptive interpolation precision is supported, precision selection information, etc.) may be implicitly determined or explicitly generated, and may be included in units of video, sequence, picture, slice, tile group, tile, brick, etc. Alternatively, whether adaptive interpolation precision, precision selection information, precision candidate groups, etc. are supported may be determined based on coding settings defined by one or more coding elements of picture type, reference picture setting, supported intra prediction method, supported motion model, etc.

[0426] The motion estimation and compensation processing may be performed according to the interpolation accuracy, and the representation unit and storage unit of the motion vector may also be determined based on the interpolation accuracy.

[0427] For example, when the interpolation accuracy is 1 / 2 units, motion estimation and compensation processing can be performed in 1 / 2 units, motion vectors can be expressed in 1 / 2 units, and motion vectors can be used in encoding processing. In addition, motion vectors can be stored in 1 / 2 units and referenced in motion information encoding processing of other blocks.

[0428] Alternatively, when the interpolation accuracy is 1 / 8 unit, motion estimation and compensation processing may be performed in 1 / 8 unit, motion vectors may be expressed in 1 / 8 unit, used in encoding processing in 1 / 8 unit, and stored.

[0429] In addition, motion estimation and compensation processing and motion vectors can be performed, represented and stored in units different from the interpolation accuracy, for example, 1 / 2 units, 1 / 4 units, etc., and the motion estimation and compensation processing and motion vectors can be adaptively determined based on the inter-frame prediction method / settings (for example, motion estimation / compensation method, motion model, motion information encoding mode <contents mentioned later>, etc.).

[0430] In the example, when the interpolation accuracy is assumed to be 1 / 8 unit, for the translational motion model, the motion estimation and compensation process may be performed in 1 / 4 unit, and the motion vector may be expressed in 1 / 4 unit (this example assumes the unit in the encoding process) and stored in 1 / 8 unit. For the non-translational motion model, the motion estimation and compensation process may be performed in 1 / 8 unit, and the motion vector may be expressed in 1 / 4 unit and stored in 1 / 8 unit.

[0431] In the example, when the interpolation accuracy is assumed to be 1 / 8 unit, for block matching, motion estimation and compensation processing can be performed in 1 / 4 unit, and motion vectors can be expressed in 1 / 4 unit and stored in 1 / 8 unit. For template matching, motion estimation and compensation processing can be performed in 1 / 8 unit, and motion vectors can be expressed in 1 / 8 unit and stored in 1 / 8 unit.

[0432] In the example, when the interpolation accuracy is assumed to be 1 / 16 units, for the contention mode, the motion estimation and compensation process may be performed in 1 / 4 units, and the motion vector may be represented in 1 / 4 units and stored in 1 / 16 units. For the merge mode, the motion estimation and compensation process may be performed in 1 / 8 units, and the motion vector may be represented in 1 / 4 units and stored in 1 / 16 units. For the skip mode, the motion estimation and compensation process may be performed in 1 / 16 units, and the motion vector may be represented in 1 / 4 units and stored in 1 / 16 units.

[0433] In summary, the representation and storage units of motion estimation and compensation and motion vectors can be adaptively determined based on the inter-frame prediction method or setting and interpolation accuracy. In detail, the representation units of motion estimation and compensation and motion vectors are usually adaptively determined based on the inter-frame prediction method or setting, and the storage units of motion vectors are determined based on the interpolation accuracy, but are not limited to this, and various modified examples are possible. In addition, this example is based on one category (e.g., motion model, motion estimation / compensation method, etc.), but it is also possible to mix two or more categories and determine the settings.

[0434] In addition, as described above, the interpolation precision information has a preset value or is selected as one of a plurality of precisions, but conversely, the reference picture interpolation precision may be determined according to the motion estimation and compensation settings supported by the inter-frame prediction method or setting. For example, when the translation motion model supports up to 1 / 8 units and the non-translation motion model supports up to 1 / 16 units, the interpolation process may be performed according to the precision unit of the non-translation motion model having the highest precision.

[0435] In other words, reference picture interpolation may be performed according to the setting of precision information supported for a translational motion model, a non-translational motion model, a contention mode, a merge mode, a skip mode, etc. In this case, the precision information may be determined implicitly or explicitly, and when relevant information is explicitly generated, the precision information may be included in units of a video, a sequence, a picture, a slice, a tile group, a tile, a brick, etc.

[0436] The motion estimation unit performs a process of estimating (or searching) which block of which reference picture has a high correlation with the target block. The size and shape (M×N) of the target block where prediction is performed can be obtained from the block partitioning unit. In the example, the target block can be determined within the range of 4×4 to 128×128. Typically, inter-frame prediction is performed in units of prediction blocks, but inter-frame prediction can be performed in units of coding blocks, transform blocks, etc. according to the setting of the block partitioning unit. When performing estimation within the available range of the reference area, at least one motion estimation method can be used. The estimation order and conditions in pixels, etc. can be defined in the motion estimation method.

[0437] Motion estimation may be performed based on a motion estimation method. For example, in the case of block matching, the area to be compared for motion estimation processing may be a target block; in the case of template matching, the area to be compared for motion estimation processing may be a predetermined area (template) set based on a basic block. For the former, a block having the highest correlation with the target block may be found within an available range of a reference area, while for the latter, an area having the highest correlation with a template defined according to encoding settings may be found within an available range of a reference area.

[0438] In this case, the template can be set in one or more neighboring blocks such as the left, top, top left, top right, bottom left block, etc. based on the basic block. The neighboring block may be a block that has already been encoded. In the example, when the basic block is M×N, M×h and v×N may be configured as templates at the top and left of the basic block, respectively. In this case, the template may have settings such as a predetermined fixed area (e.g., a left block and an upper block) and a length (w, h), or may have an adaptive setting according to the encoding setting. In this case, the encoding setting may be defined by the size, shape, position, aspect ratio, image type, color component, etc. of the basic block. Alternatively, information about the template area, length, etc. may be explicitly generated in units of video, sequence, picture, slice, tile group, tile, brick block, etc., or information about the template area, length, etc. may be implicitly determined according to the encoding setting.

[0439] In this case, block matching may be one of the methods for explicitly generating all or some motion information, and template matching may be one of the methods for implicitly obtaining all or some motion information. The motion information (or type of motion information) explicitly generated or implicitly obtained in the motion estimation method may be determined by the inter-frame prediction setting, and the inter-frame prediction setting may be defined by a motion model, a motion information encoding mode, etc. In this case, for template matching, information based on the estimation start position, modification information (related to the x and y vectors) of the motion vector to complete the final estimation, etc. may be implicitly determined, but the relevant information may be explicitly generated.

[0440] Information about the supportable range of template matching can be explicitly generated or implicitly determined according to the encoding settings. In this case, the encoding settings can be defined by one or more elements such as the size, shape, position, image type, color component, etc. of the target block. In the example, template matching can be supported within the range of A×B to C×D, where A to D can be integers such as 4, 8, 16, 32, 64 or more, and A and B can be smaller than C and D or the same as C and D, respectively. In addition, the support range of template matching can belong to the support range of block matching, or an exception configuration (for example, when the minimum size of the block is small) is enabled.

[0441] When multiple motion estimation methods are supported (for example, when adaptive_motion_comp_enabled_flag is 0, one preset motion estimation method is used, and when adaptive_motion_comp_enabled_flag is 1, one of multiple motion estimation methods is used), motion estimation method selection information may be generated, and the motion estimation method selection information may be included in units of blocks. For example, when motion_comp_idx is 0 in a predetermined division unit of a picture, a slice, or the like, only block matching is supported; and when motion_comp_idx is 1, block matching and template matching are supported. Alternatively, when motion_comp_idx is 0 in a unit of a block, block matching is supported; and when motion_comp_idx is 1 in a unit of a block, template matching is supported. As above, a plurality of related selection information may be generated according to units, but the candidates indicated by the corresponding information may have different meanings.

[0442] The above example may be a configuration in which the classification (selection) of the motion estimation method in the unit of the block is first. In the example, when template matching is selected, there may be no information or merge mode to be additionally confirmed, and the competition mode may be supported as a candidate for the motion information encoding mode. In this case, when the merge mode is selected, the motion vector that completes the final estimation may be set as the motion vector of the target block; when the competition mode is selected, offset information for modifying the motion vector obtained by the final estimation in the horizontal direction or the vertical direction is additionally generated, and the motion vector obtained by adding the offset information to the motion vector obtained by the final estimation may be set as the motion vector of the target block.

[0443] Alternatively, template matching may be supported by being included as some of the inter-frame prediction configurations. In the example, a motion information coding mode for template matching may be supported, and a motion information coding mode for template matching may be supported by being included as a candidate in a motion information prediction candidate group configured in a predetermined motion information coding mode. For the former, it may be a configuration in which template matching is performed by classification (selection) of the motion information coding mode, and for the latter, it may be a configuration in which template matching is performed by selecting one candidate in a motion information prediction candidate group for representing the best motion information in a predetermined motion information coding mode.

[0444] The template in the above example can be set based on a basic block, and the basic block can be a coding block or a prediction block (or a transform block). It is described that when a coding block is determined by a block partitioning unit, the coding block can be set as a prediction block as is, or the coding block can be divided into two or more prediction blocks. In this case, when the coding block is not divided or is divided into two or more prediction blocks (for example, a non-square or right-angled triangle) to perform inter-frame prediction, the coding block is referred to as inter-frame prediction in units of sub-blocks. In other words, the basic block can be set as a coding block or a prediction block, and the template can be set based on the basic block.

[0445] Furthermore, in addition to the above-mentioned units, basic blocks for template setting may also be supported. In an example, a block in a vertical relationship or a horizontal relationship with a basic block may be a target.

[0446] In detail, it can be assumed that the coding block is the target block, and the higher block having a vertical relationship with the coding block is the related block <1> Alternatively, it may be assumed that the coding block is the target block and the block having a horizontal relationship with the coding block is the related block. In this case, the coding block may be changed into a prediction block and applied. In addition, it may be assumed that the prediction block is the target block and the coding block is the related block. In the example mentioned later, it is assumed that <1> The situation in.

[0447] A method of configuring a template by setting a basic block in a plurality of candidate blocks may be explicitly determined by information on whether it is supported (for 0, not supported; for 1, supported) in a unit of a sequence, a picture, a slice, a tile group, a tile, a brick block, a block, etc., and when the corresponding information is not confirmed, a predetermined value (0 or 1) may be assigned to it. Alternatively, whether it is supported may be implicitly determined, and whether it is supported may be determined based on encoding settings. In this case, the encoding settings may be defined by one or more elements of status information such as the size, shape, position, etc. of a block (target block), an image type (I / P / B), a color component, whether inter-frame prediction in units of sub-blocks is applied, and the like.

[0448] For example, when the size of the target block is greater than or equal to a predetermined first threshold, a method of setting a basic block in a plurality of candidate blocks may be supported. Alternatively, when the size of the target block is less than or equal to a predetermined second threshold, the method may be supported. In this case, the threshold size may be represented as W, H, W×H, W*H with width (W) and height (H), and the threshold size may be represented as a pre-promised value in the encoding / decoding device, and W and H may be integers equal to or greater than 1 such as 4, 8, 16. When the threshold size is represented as the sum of the width and height, W*H may be an integer such as 16, 32, 64 or more. The first threshold size is less than or equal to the second threshold size. In this case, when the method is not supported, it means that the basic block is set to a predetermined block (target block).

[0449] When multiple candidate blocks are supported, the candidate blocks (related blocks) may be defined differently. (When the coding block is the target block), the related block (e.g., the higher block) may be a block whose partition depth is 1 or more less than the partition depth of the coding block. Alternatively, it may be a block with a predetermined width (C) and height (D) at a predetermined upper left coordinate (e.g., located to the left or above the upper left coordinate of the target block). In this case, C and D may be integers such as 8, 16, 32 or more, and may be greater than or equal to the width and height of the target block. In addition, C and D may be determined based on information about block size (e.g., the maximum size of the transform block, the maximum size of the coding block, etc.).

[0450] When multiple candidate blocks are supported, candidate selection information may be explicitly generated and the corresponding candidate block may be set as a basic block. Alternatively, the basic block may be implicitly determined and it may be set based on coding.

[0451] For example, when it is determined that the encoding setting (first category) supports multiple candidate blocks, the relevant block may be set as a basic block, otherwise when it is determined that the encoding setting (second category) supports multiple candidate blocks, the target block may be set as a basic block.

[0452] Alternatively, when the encoding setting belongs to the first category, the target block may be set as the basic block, and when the encoding setting belongs to the second category, the first related block (e.g., the higher block) may be set as the basic block, and when the encoding setting belongs to the third category, the second related block (e.g., the neighboring block) may be set as the basic block. In addition, when some categories are replaced or added, information for selecting one of the target block and the related block may be generated.

[0453] The encoding setting may be defined by one or more elements of status information such as size, shape, aspect ratio, position, etc. of a target block, image type, color component, whether inter-frame prediction in units of subblocks is applied, and the like.

[0454] In summary, basic blocks can be set in various candidate blocks including target blocks, and basic blocks can be determined explicitly or implicitly. This example describes that basic blocks for template setting can be set differently, but the basic blocks can be applied to various situations for inter-frame prediction. In other words, the description can be applied to basic block settings for configuration of motion information prediction candidate groups, etc. in inter-frame prediction mentioned later, in the same or similar manner. However, it should be understood that the limitations on basic blocks or support ranges, etc. in this example can be set in the same or different manner.

[0455] Motion estimation can be performed based on a motion model. Motion estimation and compensation can be performed by using an additional motion model in addition to a translation motion model that considers only parallel translation. For example, motion estimation and compensation can be performed by using a motion model that considers motions such as rotation, perspective, zooming in / out, etc. as well as parallel translation. By generating a prediction block that reflects various types of motion generated according to regional features of an image, it can be supported to improve encoding performance.

[0456] Fig.14 is a conceptual diagram showing a non-translational motion model according to an embodiment of the present disclosure. Fig.14 As some examples about the affine model, an example in which motion information is represented based on motion vectors V0 and V1 at predetermined positions is described. Since motion can be represented based on multiple motion vectors, accurate motion estimation and compensation are possible.

[0457] As in the example, inter-frame prediction can be performed based on a predefined motion model, but inter-frame prediction based on an additional motion model can also be supported. In this case, it is assumed that the predefined motion model is a translation motion model and the additional motion model is an affine model, but it is not limited to this, and various modifications are possible.

[0458] The translational motion model may represent motion information based on a motion vector (assuming unidirectional prediction), assuming that a control point (base point) used to represent the motion information is an upper left coordinate, but is not limited thereto.

[0459] The non-translational motion model may be represented by motion information of various configurations. This example assumes a configuration represented by additional information (based on the upper left coordinate) in addition to one motion vector. Some motion estimation and compensation mentioned in the examples mentioned later may not be performed in units of blocks, but may be performed in units of predetermined sub-blocks. In this case, the predetermined size and position of the sub-block may be determined based on each motion model.

[0460] Fig.15 1 is an exemplary diagram showing motion prediction in sub-block units according to an embodiment of the present disclosure. In detail, motion prediction in sub-block units according to an affine model (2 motion vectors) is described.

[0461] For the translation motion model, the motion vectors in units of pixels included in the target block may be the same. In other words, it may have motion vectors applied identically in units of pixels, and perform motion estimation and compensation by using one motion vector (V0).

[0462] For a non-translational motion model (affine model), motion vectors in units of pixels included in a target block may not be the same, and a separate motion vector in units of pixels may be required. In this case, a motion vector in units of pixels or in units of sub-blocks may be derived based on a motion vector (V0, V1) at a predetermined control point position of the target block, and motion estimation and compensation may be performed by using the derived motion vector.

[0463] For example, according to V x =(V 1x -V 0x )×x / M-(V 1y -V 0y )×y / N+V 0x 、V y =(V 1y -V 0y )×x / M+(V 1x -V 0x )×y / N+V 0y The equation is used to obtain the sub-block unit or the target block {for example, (V x , V y )} is a motion vector in pixels. In this equation, V0 {in this example (V 0X , V 0Y )} is the motion vector at the upper left of the target block, and V1{in this example (V 1X , V 1Y )} is a motion vector at the upper right of the target block. By considering the complexity, motion estimation and motion compensation of the non-translational motion model can be performed in units of sub-blocks.

[0464] In this case, the size (M×N) of the sub-block may be determined according to the encoding setting, and the size (M×N) of the sub-block may have a fixed size or may be set to an adaptive size. In this case, M and N may be integers such as 2, 4, 8, 16 or more, and M and N may be the same or different. The size of the sub-block may be explicitly generated in units of a sequence, a picture, a slice, a tile group, a tile, a brick, etc. Alternatively, the size of the sub-block may be implicitly determined by a common commitment between the encoder and the decoder, or the size of the sub-block may be determined by the encoding setting.

[0465] In this case, the encoding settings can be limited by one or more elements of status information such as the size, shape, position, etc. of the target block, image type, color component, inter-frame prediction setting information (for example, motion information encoding mode, reference picture information, interpolation accuracy, motion model <type>, etc.).

[0466] The above example describes the following process: the size of the sub-block is derived according to some non-translational motion models, and motion estimation and compensation are performed based on the size of the sub-block. As in this example, motion estimation and compensation can be performed in units of sub-blocks or pixels according to the motion model, and its detailed description is omitted.

[0467] Next, various examples regarding motion information configured according to a motion model will be described.

[0468] In an example, a motion model representing a rotational motion may represent a translational motion of a block having one motion vector, and may represent a rotational motion having rotation angle information. The rotation angle information (0 degrees) may be measured based on a predetermined position (e.g., an upper left coordinate), and may be represented by k (k is an integer such as 1, 2, 3 or more) candidates having a predetermined interval (e.g., an angle difference of 0 degrees, 11.25 degrees, 22.25 degrees, etc.) within a predetermined angle range (e.g., between -90 degrees and 90 degrees).

[0469] In this case, the rotation angle information may be encoded as it is in the motion information encoding process, or may be encoded based on the motion information (eg, motion vector, rotation angle information) of a neighboring block (eg, prediction+difference information).

[0470] Alternatively, the translational motion of the block may be represented by one motion vector, and the rotational motion of the block may be represented by one or more additional motion vectors. In this case, the number of additional motion vectors may be an integer such as 1, 2 or more, and the control point of the additional motion vector may be determined in the upper right, lower left or lower right coordinates, or other coordinates in the block may be set as the control point.

[0471] In this case, the additional motion vector may be encoded as is in the motion information encoding process, or may be encoded based on motion information of a neighboring block (e.g., a motion vector according to a translational motion model or a non-translational motion model) (e.g., prediction+difference information), or may be encoded based on other motion vectors representing rotational motion of the block (e.g., prediction+difference information).

[0472] In an example, for a motion model representing a resizing or scaling motion such as a zoom-in / zoom-out situation, it may represent a translation motion of a block with one motion vector, and may represent a resizing motion with scaling information. The scaling information may be represented by scaling information representing expansion or reduction in a horizontal direction or a vertical direction based on a predetermined position (e.g., an upper left coordinate).

[0473] In this case, scaling may be applied in at least one of the horizontal direction or the vertical direction. In addition, scaling information applied in each of the horizontal direction and the vertical direction may be supported, or scaling information applied in common may be supported. The position of motion estimation and compensation may be determined by adding the width and height of the scaled block to a predetermined position (upper left coordinate).

[0474] In this case, the scaling information may be encoded as it is in the motion information encoding process, or may be encoded based on the motion information (eg, motion vector, scaling information) of a neighboring block (eg, prediction+difference information).

[0475] Alternatively, the translational motion of the block may be represented by one motion vector, and the resizing of the block may be represented by one or more additional motion vectors. In this case, the number of additional motion vectors may be an integer such as 1, 2 or more, and the control point of the additional motion vector may be determined in the upper right, lower left or lower right coordinates, or other coordinates in the block may be set as the control point.

[0476] In this case, the additional motion vector may be encoded as is in the motion information encoding process, or may be encoded based on the motion information of a neighboring block (e.g., a motion vector according to a translational motion model or a non-translational motion model) (e.g., prediction + difference information), or may be encoded based on predetermined coordinates in the block (e.g., lower right coordinates) (e.g., prediction + difference information).

[0477] The above example describes the case about representation for representing some motions, and it can be represented as motion information for representing a plurality of motions.

[0478] For example, for a motion model representing various motions or complex motions, it can represent the translation motion of a block with a motion vector, represent the rotation motion with rotation angle information, and represent the resizing with scaling information. A description of each motion can be derived from the examples mentioned above, so a detailed description is omitted.

[0479] Alternatively, it may represent the translational motion of the block with one motion vector, and represent other motions of the block with one or more additional motion vectors. In this case, the number of additional motion vectors may be an integer such as 1, 2 or more, and the control point of the additional motion vector may be determined in the upper right, lower left or lower right coordinates, or other coordinates in the block may be set as the control point.

[0480] In this case, the additional motion vector may be encoded as is in the motion information encoding process, or may be encoded based on motion information of a neighboring block (e.g., a motion vector according to a translational motion model or a non-translational motion model) (e.g., prediction+difference information), or may be encoded based on other motion vectors representing various motions of the block (e.g., prediction+difference information).

[0481] The description may be about the affine model, and mainly describes the case where the number of additional motion vectors is 1 or 2. In short, the number of motion vectors used according to the motion model may be 1, 2, 3, and it is assumed that it can be regarded as a separate motion model according to the number of motion vectors used to represent motion information. In addition, when the number of motion vectors is 1, it is assumed that it is a predefined motion model.

[0482] Multiple motion models for inter-frame prediction can be supported and can be determined by a signal indicating support for additional motion models (e.g., adaptive_motion_mode_enabled_flag). In this case, when the signal is 0, predefined motion models can be supported; and when the signal is 1, multiple motion models can be supported. The signal can be generated in units of video, sequence, picture, slice, tile group, tile, brick block, block, etc., but when it is not possible to confirm the signal separately, the value of the signal can be assigned according to a predetermined setting. Alternatively, support can be implicitly determined based on the encoding setting. Alternatively, whether it is an implicit or explicit case can be determined based on the encoding setting. In this case, the encoding setting can be defined by one or more elements such as image type, image category (e.g., for 0, general image; for 1, 360-degree image), color component, etc.

[0483] Whether to support multiple motion models may be determined in the above process. Next, it is assumed that two or more motion models are additionally supported, and it is determined that multiple motion models are to be supported in units of sequences, pictures, slices, tile groups, tiles, bricks, etc., but there may be some exceptional configurations. In the example mentioned later, it is assumed that motion models A, B, and C may be supported, and A is a motion model that is basically supported, while B and C are motion models that may be additionally supported.

[0484] Configuration information about supported motion models may be generated in the above units. In other words, supported motion model configurations such as {A, B}, {A, C}, {A, B, C} are possible.

[0485] For example, indexes (0 to 2) may be assigned to configuration candidates and selected. When index 2 is selected, a motion model configuration supporting {A, C} may be determined, and when index 3 is selected, a motion model configuration supporting {A, B, C} may be determined.

[0486] Alternatively, information indicating whether a predetermined motion model is supported may be supported separately. In other words, a flag regarding whether B is supported or a flag regarding whether C is supported may be generated. When both flags are 0, it may be the case that only A is supported. This example may be an example processed without generating information indicating whether multiple motion models are supported.

[0487] As in the above example, when a candidate group of supported motion models is configured, one motion model of the candidate group may be explicitly determined and used, or may be implicitly used in units of blocks.

[0488] Generally, the motion estimation unit may be a configuration existing in the encoding device, but it may be a configuration that may be included in the decoding device according to a prediction method (e.g., template matching, etc.). For example, for template matching, this is because the motion information of the target block can be obtained by performing motion prediction based on a neighboring template of the target block in the decoder. In this case, information related to motion estimation (e.g., motion estimation range, motion estimation method <scanning order>, etc.) may be implicitly determined or explicitly generated and may be included in units of video, sequence, picture, slice, tile group, tile, brick, etc.

[0489] The motion compensation unit performs a process for obtaining data of some blocks of some reference pictures determined as the prediction block of the target block in the motion estimation process. In detail, the prediction block of the target block can be generated from at least one region (or block) of at least one reference picture based on the motion information (e.g., reference picture information, motion vector information, etc.) obtained in the motion estimation process.

[0490] Motion compensation may be performed based on the following motion compensation method.

[0491] For block matching, the motion vector (V x , V y ) and the upper left coordinate of the target block (P x , P y ) to obtain the coordinates (P x +V x , P y +V y ), the data in the area corresponding to the right M and the bottom N can be compensated as the prediction block of the target block.

[0492] For template matching, the motion vector (V x , V y ) and the upper left coordinate of the target block (P x , P y ) to obtain the coordinates (P x +V x , P y +V y ), the data in the area corresponding to the right M and the bottom N can be compensated as the prediction block of the target block.

[0493] Furthermore, motion compensation may be performed based on the following motion model.

[0494] For the translational motion model, a motion vector (V x , V y ) and the upper left coordinate of the target block (P x , P y ) to obtain the coordinates (P x +V x , P y +V y ), the data in the area corresponding to the right M and the bottom N can be compensated as the prediction block of the target block.

[0495] For the non-translational motion model, we use a multiple motion vector (V) of the target block (M×N) obtained explicitly in the reference picture. 0x , V 0y )、(V 1x , V 1y ) implicitly obtains the motion vectors (V mx , V my ) and the upper left coordinate of each sub-block (P mx , P ny ) to obtain the coordinates (P mx +V nx , P my +V ny ), the data in the area corresponding to the right M / m and the lower N / n can be compensated as the prediction block of the target block. In other words, the prediction block of the target block can be used for compensation by collecting the prediction blocks of the sub-blocks.

[0496] In the motion information determination unit, a process for selecting the best motion information of the target block may be performed. Generally, the best mode information may be determined from the perspective of encoding cost by using block distortion {e.g., distortion of the target block and the reconstructed block, SAD (sum of absolute difference), SSD (sum of square difference), etc.} and a rate-distortion method taking into account the amount of bits generated according to the corresponding motion information. The prediction block generated based on the motion information determined in the above process may be sent to the subtraction unit and the addition unit. In addition, it may be a configuration that may be included in the decoding device according to some prediction methods (e.g., template matching, etc.), in which case it may be determined based on block distortion.

[0497] For the motion information determination unit, setting information related to inter-frame prediction such as motion compensation method, motion model, etc. may be considered. For example, when multiple motion compensation methods are supported, motion compensation method selection information and the resulting motion vector, reference picture information, etc. may be the optimal motion information. Alternatively, when multiple motion models are supported, motion model selection information and the resulting motion vector, reference picture information, etc. may be the optimal motion information.

[0498] In the motion information encoding unit, the motion information of the target block obtained in the motion information determination process may be encoded. In this case, the motion information may be configured with information about an image and a region referenced for prediction of the target block. In detail, the motion information may be configured with information about the referenced image (e.g., reference image information, etc.) and information about the referenced region (e.g., motion vector information, etc.).

[0499] In addition, setting information (or selection information, etc., such as motion estimation / compensation method, selection information of motion model, etc.) related to inter-frame prediction may also be included in the motion information of the target block. Information about the referenced image and region (e.g., the number of motion vectors, etc.) may be configured based on the settings related to inter-frame prediction.

[0500] The motion information may be encoded by configuring information about a referenced image and a referenced region as one combination, and the combination of the information about the referenced image and the referenced region may be configured as a motion information encoding mode.

[0501] In this case, information about the referenced image and the referenced area can be obtained based on neighboring blocks or predetermined information (e.g., images encoded before or after the current picture, zero motion vector, etc.), and the neighboring blocks can be blocks (related blocks) having a horizontal relationship with the basic block. In other words, when the category is classified as a block that belongs to the same space as the basic block and is closest to the target block<inter_blk_A> , blocks that belong to the same space as the basic block and are distant neighbors of the target block<inter_blk_B> and blocks that do not belong to the same space as the basic block<inter_blk_C> When , blocks belonging to one or more categories among them can be designated as related blocks.

[0502] For example, the motion information of the target block may be encoded based on the motion information of the related block or the reference picture information, and the motion information of the target block may be encoded based on information derived from the motion information of the related block or the reference picture information (or information through an intermediate value, a transform process, etc.). In other words, for the motion information of the target block, prediction may be performed from neighboring blocks to encode information thereon.

[0503] The motion information of the target block may be predicted and encoded, or the motion information itself may be encoded, and it may be based on a signal indicating whether prediction of the motion information is performed (for example, when mvp_enabled_flag is 0, the motion information is encoded as it is; and when mvp_enabled_flag is 1, the motion information is predicted and encoded. In other words, only for 1, the motion information encoding modes mentioned later, such as skip mode, merge mode, contention mode, etc., can be used). In the present disclosure, the description is made under the assumption that the signal is 1. In other words, in the examples mentioned later, it is assumed that all or some of the motion information of the target block is encoded based on prediction.

[0504] In the above description, a basic block may be set in a target block and a block having a horizontal relationship or a vertical relationship with the target block. In detail, this means that a block may be set differently, and when the motion information of the target block is encoded based on the reference picture information or the motion information of the related block (by prediction), the block becomes a standard. Since its content can be derived from the various examples mentioned above, a detailed description is omitted.

[0505] In summary, the target block may be a block that is a related party having motion information to be encoded, and the basic block may be a block that becomes a standard when the motion information prediction candidate group is configured (for example, a block that becomes a standard when neighboring blocks in the left and upper directions are specified). In this case, the basic block may be set as the target block, or may be set as a related block (a block in a vertical relationship / horizontal relationship). The basic block mentioned in the later-mentioned example may be derived from the various examples mentioned above including this example of the present disclosure.

[0506] In the present disclosure, the motion information of the target block may be encoded based on one or more motion information encoding modes. In this case, the motion information encoding mode may be defined differently and may include one or more of a skip mode, a merge mode, a competitive mode (Comp mode), etc.

[0507] It can be combined with the motion information coding mode based on the template matching (tmp) mentioned above, or can be supported as a separate motion information coding mode, or can be included in all or some of the detailed configurations of the motion information coding mode. The premise is that it is determined that template matching is supported in a higher unit (e.g., picture, slice, etc.), but the flag as to whether it is supported can be considered as a partial element for the inter-frame prediction setting.

[0508] It can be combined with the motion information coding mode based on the method of performing block matching in the current picture (ibc) mentioned above, or can be supported as a separate motion information coding mode, or can be included in all or some of the detailed configurations of the motion information coding mode. The premise is that it is determined that block matching is supported in higher units for the current picture, but the flag as to whether it is supported can be considered as a partial element for the inter-frame prediction setting.

[0509] It can be combined with the motion information coding mode based on the motion model (affine) mentioned above, or can be supported as a separate motion information coding mode, or can be included in all or some of the detailed configurations of the motion information coding mode. The premise is that it is determined that the non-translational motion model is supported in a higher unit, but the flag as to whether it is supported can be considered as a partial element for the inter-frame prediction setting.

[0510] For example, separate motion information coding modes such as temp_inter, temp_tmp, temp_ibc, and temp_affine may be supported. Alternatively, combined motion information coding modes such as temp_inter_tmp, temp_inter_ibc, temp_inter_affine, and temp_inter_tmp_ibc may be supported. Alternatively, it may be configured by including template-based candidates, candidates based on a method of performing block matching in the current picture, and affine-based candidates in the motion information prediction candidate group configuration temp.

[0511] In this case, temp can represent skip mode (skip), merge mode (merge), and contention mode (comp). In an example, for skip mode, motion information coding modes such as skip_inter, skip_tmp, skip_ibc, and skip_affine can be supported; for merge mode, motion information coding modes such as merge_inter, merge_tmp, merge_ibc, and merge_affine can be supported; and for contention mode, motion information coding modes such as comp_inter, comp_tmp, comp_ibc, and comp_affine can be supported.

[0512] When skip mode, merge mode and competition mode are supported and candidates considering the above elements are included in the motion information prediction candidate group of each mode, a mode can be selected by identifying the flags of skip mode, merge mode and competition mode. In the example, when a flag indicating whether it is a skip mode is supported and the value of the flag is 1, the skip mode can be selected; when a flag indicating whether it is a merge mode when the value is 0 is supported and the value of the flag is 1, the merge mode can be selected; when the value of the flag is 0, the competition mode can be selected. Moreover, candidates based on inter, tmp, ibc, and affine can be included in the motion information prediction candidate group of each mode.

[0513] Alternatively, when multiple motion information encoding modes are supported in one common mode, in addition to a flag for selecting one of the skip mode, merge mode, and contention mode, an additional flag for identifying the detailed mode of the selected mode may be supported. In the example, when the merge mode is selected, this means that a flag for selecting in detailed modes regarding the merge mode such as merge_inter, merge_tmp, merge_ibc, merge_affine, etc. is additionally supported. Alternatively, a flag indicating whether it is merge_inter is supported, and when it is not merge_inter, a flag for selecting in merge_tmp, merge_ibc, merge_affine, etc. may be additionally supported.

[0514] All or some of the motion information coding mode candidates may be supported according to the encoding setting. In this case, the encoding setting may be defined by one or more elements of state information such as size, shape, aspect ratio, position, etc. of a basic block (e.g., a target block), image type, image category, color component, inter-frame prediction support setting (e.g., whether template matching is supported, whether block matching is supported in the current picture, non-translational motion model support elements, etc.).

[0515] In an example, the supported motion information encoding mode can be determined according to the size of the block. In this case, for the size of the block, the support range can be determined by the first threshold size (minimum value) or the second threshold size (maximum value), and each threshold size can be represented as W, H, W×H, W*H with the width (W) and height (H) of the block. For the first threshold size, W and H can be integers such as 4, 8, 16 or more, and W*H can be integers such as 16, 32, 64 or more. For the second threshold size, W and H can be integers such as 16, 32, 64 or more, and W*H can be integers such as 64, 128, 256 or more. The range can be determined by one of the first threshold size or the second threshold size, or can be determined by using both the first threshold size and the second threshold size.

[0516] In this case, the threshold size may be fixed or may be adaptive according to an image (e.g., image type, etc.) In this case, the first threshold size may be set based on the size of a minimum coding block, a minimum prediction block, a minimum transform block, etc., and the second threshold size may be set based on the size of a maximum coding block, a maximum prediction block, a maximum transform block, etc.

[0517] In the example, the supported motion information encoding mode can be determined according to the image type. In this case, for image type I, at least one of skip mode, merge mode, and competition mode can be included. In this case, a separate motion information encoding mode for a method of performing block matching (or template matching) in the current picture and an affine model (hereinafter, referred to as the term "element") can be supported, or the motion information encoding mode can be supported by combining two or more elements. Alternatively, the elements according to the predetermined motion information encoding mode can be configured in the motion information prediction candidate group.

[0518] For the picture type P / B, at least one of the skip mode, the merge mode, or the competitive mode may be included. In this case, a separate motion information encoding mode regarding general inter-frame prediction, template matching, block matching, and an affine model (hereinafter, referred to as the term "element") in the current picture may be supported, or the motion information encoding mode may be supported by combining two or more elements. Alternatively, an element according to a predetermined motion information encoding mode may be configured in a motion information prediction candidate group.

[0519] Next, definition and configuration according to the motion information encoding mode are described.

[0520] The skip mode, merge mode, and contention mode may use / reference the motion information of a related block (candidate block) set based on a basic block for motion information encoding of a target block. In other words, motion information (e.g., a reference image or a reference area) may be derived from the related block, and motion information (predicted motion information) obtained based on the related block may be predicted as the motion information of the target block.

[0521] In this case, the difference information between the motion vector of the target block and the predicted motion vector may not be generated in the skip mode and the merge mode, and the difference information may be generated in the competitive mode. In other words, this may mean that the predicted motion vector is used as the motion vector of the target block as is in the skip mode and the merge mode, but a modification to generate the difference information in the merge mode is possible.

[0522] In this case, information about the reference image of the target block can be used as predicted reference image information as is in the skip mode and the merge mode, and information about the reference image of the target block can be encoded as reference image information without prediction or based on the predicted reference image information in the competitive mode (for example, encoding the difference after prediction).

[0523] In this case, the residual component of the target block may not be generated in the skip mode, and the residual component of the target block may be generated in the merge mode and the competitive mode. In other words, this means that the residual component may be generated in the merge mode and the competitive mode, and processing (transformation, quantization, inverse processing of the residual component) for the residual component may be performed (for example, the processing may not be performed by confirming whether there is a residual signal in a block such as coded_block_flag), but a modification in which the residual component is not generated in the merge mode is possible.

[0524] The motion information prediction candidate groups of the motion information coding modes may be configured differently. In an example, the skip mode and the merge mode may configure a candidate group together, and the competitive mode may configure a separate candidate group. The candidate group configuration setting may be determined based on the motion information coding mode.

[0525] In this case, the candidate group configuration setting may be defined by the number of candidate groups, the category and position of the candidate blocks (related blocks), the candidate configuration method, and the like.

[0526] The number of candidate groups may be k, and k may be an integer from 1 to 6 or more. In this case, when the number of candidate groups is 1, it means that candidate group selection information is not generated, and the motion information of the predetermined candidate block is set as the predicted motion information, and when the number of candidate groups is equal to or greater than 2, candidate group selection information may be generated.

[0527] The category of the candidate block may be one or more of inter_blk_A, inter_blk_B, and inter_blk_C. In this case, inter_blk_A may be a category that is basically included, and the other categories may be categories that are additionally supported, but are not limited thereto.

[0528] For inter_blk_A or inter_blk_C, the position of the candidate block can be adjacent to the basic block in the left, top, upper left, upper right, and lower left directions, and can be selected from the Figure 6 For inter_blk_B, it can be specified as a block having pattern information having identity / similarity with the encoding information of the target block, and the relevant description can be derived from the portion specifying the block having the horizontal relationship mentioned above. In the example, when the motion information prediction candidate group is configured in the motion information encoding mode based on the motion model, the motion information of the blocks having the same pattern can be included in the candidate group.

[0529] As in the description, the positions of the candidate blocks referred to to configure the motion information prediction candidate group are described. Next, a method of obtaining prediction motion information based on the corresponding candidate blocks will be described.

[0530] The candidate block used to configure the motion information prediction candidate group for inter-frame prediction may represent a related block in the relationship between the above-mentioned blocks. In other words, a block specified among many related blocks based on a basic block may be called a candidate block, and the motion information prediction candidate group may be configured based on motion information of the candidate block, etc. In the example mentioned later, it should be understood that the candidate block may represent a related block.

[0531] Fig.16 is an exemplary diagram of an arrangement of blocks that are spatially or temporally adjacent to a basic block according to an embodiment of the present disclosure. In detail, Fig.16 It may be an exemplary diagram about the arrangement of blocks (related blocks or candidate blocks) having a horizontal relationship with basic blocks belonging to the categories of inter_blk_A and inter_blk_C. In the latter mentioned example, blocks belonging to inter_blk_A are called spatial candidates, and blocks belonging to inter_blk_C are called temporal candidates.

[0532] Reference Fig.16 , blocks adjacent to the basic block in the left, top, upper left, upper right, lower left directions, etc. and blocks adjacent to the basic block in the center, left, right, top, bottom, upper left, upper right, lower left, lower right directions, etc. based on blocks corresponding to the basic block in a temporally different space (Col_Pic) can be configured as candidate blocks for predicting motion information of the target block.

[0533] Furthermore, blocks adjacent in the above direction may be divided (classified) in units of one or more sub-blocks (for example, L blocks in the left direction may be divided into l0, l1, l2, l3 sub-blocks), Fig.16 , is an example in which adjacent blocks in each direction are configured with 4 sub-blocks (the center of the corresponding block is configured with 16 sub-blocks), but it can be divided into various p sub-blocks, but is not limited thereto. In this case, p may be an integer such as 1, 2, 3, 4 or more. In addition, p may be adaptively determined based on the position (direction) of the adjacent block.

[0534] In this case, Col_Pic may be an image adjacent before or after the current image (eg, when the interval between images is 1), and the corresponding block may be set to have the same position in the image as the basic block.

[0535] Alternatively, Col_Pic can be an image that predefines the interval between images based on the current image (for example, the interval between images is z. z is an integer such as 1, 2, 3), and the corresponding block can be set to have a position moved by a predetermined disparity vector on a predetermined coordinate (for example, upper left) of the basic block, and the disparity vector can be set to a predefined value.

[0536] Alternatively, Col_Pic may be set based on motion information of a neighboring block (eg, a reference image), and a disparity vector may be set based on motion information (eg, a motion vector) of the neighboring block to determine the position of the corresponding block.

[0537] In this case, k neighboring blocks may be referenced, and k may be an integer such as 1, 2, or more. When k is equal to or greater than 2, Col_Pic and the disparity vector may be obtained based on operations such as a maximum value, a minimum value, a median value, a weighted average value, etc. of motion information (e.g., a reference image or a motion vector) of the neighboring blocks. For example, the disparity vector may be set to a motion vector of a left block or an upper block, and may be set to a median value or an average value of motion vectors of a left block and a lower left block.

[0538] The above setting of the temporal candidate may be determined based on the motion information configuration setting, etc. For example, the position of Col_Pic, the position of the corresponding block, etc. may be determined according to whether the motion information in block units or the motion information in sub-block units to be included in the motion information prediction candidate group is configured in block units or in sub-block units. In the example, when the motion information in sub-block units is obtained, the block at the position moved by a predetermined disparity vector may be set to have the position of the corresponding block.

[0539] The above example represents a case where information about the position of a block corresponding to Col_Pic is implicitly determined, and related information may be explicitly generated in units of a sequence, a picture, a slice, a tile group, a tile, a brick, or the like.

[0540] According to the motion information encoding mode, the above-mentioned motion information (spatial candidates and temporal candidates, respectively) of spatially or temporally neighboring blocks (or candidate blocks) may be included in the motion information prediction candidate group.

[0541] In example (1), singular motion information may be included in the candidate group as is. In other words, all or some of the singular motion vector, singular reference picture, singular prediction direction, etc. may be used as prediction values ​​for all or some of the motion vector, reference picture, prediction direction, etc. of the target block without change.

[0542] In this case, singular motion information can be obtained from one candidate block of a spatial candidate or a temporal candidate. In the example mentioned later, it is assumed that the candidate obtained in this example is a spatial candidate. In addition, it can be applied to all modes in skip mode, merge mode and competitive mode.

[0543] In example (2), after the adjustment (or transformation) process, the singular motion information may be included in the candidate group. This example also assumes a case where the singular motion information is obtained from one candidate block. In detail, the adjustment process of the motion information (e.g., motion vector) of the candidate block may be performed based on the distance interval between the current picture and the reference picture and the distance interval between the picture to which the candidate block belongs and the reference picture of the candidate block.

[0544] In other words, the motion vector of the candidate block can be adjusted in the scaling process based on the distance interval between the current picture and the reference picture. And the adjusted motion vector of the candidate block can be included in the candidate group. For the motion information or prediction direction of the reference picture, etc., the reference picture or prediction direction information of the candidate block can be included in the candidate group without change, or can be included in the candidate group based on the distance interval of the reference picture. Alternatively, information set based on predetermined information (for example, the reference picture is a picture before or after the current picture <reference picture index is 0>, the prediction direction is one of forward, backward, bidirectional, etc.) or the prediction direction information of the reference picture or the target block can be included in the candidate group.

[0545] In this case, the singular motion information may be a temporal candidate. In the example mentioned later, it is assumed that the candidate obtained in the example is a temporal candidate. In addition, it may be a case where it is applied to all or some of the skip mode, merge mode, and contention mode. For example, it may be applied to the skip mode, merge mode, and contention mode, or it may not be applied to the skip mode and merge mode and may be applied to the contention mode.

[0546] In example (3), after the combination process, multiple motion information may be included in the candidate group. The combination process may be performed on all or some of the motion vectors, reference images, and prediction directions. The combination process represents the processing of the median value, weighted average value, etc. of the motion information. In detail, q motion information obtained based on the median value, weighted average value, etc. of p motion information may be included in the candidate group. In this case, p may be an integer such as 2, 3, 4 or more, and q may be an integer such as 1, 2 or more. In the example, p and q may be 2 and 1, respectively. In this case, for the average value, the same weight (e.g., 1:1, 1:1:1, etc.) may be applied to the motion information or different weights (e.g., 1:2, 1:3, 2:3, 1:1:2, 1:2:3, etc.) may be applied.

[0547] A plurality of motion information may be obtained from a plurality of candidate blocks. In this case, it is assumed that one motion information is obtained from one candidate block, but it is not limited thereto. The motion information of the candidate block (input value of the combination process) may be limited to one of (1) or (2) or to both (1) and (2).

[0548] In addition, multiple motion information can be derived from either the spatial candidate or the temporal candidate, and can be derived from both the spatial candidate and the temporal candidate. For example, the combined processing can be performed by using the spatial candidate, or the combined processing can be performed by using the temporal candidate. Alternatively, the combined processing can be performed by using one or more spatial candidates and one or more temporal candidates.

[0549] A plurality of motion information may be obtained from pre-included candidates of the candidate group. In other words, a plurality of candidate motion information previously included in the candidate group configuration process (e.g., spatial candidates or temporal candidates) may be used as input values ​​for the combination process. In this case, the candidate motion information used as the input value for the combination process may be derived from either the spatial candidate or the temporal candidate, and may be derived from both the spatial candidate and the temporal candidate.

[0550] The description may be applicable to all or some of the skip mode, merge mode, and contention mode. For example, it may be applicable to the skip mode, merge mode, and contention mode, or it may be applicable to the skip mode and merge mode and may not be applicable to the contention mode.

[0551] For motion information obtained in the combination process, motion information derived from a spatial candidate or a temporal candidate is called a spatial derived candidate or a temporal derived candidate, and motion information derived from a spatial candidate and a temporal candidate is called a spatial and temporal derived candidate. In the following example, the process of deriving candidates will be described.

[0552] As an example of a spatially derived candidate, a spatially derived candidate may be obtained by applying a process of obtaining a weighted average or median value based on motion vectors of all or some of the blocks in the left, top, upper left, upper right, and lower left directions of the basic block. Alternatively, a spatially derived candidate may be obtained by applying a process of obtaining a weighted average or median value based on all or some of the spatial candidates already included in the candidate group. Next, it is assumed that motion vectors l3, t3, tl, tr, bl are regarded as obtained. Fig.16 The weighted average or median of the processed input values ​​of Curr_Pic.

[0553] For example, the x component of the motion vector may be derived from avg(l3_x, t3_x, tl_x, tr_x, bl_x), the y component of the motion vector may be derived from avg(l3_y, t3_y, tl_y, tr_y, bl_y), and avg may be assumed to be a function that calculates the average value of the motion vectors in the brackets. Alternatively, the x component of the motion vector may be derived from median(l3_x, t3_x, tl_x, tr_x, bl_x), and the y component of the motion vector may be derived from median(l3_y, t3_y, tl_y, tr_y, bl_y), and median may be assumed to be a function that calculates the median value of the motion vectors in the brackets.

[0554] As an example of a temporal derived candidate, a temporal derived candidate may be obtained by applying a process of obtaining a weighted average or median value based on motion vectors of all or some of the blocks in the left, right, top, bottom, upper left, upper right, lower left, lower right, and center directions of the block corresponding to the basic block. Alternatively, a temporal derived candidate may be obtained by applying a process of obtaining a weighted average or median value based on all or some of the temporal candidates already included in the candidate group. Next, it is assumed that the motion vectors br, r2, b2 are regarded as the obtained Fig.16 The weighted average or median of the processed input values ​​in Col_Pic.

[0555] For example, the x component of the motion vector can be derived from avg(br_x, r2_x, b2_x), and the y component of the motion vector can be derived from avg(br_y, r2_y, b2_y). Alternatively, the x component of the motion vector can be derived from median(br_x, r2_x, b2_x), and the y component of the motion vector can be derived from median(br_y, r2_y, b2_y).

[0556] As an example of spatial and temporal derived candidates, the spatial and temporal derived candidates may be obtained by applying a process of obtaining a weighted average or median value based on motion vectors of all or some of the spatial neighboring blocks of a basic block and a block corresponding to the basic block. Alternatively, the spatial and temporal derived candidates may be obtained by applying a process of obtaining a weighted average or median value based on all or some of the spatial candidates or temporal candidates already included in the candidate group. Examples thereof are omitted.

[0557] The motion information obtained in the above process may be added to (included in) the motion information prediction candidate group of the target block.

[0558] The embodiments in (1) to (3) describe the following examples: a spatial candidate or a temporal candidate is included in the motion information prediction candidate group without change, or a spatially derived candidate, a temporally derived candidate, and a spatially and temporally derived candidate obtained based on a plurality of spatial candidates or temporal candidates are included in the motion information prediction candidate group. In other words, the embodiments in (1) and (2) may be descriptions about spatial or temporal candidates, and the embodiment in (3) may be descriptions about spatially derived candidates, temporally derived candidates, and spatially and temporally derived candidates.

[0559] The example mentioned above may be a case where the motion vector in pixel units is the same in the target block. As an example of a case where the motion vector in pixel units or in sub-block units is different in the target block, the affine model is described above. In the example mentioned later, the motion vector in sub-block units partially changes in the target block, but an example of a case where it is represented as a candidate based on spatial or temporal correlation will be described.

[0560] Next, a case where a prediction value of motion information in units of sub-blocks is obtained based on one or more motion information is described. It may be an example in which a process similar to the process of obtaining motion information based on a plurality of motion information (3) is performed, but motion information in units of sub-blocks (4) is obtained. The motion information obtained in the process of (4) may be included in the motion information prediction candidate group. The embodiment (4) is described later by using a spatial derived candidate, a temporal derived candidate, and a spatial and temporal derived candidate, which are terms used in this example.

[0561] When the prediction value of the motion information in sub-block units is obtained, the basic block can be limited to the target block. In addition, when the candidate group is configured by mixing the motion information in sub-block units and the motion information in block units, it is assumed that the basic block is limited to the target block.

[0562] As an example of spatially derived candidates, a candidate based on a sub-block in the target block (in this example, it is configured with 4 sub-blocks < Fig.16 In Curr_Pic, c0+c1+c4+c5 / c2+c3+c6+c7 / c8+c9+c12+c13 / c10+c11+c14+c1 are respectively referred to as D0, D1, D2, D3>) to obtain the spatial neighboring blocks (in this example, Fig.16 The various blocks in Curr_Pic of the sub-block are used as the spatial derivation candidate, or the spatial derivation candidate can be obtained by a weighted average or median value based on multiple motion information. The obtained spatial derivation candidate can be added to the motion information prediction candidate group of the target block.

[0563] For example, for sub-block D0, l1, t1, t2 (the lower area of ​​the left block, the right area of ​​the upper block, the upper left block based on the sub-block) can be the spatial neighboring blocks of the corresponding sub-block; for sub-block D1, l1, t3 can be the spatial neighboring blocks of the corresponding sub-block; for sub-block D2, l3, t1, t2 can be the spatial neighboring blocks of the corresponding sub-block; for sub-block D3, l3, t3, tr can be the spatial neighboring blocks of the corresponding sub-block. In this case, what is added to the motion information prediction candidate group of the target block is a spatial derived candidate (this example assumes that one is supported), but the motion information derived therefrom can be derived in units of each sub-block (in this example, up to 4 sub-blocks). In other words, when the decoder parses the information for selecting the spatial derived candidate, the motion information prediction value of each sub-block can be obtained as in the above process.

[0564] As an example of temporal candidate derivation, a temporal neighboring block obtained based on a sub-block in the target block can be obtained (in this example, Fig.16 The block used in this example is located at the center, lower right, right or lower position adjacent to the corresponding sub-block as one of the motion information as the temporal derivation candidate, or the temporal derivation candidate can be obtained by a weighted average or median value based on a plurality of motion information. The obtained temporal derivation candidate can be added to the motion information prediction candidate group of the target block.

[0565] For example, for sub-block D0, c10, c6, c9 (lower left block, lower area of ​​right block, right area of ​​lower block based on corresponding sub-block) may be time neighboring blocks of corresponding sub-block; for sub-block D1, r2, r1, c11 may be time neighboring blocks of corresponding sub-block; for sub-block D2, b2, c14, b1 may be time neighboring blocks of corresponding sub-block; for sub-block D3, br, r3, b3 may be time neighboring blocks of corresponding sub-block. As added in this example, temporal derived candidates are obtained, but motion information obtained therefrom may be obtained in units of each sub-block.

[0566] As an example of spatial and temporal derivation candidates, a weighted average value or median value of motion information of spatially or temporally adjacent blocks (the blocks used in this example are located at spatially adjacent left and upper positions and at temporally adjacent lower right positions based on the sub-blocks) obtained based on a sub-block in a target block may be obtained as a spatial and temporal derivation candidate, and the weighted average value or median value may be added to the motion information prediction candidate group of the target block.

[0567] For example, for sub-block D0, l1, t1, c10 may be neighboring blocks of the corresponding sub-block; for sub-block D1, l1, t3, r2 may be neighboring blocks of the corresponding sub-block; for sub-block D2, l3, t1, b2 may be neighboring blocks of the corresponding sub-block; for sub-block D3, l3, t3, br may be neighboring blocks of the corresponding sub-block.

[0568] In summary, motion information obtained based on one or more motion information in implementation (4) may be included in the candidate group. In detail, q motion information obtained based on the median value, weighted average, etc. of p motion information may be included in the candidate group. In this case, p may be an integer such as 1, 2, 3 or more, and q may be an integer such as 1, 2 or more. In this case, for the average value, the same weight may be applied to the motion information (e.g., 1:1, 1:1:1, etc.), or different weights may be applied to the motion information (e.g., 1:2, 1:3, 2:3, 1:1:2, 1:2:3, etc.). The input value in the above processing may be information limited to one of (1) or (2) or belonging to both (1) and (2).

[0569] The size of the sub-block may be m×n, m and n may be integers such as 4, 8, 16 or more, and m and n may be the same or different. The size of the sub-block may have a fixed value, or may be adaptively set based on the size of the target block. In addition, the size of the sub-block may be implicitly determined according to the encoding setting, or the relevant information may be explicitly generated in various units. In this case, for the definition of the encoding setting, reference may be made to the various examples mentioned above.

[0570] This description may be applied to all or some of the skip mode, merge mode, and contention mode. For example, it may be applied to the skip mode, merge mode, and contention mode, or it may be applied to the skip mode and merge mode and may not be applied to the contention mode.

[0571] Regarding whether the implementations in (3) and (4) are supported, relevant information may be explicitly generated in units of sequences, pictures, slices, tile groups, tiles, brick blocks, etc. Alternatively, support may be determined based on encoding settings. In this case, the encoding settings may be defined by one or more elements of status information such as the size, shape, aspect ratio, position, etc. of a basic block (e.g., a target block), image type, image category, color component, settings related to inter-frame prediction (e.g., motion information coding mode, whether template matching is supported, whether block matching is supported in the current picture, non-translational motion model support elements, etc.).

[0572] For example, whether the candidate group of the implementation in (3) or (4) is supported can be determined based on the size of the block. In this case, for the size of the block, the support range can be determined by the first threshold size (minimum value) or the second threshold size (maximum value), and each threshold size can be represented as W, H, W×H, W*H using the width (W) and height (H) of the block. For the first threshold size, W and H can be integers such as 4, 8, 16 or more, and W*H can be an integer such as 16, 32, 64 or more. For the second threshold size, W and H can be integers such as 16, 32, 64 or more, and W*H can be an integer such as 64, 128, 256 or more. The range can be determined by one of the first threshold size or the second threshold size, or can be determined by using both the first threshold size and the second threshold size.

[0573] In this case, the threshold size may be fixed or may be adaptive according to an image (e.g., image type, etc.) In this case, the first threshold size may be set based on the size of a minimum coding block, a minimum prediction block, a minimum transform block, etc., and the second threshold size may be set based on the size of a maximum coding block, a maximum prediction block, a maximum transform block, etc.

[0574] For the affine model, according to the motion information encoding mode, the motion information of the spatially or temporally adjacent blocks (candidate blocks) may be included in the motion information prediction candidate group. In this case, the positions of the spatially or temporally adjacent blocks may be the same or similar to the previous embodiment.

[0575] In this case, the candidate group configuration may be determined according to the motion model of the candidate block.

[0576] For example, when the motion model of the candidate block is an affine model, the motion vector set configuration of the candidate block may be included in the candidate group without change.

[0577] Alternatively, when the motion model of the candidate block is a translation motion model, it may be included as a candidate for a motion vector at a position of a control point based on a relative position of the target block. In an example, when the upper left, upper right, and lower left coordinates are used as control points, the motion vector of the upper left control point may be predicted based on a motion vector of a left block, an upper block, or an upper left block of a basic target block (e.g., for a translation motion model), the motion vector of the upper right control point may be predicted based on a motion vector of an upper block or an upper right block of a basic block (e.g., for a translation motion model), and the motion vector of the lower left control point may be predicted based on a motion vector of a left block or a lower left block of a basic block.

[0578] In summary, when the motion model of the candidate block is an affine model, the motion vector set of the corresponding block can be included in the candidate group (A); and when the motion model of the candidate block is a translation motion model, it can be considered as a candidate for the motion vector of a predetermined control point of the target block, and the motion vector set obtained according to the combination of each control point can be included in the candidate group (B).

[0579] In this case, the candidate group may be configured by using only one of methods A or B, or may be configured by using both methods A and B. And method A may be configured first, and method B may be configured subsequently, but is not limited thereto.

[0580] Next, a method of configuring a motion information prediction candidate group according to a motion information encoding mode will be described.

[0581] For ease of description, the merge mode and the contention mode are described under the assumption that the skip mode has the same candidate group configuration as that of the merge mode, but some elements of the candidate group configuration of the skip mode may be different from those of the merge mode.

[0582] (Merge_inter mode)

[0583] The motion information prediction candidate group for merge mode (hereinafter, merge mode candidate group) may include k candidates, k may be an integer such as 2, 3, 4, 5, 6 or more. The merge mode candidate group may include at least one of a spatial candidate or a temporal candidate.

[0584] A spatial candidate may be derived from at least one of blocks adjacent in the left, top, upper left, upper right, lower left direction, etc. based on the basic block. There may be a priority for configuring the candidate group, and priorities such as left-top-lower left-upper right-upper left, left-top-upper right-lower left-upper left, upper-left-lower left-upper left-upper right, etc. may be set. In the example, Fig.16 In Curr_Pic, the priority can be set in the order of l3-t3-bl-tr-tl.

[0585] All or some of the candidates may be included in the candidate group based on the priority, the availability of each candidate block (e.g., determined based on the coding mode, the position of the block, etc.), and the maximum allowed number of spatial candidates (an integer between p. 1 and the number of merge mode candidate groups). Depending on the maximum allowed number and availability, they may not be included in the candidate group in the order of tl-tr-bl-t3-l3, when the maximum allowed number is 4 and the availability of the candidate blocks is completely true, the motion information of t1 may not be included in the candidate group, and when the availability of some candidate blocks is false, the motion information of t1 may be included in the candidate group.

[0586] A temporal candidate may be derived from at least one of blocks adjacent in the center, left, right, top, bottom, upper left, upper right, lower left, lower right, etc. directions based on a block corresponding to a basic block. There may be a priority for configuring a candidate group, and priorities such as center-lower left-lower right-lower left, lower left-center-upper left, etc. may be set. In an example, Fig.16 In Col_Pic, the priority can be set in the order of cl0-br.

[0587] Based on the priority, availability of each candidate block, and the maximum allowed number of temporal candidates (an integer between q.1 and the number of merge mode candidate groups), all or some of the candidates may be included in the candidate group. When the maximum allowed number is 1 and the availability of c10 is true, the motion information of c10 may be included in the candidate group, and when the availability of c10 is false, the motion information of br may be included in the candidate group.

[0588] In this case, the motion vector of the temporal candidate may be obtained based on the motion vector of the candidate block, and the reference image of the temporal candidate may be obtained based on the reference image of the candidate block, or may be obtained as a predefined reference image (eg, reference picture index 0).

[0589] For the priorities included in the merge mode candidate group, spatial candidate-temporal candidate and vice versa can be set; and the priorities of mixing and configuring spatial candidates and temporal candidates can be supported. This example assumes the case of spatial candidate-temporal candidate.

[0590] In addition, motion information (statistical candidates) of blocks belonging to the same space and distant neighbors or derived candidates in blocks (spatial derived candidates, temporal derived candidates, spatial and temporal derived candidates) may be additionally included in the merge mode candidate group. Statistical candidates and derived candidates in blocks may be configured after spatial candidates and temporal candidates, but various priorities are possible and are not limited to this. In the present disclosure, it is assumed that candidate groups are configured in the order of statistical candidates-derived candidates in blocks, but the reverse order is possible and is not limited to this. In this case, derived candidates in blocks may use only any one of spatial derived candidates or temporal derived candidates or spatial and temporal derived candidates, or derived candidates in blocks may use all of them. This example assumes the case where spatial derived candidates are used.

[0591] In this example, the statistical candidate is the motion information of a block far from the basic block (or not the nearest block), and it can be limited to blocks with the same coding mode as the target block. In addition, (when the statistical candidate is not motion information in sub-block units) it can be limited to blocks with motion information in block units.

[0592] For statistical candidates, up to n motion information can be managed by a method such as FIFO, and z of the motion information can be included in the merge mode candidate group as statistical candidates. Depending on the candidate configuration that has been included in the merge mode candidate group, z can be variable, and z can be an integer such as 0, 1, 2 or more and can be less than or equal to n.

[0593] Candidates derived in block units can be derived by combining n candidates that have been included in the merge mode candidate group, where n may be an integer such as 2, 3, 4 or more. The number of combined candidates (n) may be information explicitly generated in units of sequences, pictures, slices, tile groups, tiles, bricks, blocks, etc. Alternatively, n may be implicitly determined according to encoding settings. In this case, the encoding settings may be defined based on one or more elements such as the size, shape or position of the basic block, the image type, the color component, etc.

[0594] In addition, the number of combined candidates can be determined based on the number of candidates that are not populated in the merge mode candidate group. In this case, the number of candidates that are not populated in the merge mode candidate group can be the difference between the number of merge mode candidate groups and the number of candidates that have been populated. In other words, when the configuration of the merge mode candidate group has been completed, derived candidates in blocks may not be added. When the configuration of the merge mode candidate group has not been completed, derived candidates in blocks may be added, but when the number of candidates populated in the merge mode candidate group is equal to or less than 1, derived candidates in blocks are not added.

[0595] When the number of unfilled candidates is 1, derived candidates in blocks may be added based on 2 candidates, and when the number of unfilled candidates is equal to or greater than 2, derived candidates in blocks may be added based on 3 or more candidates.

[0596] The positions of the n combined candidates may be preset positions in the merge mode candidate group. For example, an index (eg, 0 to <k-1>). In this case, k represents the number of merge mode candidate groups. In this case, the positions of the n combined candidates may correspond to index 0 to index (n-1) in the merge mode candidate group. In an example, derived candidates in units of blocks may be obtained according to the index combination of (0, 1)-(0, 2)-(1, 2)-(0, 3).

[0597] Alternatively, n combined candidates may be determined by considering the prediction direction of each candidate belonging to the merge mode candidate group. In an example, among the candidates belonging to the merge mode candidate group, only bidirectional prediction candidates may be selectively used, or only unidirectional prediction candidates may be selectively used.

[0598] Through the above process, the configuration of the merge mode candidate group may or may not be completed. When the configuration of the merge mode candidate group is not completed, the configuration of the merge mode candidate group may be completed by using a zero motion vector.

[0599] In the candidate group configuration process, the candidate group may not include a posteriori motion information that is redundant with the candidates already included in the candidate group, but is not limited thereto, and an exception case where redundant motion information is included may also be generated. In addition, redundancy means that the motion vector, reference picture, and prediction direction are the same, but an exception configuration that allows a predetermined error range (e.g., motion vector) is also possible.

[0600] In addition, the merge mode candidate group can be configured based on the correlation between blocks. For example, for spatial candidates, a low priority can be assigned to the motion information of a candidate block that is judged to have a low correlation with the basic block.

[0601] For example, it is assumed that the priority of the candidate group for configuring the spatial candidate is left-upper-lower-left-upper-right-lower-left. When it is confirmed that the left block has a low correlation with the basic block according to the correlation between the blocks, the priority can be changed to set the left block in the a posteriori order. In the example, the priority can be changed to the order of upper-lower-left-upper-right-lower-left-left.

[0602] Alternatively, it is assumed that the priority of the candidate group for configuring the derived candidate in block units is in the order of (0, 1)-(0, 2)-(1, 2)-(0, 3). When the candidate corresponding to the 1st index in the candidate group is the motion information of the left block (spatial candidate), the priority may be changed to set the candidate corresponding to the 1st index to the a posteriori order. In the example, the priority may be changed to the order of (0, 2)-(0, 3)-(0, 1)-(1, 2).

[0603] When merge mode candidate groups are configured based on correlation between blocks, the candidate groups can be efficiently configured and encoding performance can be improved.

[0604] (comp_inter mode)

[0605] The motion information prediction candidate group for competition mode (hereinafter, competition mode candidate group) may include k candidates, and k may be an integer such as 2, 3, 4 or more. The merge mode candidate group may include at least one of a spatial candidate or a temporal candidate.

[0606] A spatial candidate may be derived based on the basic block from at least one of blocks adjacent in the left, top, upper left, upper right, lower left directions, etc. Alternatively, at least one candidate may be derived from blocks adjacent in the left direction (left, lower left blocks) and blocks adjacent in the upper direction (upper left, upper, upper right blocks), which will be described later by assuming such a setting.

[0607] There may be two or more priorities for configuring the candidate group. A priority of lower left-left may be set in an area adjacent in the left direction, and a priority of upper right-upper-upper left may be set in an area adjacent in the upper direction.

[0608] The above example may be a configuration in which a spatial candidate is derived only from a block having the same reference picture as the target block, and the spatial candidate may be derived by a scaling process (next, marked with *) based on the reference picture of the target block. In this case, a priority of left-lower-left-left*-lower-left* or left-lower-left-lower-left*-left* may be set in an area adjacent in the left direction, and a priority of upper right-upper-upper-left-upper-right*-upper*-upper-left* or upper right-upper-upper-left-upper-left*-upper*-upper-right* may be set in an area adjacent in the upper direction.

[0609] A temporal candidate may be derived from at least one of blocks adjacent in the center, left, right, top, bottom, top left, top right, bottom left, bottom right, etc. based on a block corresponding to a basic block. There may be a priority for configuring a candidate group, and priorities such as center-bottom left-bottom right-bottom, bottom left-center-top left, etc. may be set. In an example, Fig.16 In Col_Pic of, you can set the priority in the order of c10-br.

[0610] When there is a setting in which the sum of the maximum allowed number of spatial candidates and the maximum allowed number of temporal candidates is smaller than the number of contention mode candidate groups, temporal candidates may be included in the candidate group regardless of the candidate group configuration of spatial candidates.

[0611] Based on the priority, availability of each candidate block, and the maximum allowed number of temporal candidates (an integer between q. 1 and the number of contention mode candidate groups), all or some of the candidates may be included in the candidate group.

[0612] In this case, when the maximum allowed number of spatial candidates is set to be the same as the number of merge mode candidate groups, temporal candidates may not be included in the candidate group, and when the maximum allowed number is not filled based on spatial candidates, temporal candidates may be included in the candidate group. This example assumes the latter case.

[0613] In this case, the motion vector of the temporal candidate can be obtained based on the motion vector of the candidate block and the distance interval between the current image and the reference image of the target block, and the reference image of the temporal candidate can be obtained based on the distance interval between the current image and the reference image of the target block, or the reference image of the temporal candidate can be obtained based on the reference image of the temporal candidate, or the reference image of the temporal candidate can be obtained as a predefined reference image (for example, the reference picture index is 0).

[0614] Through the above process, the configuration of the contention mode candidate group may or may not be completed. When the configuration of the contention mode candidate group is not completed, the contention mode candidate group may be configured by using a zero motion vector.

[0615] In the candidate group configuration process, the candidate group may not include a posteriori motion information that is redundant with the candidates already included in the candidate group, but is not limited thereto, and an exception case where redundant motion information is included may also be generated. In addition, redundancy means that the motion vector, reference picture, and prediction direction are the same, but an exception configuration that allows a predetermined error range (e.g., motion vector) is also possible.

[0616] In addition, the competitive mode candidate group can be configured based on the correlation between blocks. For example, for spatial candidates, the motion information of candidate blocks that are judged to have low correlation with the basic block can be excluded from the candidate group.

[0617] For example, it is assumed that in the blocks adjacent in the left direction, the priority of the candidate group for configuring the spatial candidate is left-lower left-left*-lower left*; in the blocks adjacent in the upper direction, the priority of the candidate group for configuring the spatial candidate is upper right-upper-upper left-upper right*-upper*-upper left*. When it is confirmed that the upper block has a low correlation with the basic block based on the correlation between the blocks, the candidate about the upper block can be removed from the priority. In the example, the priority can be changed to the order of upper right-upper left-upper right*-upper left*.

[0618] When the contention mode candidate groups are configured based on the correlation between blocks, the candidate groups can be efficiently configured and the encoding performance can be improved.

[0619] Next, a process of performing inter prediction according to the motion information encoding mode will be described.

[0620] A motion information encoding mode of a target block is derived (1). A reference block is specified according to the derived motion information encoding mode (2). A motion information prediction candidate group is configured using motion information obtained based on the specified reference block and the motion information encoding mode (3). Motion information of the target block is derived from the motion information prediction candidate group (4). Inter-frame prediction may be performed by using the motion information of the target block.

[0621] In (1), flag information about the motion information encoding mode of the target block may be signaled. In this case, the motion information encoding mode may be determined as one of a skip mode, a merge mode, and a contention mode; and the signaled flag may be configured with one or two or more information according to the determined mode. In addition, additional flag information for selecting one of the detailed categories of the determined mode may be signaled.

[0622] In (2), a reference block is designated according to the derived motion information encoding mode. The position of the reference block according to the motion information encoding mode may be configured differently. In (3), which motion information is to be derived from the designated reference block may be determined based on the motion information encoding mode, and the derived motion information may be included in the candidate group.

[0623] In (4), motion information corresponding to a corresponding index in a candidate group may be derived based on candidate selection information of a target block. In this case, the derived motion information configuration may be determined according to a motion information encoding mode. In the example, for a skip mode or a merge mode, information about a motion vector, a reference image, and a prediction direction may be derived based on one candidate selection information. Alternatively, for a competitive mode, information about a motion vector may be derived based on one candidate selection information, and information about a reference image and a prediction direction may be derived based on other predetermined flags. In (5), inter-frame prediction may be performed by using the motion information of the target block.

[0624] The prediction block may be generated by using the motion information obtained through the above processing. The target block may be reconstructed by adding the residual component of the target block. In this case, the residual component may be obtained by performing at least one of dequantization or inverse transformation on the residual coefficients signaled from the bitstream.

[0625] Next, the differential motion vector, that is, the difference between the motion vector of the target block and the predicted motion vector, will be described later, and it can be a description corresponding to the competition mode.

[0626] The differential motion vector (in this example, description is made based on the absolute value of the differential motion vector) can be expressed according to the motion vector accuracy (in this example, it is assumed that the motion vector accuracy is determined according to the reference picture interpolation accuracy. For the above-mentioned content <for example, an example in which a motion vector other than a differential motion vector is encoded without change>, when the motion vector accuracy is adaptively determined, it does not correspond to the example).

[0627] For example, when the motion vector precision is 1 / 4 unit, the motion vector of the target block is (2.5, -4), and the predicted motion vector is (3.5, -1), the differential motion vector may be (-1, -3).

[0628] When the differential motion vector precision is the same as the motion vector precision (ie, 1 / 4 unit), the x differential component may require a codeword and a negative sign according to the 4th index, while the y differential component may require a codeword and a positive sign according to the 12th index.

[0629] Furthermore, in addition to the motion vector accuracy, the accuracy of the differential motion vector may also be supported. In this case, the differential motion vector accuracy may have an accuracy less than or equal to that of the motion vector accuracy of the candidate group.

[0630] The differential motion vector may be represented differently according to its accuracy.

[0631] For example, when the motion vector accuracy is 1 / 4 unit, the motion vector of the target block is (7.75, 10.25), and the predicted motion vector is (2.75, 3.25), the differential motion vector may be (5, 7).

[0632] When fixed differential motion vector precision is supported (i.e., when the differential motion vector precision is the same as the motion vector precision. In this example, the motion vector precision is 1 / 4 unit), the x differential component may need to be indexed according to the codeword and sign of the 20th index, while the y differential component may need to be indexed according to the codeword and sign of the 28th index. This example may be an example according to fixed differential motion vector precision.

[0633] When adaptive differential motion vector precision is supported (i.e., when the differential motion vector precision is the same as or different from the motion vector precision. In other words, when one of multiple precisions is selected. In this example, it is assumed that the precision is an integer (1) unit), the x differential component may need to be indexed according to the codeword and sign of the 5th index, while the y differential component may need to be indexed according to the codeword and sign of the 7th index.

[0634] When it is assumed that the same codewords are assigned to the index order regardless of the accuracy in the example (for example, the binarization method for the differential motion vector is the same); when fixed differential motion vectors are supported, long codewords such as codewords according to the 20th index and the 28th index (in this example, it is assumed that short bits are assigned in the case of a small index and long bits are assigned in the case of a large index) are assigned to the x differential component and the y differential component; but when adaptive differential motion vectors are supported, short codewords such as codewords according to the 5th index and the 7th index can be assigned to the x differential component and the y differential component.

[0635] The above example may be a case where differential motion vector accuracy is supported for one differential motion vector. Alternatively, a case where differential motion vector accuracy is supported for an x ​​differential component and a y differential component configuring one differential motion vector may be as follows.

[0636] For example, when the differential motion vector is (5, -1.75) and adaptive differential motion vector accuracy (in this example, integer (1) unit, 1 / 4 unit) is supported, the following configuration is possible.

[0637] For an integer (1) unit, the x differential component may need a codeword and a positive sign according to the 5th index, and for a 1 / 4 unit, the y differential component may need a codeword and a negative sign according to the 7th index. This example may be the best case, and the differential motion vector precision selection information of the x differential component may be determined in integer units, and the differential motion vector precision selection information of the y differential component may be determined in 1 / 4 units.

[0638] As in the above example, the precision commonly applied to the differential motion vector may be supported, or the precision individually applied to the components of the differential motion vector may be supported, and the precision may be determined according to the encoding setting.

[0639] Additionally, as a description about the bitstream configuration, for the former, when at least one differential component is not 0, adaptive differential motion vector accuracy can be supported.

[0640] abs_mvd_x

[0641] if(abs_mvd_x)

[0642] mvd_sign_x

[0643] abs_mvd_y

[0644] if(abs_mvd_y)

[0645] mvd_sign_y

[0646] if((abs_mvd_x||abs_mvd_y)&&adaptive_mvd_precision_enabled_flag)

[0647] mvd_precision_flag

[0648] For the latter, adaptive differential motion vector accuracy can be supported when each differential component is not zero.

[0649] abs_mvd_x

[0650] if(abs_mvd_x)

[0651] mvd_sign_x

[0652] abs_mvd_y

[0653] if(abs_mvd_y)

[0654] mvd_sign_y

[0655] if(abs_mvd_x&&adaptive_mvd_precision_enabled_flag)

[0656] mvd_precision_flag_x

[0657] if(abs_mvd_y&&adaptive_mvd_precision_enabled_flag)

[0658] mvd_precision_flag_y

[0659] In summary, the differential motion vector accuracy may be determined according to the encoding setting, and the differential motion vector accuracy may be one of integer units and fractional units. The differential motion vector accuracy may have one preset differential motion vector accuracy or one of a plurality of differential motion vector accuracy. For the former, the differential motion vector accuracy may be an example determined based on the reference picture interpolation accuracy.

[0660] When multiple differential motion vector precisions are supported (e.g., adaptive_mvd_precision_enabled_flag. If it is 0, a preset differential motion vector precision is used; if it is 1, one of the multiple differential motion vector precisions is used), differential motion vector precision selection information (e.g., mvd_precision_flag) may be generated. In this case, the precision candidate group may be configured with precision in units of 4, 2, 1, 1 / 2, 1 / 4, 1 / 8, or 1 / 16. In other words, the number of candidate groups may be an integer such as 2, 3, 4, 5, or more.

[0661] Although the motion vector accuracy and the differential motion vector accuracy in the examples in the above-mentioned embodiments are classified and described according to the motion information encoding setting, they may have the same or similar configuration because they have fixed or adaptive accuracy for the motion information to be encoded.

[0662] In the present disclosure, description is made on the assumption that a motion vector to be encoded is configured with a differential motion vector and sign information about the differential motion vector.

[0663] In the encoding mode, the skip mode and the merge mode obtain prediction information, and immediately use the prediction information as the motion information of the corresponding block without generating difference information, so adaptive differential motion vector accuracy is not supported. When at least one differential motion vector is not 0 in the competition mode, adaptive differential motion vector accuracy can be supported.

[0664] The reason why the motion information is stored is as follows: because the motion information of the neighboring blocks is used to encode the motion information of the target block, and after the target block is encoded, the target block can also be referenced when encoding the motion information of the subsequent blocks. The description is made under the following setting: the accuracy of the motion vector storage unit is determined according to the reference picture interpolation accuracy. In other words, the description is made under the assumption that the representation unit of the motion vector is set to be different from the storage unit, and the examples mentioned later assume the case where the adaptive differential motion vector accuracy setting is supported.

[0665] As described above, the accuracy of representing the motion vector (assuming that it is 1 / 8 unit) may be the same as the accuracy of storing the motion vector. A case where the accuracy of representing the motion vector is different from the accuracy of storing the motion vector has also been described, but for ease of description, in this example, the description is made under the assumption that the accuracy of representing the motion vector is the same as the accuracy of storing the motion vector, and the contents thereon may be expanded and applied to a case where the accuracy of representing the motion vector is different from the accuracy of storing the motion vector.

[0666] In the example, since the differential component is not generated in the skip mode (or merge mode), the motion vector can be stored according to the preset motion vector storage unit. In other words, since the accuracy of the predicted motion vector (because the prediction is performed by the stored motion vector of the neighboring block) is the same as the preset motion vector accuracy, the predicted motion vector can be regarded as a reconstructed motion vector and is stored without change (as the motion vector of the target block).

[0667] In the example, the differential component is generated in the competition mode, and the accuracy of the differential motion vector may be the same as or different from the preset motion vector storage unit, so the motion vector may be stored according to the preset motion vector storage unit. In other words, since the accuracy of the predicted motion vector may be the same as or different from the accuracy of the differential motion vector, a process of matching the accuracy of the differential motion vector with the accuracy of the predicted motion vector may be performed before the predicted motion vector is stored in the memory.

[0668] The differential motion vector accuracy is equal to or lower than the motion vector accuracy, so in this case, processing to match the accuracy of the differential motion vector with the accuracy of the predicted motion vector may be required (e.g., multiplication, division, rounding / rounding down / rounding up, shift operation, etc. In this example, a left shift operation). In this example, it is assumed that integer (1) units, 1 / 4 units, and 1 / 8 units are included as candidates for the differential motion vector accuracy.

[0669] Assume that each candidate has the following settings.

[0670] mvd_pres_idx=0 (1 / 1 unit)->mvd_shift=3 (number of shifts)

[0671] mvd_pres_idx=1 (1 / 2 unit)-> mvd_shift=2 (number of shifts)

[0672] mvd_pres_idx=2 (1 / 8 unit)->mvd_shift=0 (number of shifts)

[0673] The motion vector can be stored in the memory through the following process.

[0674] if(adaptive_mvd_precision_enabled_flag==0)

[0675] {

[0676] mv_x=mvp_x+mvd_x

[0677] _y=mvp_y+mvd_y

[0678] }

[0679] else

[0680] {

[0681] mv_x=mvp_x+(mvd_x< <mvd_shift_x)

[0682] mv_y=mvp_y+(mvd_y< <mvd_shift_y)

[0683] }

[0684] In the example, mv represents a reconstructed motion vector, mvp represents a predicted motion vector, and mvd represents a differential motion vector. _x and _y following mvd_shift are used to classify under the precision settings respectively applied to the differential motion vector configuration components, and they can be removed and represented with the precision settings normally applied to the differential motion vector. In addition, for a fixed differential motion vector precision setting, the reconstructed motion vector can be obtained by adding the predicted motion vector and the differential motion vector. The example may be a case where the motion estimation and compensation processing, the representation unit (precision) and the storage unit of the motion vector are fixed to one precision (for example, it is unified i...

Claims

1. An inter-frame prediction method performed by an image decoding device, comprising: determining a motion vector prediction candidate from the motion vectors of the reference block; determining a motion vector prediction candidate group including the motion vector prediction candidate; deriving a predicted motion vector of a target block from the motion vector prediction candidate group; determining a differential motion vector accuracy among a plurality of accuracy candidates based on the differential motion vector accuracy information; Determining a differential motion vector of the target block; scaling the differential motion vector of the target block based on a precision difference between the differential motion vector precision and a motion vector precision of the predicted motion vector; deriving a motion vector of a target block based on the predicted motion vector of the target block and the scaled differential motion vector of the target block; as well as Inter prediction is performed by using a motion vector of a target block.

2. An inter-frame prediction method performed by an image encoding device, comprising: derive the motion vector of the target block; performing inter-frame prediction by using the motion vector of the target block; Specifying a reference block of the target block; determining a motion vector prediction candidate from a motion vector of the reference block; determining a motion vector prediction candidate group including the motion vector prediction candidate; deriving a predicted motion vector of the target block from the motion vector prediction candidate group; determining a differential motion vector of the target block based on the predicted motion vector of the target block; determining a differential motion vector accuracy among a plurality of accuracy candidates; scaling the differential motion vector of the target block based on a precision difference between the differential motion vector precision and a motion vector precision of the predicted motion vector; as well as The motion vector of the target block is encoded based on the predicted motion vector of the target block and the scaled differential motion vector of the target block, and differential motion vector accuracy information indicating accuracy of the differential motion vector is encoded.

3. A non-transitory computer-readable recording medium storing a bit stream generated by an inter-frame prediction method performed by an image encoding device, the method comprising: derive the motion vector of the target block; performing inter-frame prediction by using the motion vector of the target block; Specifying a reference block of the target block; determining a motion vector prediction candidate from a motion vector of the reference block; determining a motion vector prediction candidate group including the motion vector prediction candidate; deriving a predicted motion vector of the target block from the motion vector prediction candidate group; determining a differential motion vector of the target block based on the predicted motion vector of the target block; determining a differential motion vector accuracy among a plurality of accuracy candidates; scaling the differential motion vector of the target block based on a precision difference between the differential motion vector precision and a motion vector precision of the predicted motion vector; as well as The motion vector of the target block is encoded based on the predicted motion vector of the target block and the scaled differential motion vector of the target block, and differential motion vector accuracy information indicating accuracy of the differential motion vector is encoded.

4. An inter-frame prediction method performed by an image decoding device, comprising: Determining a merge candidate list for the current block including the merge candidate; Selecting a merge candidate from the merge candidate list; as well as performing inter prediction based on the motion vector and reference picture of the selected merge candidate, and The merge candidate list includes spatial merge candidates and temporal merge candidates. In a case where the number of the spatial merge candidates and the temporal merge candidates is less than a maximum number of the merge candidate list, the merge candidate list further includes a statistical merge candidate derived from a previously decoded block that belongs to the current picture and is not adjacent to the current block, In a case where the number of the spatial merge candidates, the temporal merge candidates, and the statistical merge candidates is less than a maximum number of the merge candidate list, the merge candidate list further includes a combined merge candidate derived from two pre-included merge candidates, and When the merge flag information indicates that the current block is predicted by a merge mode, the merge candidate list is determined.

5. The method according to claim 4, wherein: The statistical merging list includes motion information of a predetermined number of previously decoded blocks, and the statistical merging list is managed using a FIFO (First In First Out) method.

6. The method according to claim 5, wherein: The statistical merge list includes motion information of previously decoded blocks and does not include motion information of previously decoded sub-blocks. 7 . The method according to claim 4 , wherein the motion vector of the combined merge candidate is derived by averaging the motion vectors of the first merge candidate and the second merge candidate in the merge candidate list.

8. An inter-frame prediction method performed by an image encoding device, comprising: Determining a merge candidate list for the current block including the merge candidate; Selecting a merge candidate from the merge candidate list; as well as performing inter prediction based on the motion vector and reference picture of the selected merge candidate, and The merge candidate list includes spatial merge candidates and temporal merge candidates. In a case where the number of the spatial merge candidates and the temporal merge candidates is less than a maximum number of the merge candidate list, the merge candidate list further includes a statistical merge candidate derived from a previously decoded block that belongs to the current picture and is not adjacent to the current block, In a case where the number of the spatial merge candidates, the temporal merge candidates, and the statistical merge candidates is less than a maximum number of the merge candidate list, the merge candidate list further includes a combined merge candidate derived from two pre-included merge candidates, and When the merge flag information indicates that the current block is predicted by a merge mode, the merge candidate list is determined.

9. A non-transitory computer-readable recording medium storing a bit stream generated by an inter-frame prediction method performed by an image encoding device, the method comprising: Determining a merge candidate list for the current block including the merge candidate; Selecting a merge candidate from the merge candidate list; as well as performing inter prediction based on the motion vector and reference picture of the selected merge candidate, and The merge candidate list includes spatial merge candidates and temporal merge candidates. In a case where the number of the spatial merge candidates and the temporal merge candidates is less than a maximum number of the merge candidate list, the merge candidate list further includes a statistical merge candidate derived from a previously decoded block that belongs to the current picture and is not adjacent to the current block, In a case where the number of the spatial merge candidates, the temporal merge candidates, and the statistical merge candidates is less than a maximum number of the merge candidate list, the merge candidate list further includes a combined merge candidate derived from two pre-included merge candidates, and When the merge flag information indicates that the current block is predicted by a merge mode, the merge candidate list is determined.

10. An inter-frame prediction method performed by an image decoding device, comprising: Deriving the motion information coding mode of the target block; specifying a reference block according to a derived motion information coding mode; configuring a motion information prediction candidate group based on a specified reference block and the motion information coding mode; deriving motion information of the target block from the motion information prediction candidate group; as well as performing inter-frame prediction by using the motion information of the target block, Wherein, in a case where the affine model is applied to the target block, configuring the motion information prediction candidate group includes: determining whether the affine model is applied to the reference block; In a case where the affine model is determined to be applied to the reference block, including the reference block as a candidate in the motion information prediction candidate group; constructing a combination candidate by combining a plurality of translation motion information; and Including the combined candidate into the motion information prediction candidate group, Wherein, after the reference block to which the affine model is applied is included in the motion information prediction candidate group, the combined candidate is included in the motion information prediction candidate group.

11. An inter-frame prediction method performed by an image encoding device, comprising: Export the motion information of the target block; performing inter-frame prediction by using the motion information of the target block; Determining a motion information coding mode of the target block; specifying a reference block according to the determined motion information coding mode; configuring a motion information prediction candidate group based on a specified reference block and the motion information coding mode; as well as encoding the motion information of the target block based on the motion information prediction candidate group, Wherein, in a case where the affine model is applied to the target block, configuring the motion information prediction candidate group includes: determining whether the affine model is applied to the reference block; In a case where the affine model is determined to be applied to the reference block, including the reference block as a candidate in the motion information prediction candidate group; constructing a combination candidate by combining a plurality of translation motion information; and Including the combined candidate into the motion information prediction candidate group, Wherein, after the reference block to which the affine model is applied is included in the motion information prediction candidate group, the combined candidate is included in the motion information prediction candidate group.

12. A non-transitory computer-readable recording medium storing a bit stream generated by an inter-frame prediction method performed by an image encoding device, the method comprising: Export the motion information of the target block; performing inter-frame prediction by using the motion information of the target block; Determining a motion information coding mode of the target block; specifying a reference block according to the determined motion information coding mode; configuring a motion information prediction candidate group based on a specified reference block and the motion information coding mode; as well as encoding the motion information of the target block based on the motion information prediction candidate group, Wherein, in a case where the affine model is applied to the target block, configuring the motion information prediction candidate group includes: determining whether the affine model is applied to the reference block; In a case where the affine model is determined to be applied to the reference block, including the reference block as a candidate in the motion information prediction candidate group; constructing a combination candidate by combining a plurality of translation motion information; and Including the combined candidate into the motion information prediction candidate group, Wherein, after the reference block to which the affine model is applied is included in the motion information prediction candidate group, the combined candidate is included in the motion information prediction candidate group.