Encoding method and apparatus therefor and decoding method and apparatus therefor

Abstract

An encoding method and apparatus thereof, and a decoding method and apparatus thereof, are provided. The decoding method includes the following steps: when a current block is inter-frame predicted, determining an inter-frame prediction mode for the current block; determining at least one reference sample location for the current block based on the inter-frame prediction mode; determining filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location based on the inter-frame prediction mode; performing filtering on the at least one reconstructed reference sample based on the filter information; and decoding the current block using prediction samples generated via filtering.

Description

[0001] This application is a divisional application of the invention patent application filed on January 7, 2019, with application number "201980008315.1" and titled "Encoding method and apparatus thereof and decoding method and apparatus thereof". Technical Field

[0002] This disclosure relates to a video encoding and decoding method, and more specifically, to a method of applying an interpolation filter to reference samples for inter-frame prediction. Background Technology

[0003] With the development and availability of hardware capable of reproducing and storing high-resolution or high-quality image content, the demand for codecs capable of efficiently encoding or decoding such high-resolution or high-quality image content has increased. Encoded image content can be reproduced by decoding. Recently, methods for efficiently compressing such high-resolution or high-quality image content have been implemented.

[0004] To compress an image, various data units can be used, and there can be inclusion relationships between data units. Data units can be divided according to various methods to determine the size of the data units used for image compression, and the image can be encoded or decoded by determining the optimal data units based on the characteristics of the image.

[0005] Image data can be compressed by removing spatial and temporal redundancy between pixels. Because neighboring pixels often share common characteristics, encoded information is sent in data units of multiple pixels to remove redundancy between neighboring pixels.

[0006] Instead of directly sending the pixel values ​​of the pixels included in the data unit, the method required to obtain the pixel values ​​is sent. For each data unit, a prediction method is determined to predict pixel values ​​similar to the original values, and encoded information about the prediction method is sent from the encoder to the decoder. Furthermore, because the predicted values ​​are not exactly the same as the original values, residual data about the difference between the original and predicted values ​​is sent from the encoder to the decoder.

[0007] Prediction methods include intra-frame prediction and inter-frame prediction. Intra-frame prediction is a method that predicts pixels within a block based on neighboring pixels. Inter-frame prediction is a method that predicts pixels by referencing pixels in another frame that includes the block. Therefore, spatial redundancy is removed via intra-frame prediction, and temporal redundancy is removed via inter-frame prediction. Summary of the Invention

[0008] Technical issues

[0009] This disclosure proposes a video coding method and a video coding device, as well as a video decoding method and a video decoding device, wherein a filter is adaptively applied to reference samples based on the inter-frame prediction mode of the current block.

[0010] The technical problems to be solved in this embodiment are not limited to those described above, and those skilled in the art will clearly understand from the following description other technical problems not mentioned.

[0011] Solution to the problem

[0012] According to various embodiments of this disclosure, a video decoding method includes: when a current block is inter-frame predicted, determining an inter-frame prediction mode for the current block; determining, based on the inter-frame prediction mode of the current block, at least one reference sample location to be referenced by the current block; determining, based on the inter-frame prediction mode of the current block, filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location; performing filtering on the at least one reconstructed reference sample based on the filter information; and decoding the current block using prediction samples generated via filtering.

[0013] According to various embodiments of this disclosure, a video decoding apparatus includes: at least one processor; and a memory, wherein the memory stores at least one instruction configured to be executable by the at least one processor, and the at least one instruction is configured to, when executed, cause the at least one processor to: determine an inter-frame prediction mode for a current block when the current block is inter-predicted; determine, based on the inter-frame prediction mode of the current block, at least one reference sample location to be referenced by the current block; determine, based on the inter-frame prediction mode of the current block, filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location; perform filtering on the at least one reconstructed reference sample based on the filter information; and decode the current block using prediction samples generated via filtering.

[0014] According to various embodiments of this disclosure, a video coding method includes: when a current block is inter-frame predicted, determining an inter-frame prediction mode for the current block; determining, based on the inter-frame prediction mode of the current block, at least one reference sample location to be referenced by the current block; determining, based on the inter-frame prediction mode of the current block, filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location; performing filtering on the at least one reconstructed reference sample based on the filter information; and encoding the current block using prediction samples generated via filtering.

[0015] According to various embodiments of this disclosure, a video encoding apparatus includes: at least one processor; and a memory, wherein the memory stores at least one instruction configured to be executable by the at least one processor, and the at least one instruction is configured to, when executed, cause the at least one processor to: determine an inter-frame prediction mode for a current block when the current block is inter-predicted; determine at least one reference sample location to be referenced by the current block based on the inter-frame prediction mode of the current block; determine filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location based on the inter-frame prediction mode of the current block; perform filtering on the at least one reconstructed reference sample based on the filter information; and encode the current block using prediction samples generated via filtering.

[0016] According to various embodiments of this disclosure, a non-transitory computer-readable recording medium has thereon recorded a program for performing the video encoding method.

[0017] According to various embodiments of this disclosure, a non-transitory computer-readable recording medium has thereon recorded a program for performing the video decoding method.

[0018] Publicly disclosed beneficial effects

[0019] Video coding methods and apparatus according to various embodiments, as well as video decoding methods and apparatus, can improve the efficiency of resources used in inter-frame prediction by adaptively applying filters to reference samples based on the inter-frame prediction mode of the current block during inter-frame prediction. For example, power consumption and battery usage can be reduced by adjusting the memory bandwidth used for inter-frame prediction to an appropriate level.

[0020] The effects that can be obtained in this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. Attached Figure Description

[0021] Figure 1a This is a block diagram of an image decoding device according to various embodiments.

[0022] Figure 1b This is a block diagram of an image decoding device according to various embodiments.

[0023] Figure 2 This is a block diagram of an image decoding device according to various embodiments.

[0024] Figure 3 This is a flowchart of an image decoding method according to various embodiments.

[0025] Figure 4a It is a block diagram of an image encoding device according to various embodiments.

[0026] Figure 4b It is a block diagram of an image encoding device according to various embodiments.

[0027] Figure 5 It is a block diagram of an image encoding device according to various embodiments.

[0028] Figure 6 This is a flowchart of an image encoding method according to various embodiments.

[0029] Figure 7 The diagram illustrates a process performed by an image decoding device, according to various embodiments, to determine at least one coding unit by dividing the current coding unit.

[0030] Figure 8 The diagram illustrates a process performed by an image decoding device according to various embodiments to determine at least one coding unit by dividing non-square coding units.

[0031] Figure 9 The diagram illustrates a process performed by an image decoding device according to various embodiments to divide coding units based on at least one of block shape information and partition shape pattern information.

[0032] Figure 10 The present invention illustrates a method for determining a predetermined coding unit from an odd number of coding units, performed by an image decoding device according to various embodiments.

[0033] Figure 11 The diagram illustrates the order in which an image decoding device processes multiple coding units when it determines such units by dividing the current coding unit, according to various embodiments.

[0034] Figure 12 The diagram illustrates a process performed by an image decoding device according to various embodiments, in which the current encoding unit is determined to be divided into an odd number of encoding units when the encoding units cannot be processed in a predetermined order.

[0035] Figure 13 The diagram illustrates a process performed by an image decoding device according to various embodiments, which involves dividing a first coding unit to determine at least one coding unit.

[0036] Figure 14 The diagram illustrates that, according to various embodiments, when a second coding unit having a non-square shape is determined during the division of a first coding unit in an image decoding device, the shape into which the second coding unit can be divided is limited if predetermined conditions are met.

[0037] Figure 15The diagram illustrates the process performed by an image decoding device according to various embodiments, when the partitioning shape pattern information cannot indicate that a square coding unit will be partitioned into four square coding units, in order to partition a square coding unit.

[0038] Figure 16 This illustrates how, according to various embodiments, the processing order among multiple coding units can be changed based on the process of dividing coding units.

[0039] Figure 17 The diagram illustrates a process for determining the depth of a coding unit when the shape and size of the coding unit change, as various embodiments of the diagram illustrate the process of determining the depth of a coding unit when the coding unit is recursively divided such that multiple coding units are determined.

[0040] Figure 18 The diagram illustrates the depth that can be determined based on the shape and size of the coding unit, according to various embodiments, and the partial index (PID) used to distinguish the coding unit.

[0041] Figure 19 This illustrates how multiple encoding units are determined based on multiple predetermined data units included in a screen, according to various embodiments.

[0042] Figure 20 The diagram illustrates a processing block, according to various embodiments, used as a unit for determining the order of reference coding units included in a frame.

[0043] Figure 21a and Figure 21b This is a diagram illustrating a method for performing filtering on reference samples for motion compensation during inter-frame prediction according to various embodiments.

[0044] Figure 22a The reference samples required for inter-frame prediction of the current block according to various embodiments are shown.

[0045] Figure 22b The reference samples required for inter-frame prediction of the current block when the current block references multiple reference blocks are shown according to various embodiments.

[0046] Figure 23 This illustrates the memory bandwidth required for inter-frame prediction of the current block according to various embodiments.

[0047] Figure 24 The reference samples required for inter-frame prediction of the current block are shown according to various embodiments when different filters are used in the vertical and horizontal directions.

[0048] Figure 25a This paper illustrates prediction methods for Overlapping Block Motion Compensation (OBMC) patterns without using sub-blocks, according to various embodiments.

[0049] Figure 25bThe reference samples required for inter-frame prediction of the current block in OBMC mode without using sub-blocks are shown according to various embodiments.

[0050] Figure 25c The diagram illustrates the memory bandwidth required for inter-frame prediction of the current block in OBMC mode without using sub-blocks, according to various embodiments.

[0051] Figure 26a This illustrates prediction methods for OBMC patterns using sub-blocks according to various embodiments.

[0052] Figure 26b The reference samples required for inter-frame prediction of the current block in OBMC mode using sub-blocks are shown according to various embodiments.

[0053] Figure 26c The memory bandwidth required for inter-frame prediction of the current block in OBMC mode using sub-blocks is shown according to various embodiments.

[0054] Figure 27a This paper illustrates prediction methods for decoder-side motion vector correction (DMVR) modes according to various embodiments.

[0055] Figure 27b The reference samples required for inter-frame prediction of the current block in DMVR mode are shown according to various embodiments.

[0056] Figure 27c This illustrates the memory bandwidth required for inter-frame prediction of the current block in DMVR mode according to various embodiments.

[0057] Figure 28a The reference samples required for inter-frame prediction of the current block in DMVR mode using multiple motion vector candidates are shown according to various embodiments.

[0058] Figure 28b The memory bandwidth required for inter-frame prediction of the current block in DMVR mode using multiple motion vector candidates is shown according to various embodiments.

[0059] Figure 28c The reference samples required for inter-frame prediction of the current block in a search-range-limited DMVR mode, according to various embodiments, are shown.

[0060] Figure 28d This illustrates the memory bandwidth required for inter-frame prediction of the current block in a search-range-limited DMVR mode, according to various embodiments.

[0061] Figure 29a The prediction patterns of affine patterns according to various embodiments are shown.

[0062] Figure 29bThis illustrates the memory bandwidth required for inter-frame prediction of the current block in affine mode according to various embodiments.

[0063] Figure 29c The reference samples required for inter-frame prediction of the current block in affine mode are shown according to various embodiments.

[0064] Figure 29d This illustrates the memory bandwidth required for inter-frame prediction of the current block in affine mode according to various embodiments.

[0065] Figure 30 The predicted modes of bidirectional optical flow (BIO) modes according to various embodiments are shown. Detailed Implementation

[0066] The advantages and features of one or more embodiments, as well as methods for implementing said one or more embodiments, can be more readily understood by referring to the embodiments and accompanying drawings. In this regard, embodiments of this disclosure may take different forms and should not be construed as limited to the description set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the embodiments proposed in this disclosure to those skilled in the art.

[0067] The terminology used in this specification will be briefly defined, and the embodiments will be described in detail.

[0068] All terms used herein, including descriptive or technical terms, shall be interpreted as having meanings obvious to a person skilled in the art. However, these terms may have different meanings depending on the intent of a person skilled in the art, precedent, or the emergence of new technologies. Furthermore, some terms may be arbitrarily chosen by the applicant, and in such cases, the meaning of the chosen term will be described in detail in this disclosure. Therefore, the terms used herein must be defined based on their meanings in conjunction with the description throughout the specification.

[0069] In the following description, the singular form includes the plural form unless the context clearly indicates otherwise.

[0070] When a component "comprises" or "includes" an element, the component may also include other elements, without excluding other elements, unless there is a specific description to the contrary.

[0071] The term "cell" as used in this specification refers to a software component or hardware component, such as a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC), and that the "cell" performs a specific function. However, a "cell" is not limited to software or hardware. A "cell" may be formed in addressable memory or may be formed to operate one or more processors. Thus, for example, the term "cell" may refer to a component (such as a software component, an object-oriented software component, a class component, and a task component) and may include processes, functions, attributes, procedures, subroutines, fragments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The functionality provided by components and "cells" may be associated with a smaller number of components and "cells," or may be divided into additional components and "cells."

[0072] According to embodiments of this disclosure, a "unit" may include a processor and memory. The term "processor" should be broadly interpreted to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some cases, "processor" may refer to application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc. The term "processor" may refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such configurations.

[0073] The term "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. "Memory" can refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage devices, registers, etc. When a processor can read information from and / or write information to a memory, the memory is said to be in electronic communication with the processor. Memory integrated into a processor is in electronic communication with the processor.

[0074] In the following text, “image” can be a still image, such as a video, or a dynamic image, such as a moving image, i.e., the video itself.

[0075] In the following text, "current block" may refer to one of the coding unit, prediction unit, and transform unit currently being encoded or decoded. Furthermore, "lower block" may refer to a data unit partitioned from the "current block." "Upper block" may refer to a data unit that includes the "current block."

[0076] In the following text, a "sample" refers to data assigned to a sampling location in an image, i.e., the data to be processed. For example, pixel values ​​in the spatial domain and transform coefficients in the transform domain can be samples. A unit comprising at least one such sample can be defined as a block.

[0077] In the following, embodiments will be described in detail with reference to the accompanying drawings, enabling those skilled in the art to readily implement the embodiments. In the drawings, portions irrelevant to the description have been omitted to clearly illustrate this disclosure.

[0078] In the following text, reference will be made to Figures 1a to 6 This describes an image coding apparatus and an image decoding apparatus, as well as an image coding method and an image decoding method, that adaptively perform inter-frame prediction based on the current block's inter-frame prediction mode according to various embodiments. Furthermore, reference will be made to... Figures 7 to 20 This document describes video encoding and decoding apparatuses based on tree-structured coding units according to various embodiments, as well as video encoding and decoding methods. Furthermore, reference will be made to… Figures 21a to 30 Description can be applied according to Figures 1a to 20 Various embodiments of the image encoding method, image decoding method, video encoding method, and video decoding method.

[0079] Figure 1a This is a block diagram of an image decoding device 100 according to various embodiments.

[0080] Image decoding device 100 may include receiver 105, inter-frame predictor 110, and decoder 115. Receiver 105, inter-frame predictor 110, and decoder 115 may each include at least one processor. Furthermore, receiver 105, inter-frame predictor 110, and decoder 115 may include memory storing instructions to be executed by the at least one processor.

[0081] Receiver 105 can receive bitstreams. The bitstreams include information about an image encoded by image encoding device 400, which will be described later. Furthermore, bitstreams can be transmitted from image encoding device 400. Image encoding device 400 and image decoding device 100 can be connected via wired or wireless means, and receiver 105 can receive bitstreams via wired or wireless means. Receiver 105 can receive bitstreams from storage media (such as optical media or hard disks).

[0082] Receiver 105 can obtain information about the prediction mode of the current block from the bitstream. This information may include information indicating an intra-frame prediction mode or an inter-frame prediction mode. When the prediction mode of the current block is an inter-frame prediction mode, receiver 105 can obtain information about the inter-frame prediction mode of the current block from the bitstream. This information may be about the inter-frame prediction mode applied to the current block among multiple inter-frame prediction modes. For example, the inter-frame prediction mode of the current block may be at least one of the following: merge mode, advanced motion vector prediction (AMVP) mode, inter-frame skip mode, overlapped block motion compensation (OBMC) mode, decoder-side motion vector correction (DMVR) mode, affine mode, and bidirectional optical flow (BIO) mode.

[0083] When the prediction mode of the current block is inter-frame prediction mode, the inter-frame predictor 110 can be activated. The inter-frame predictor 110 can determine the inter-frame prediction mode of the current block based on information about the inter-frame prediction mode of the current block obtained from the bitstream.

[0084] Inter-frame predictor 110 can determine the position of at least one reference sample point to be referenced by the current block based on the inter-frame prediction mode of the current block. Inter-frame predictor 110 can obtain at least one motion vector and at least one reference frame index of the current block based on the inter-frame prediction mode of the current block, and can determine the position indicated by the motion vector within the reference frame indicated by the reference frame index as the reference sample point position. According to various embodiments, the reference sample point position can be determined as coordinates obtained by adding the motion vector component of the current block to the upper-left coordinate of the current block. The reference sample point position can indicate the upper-left position of the reference block referenced by the current block within the reference frame.

[0085] According to various embodiments, for further accurate prediction, the reference sample point position may indicate a sub-pixel located between integer pixels. For example, the reference sample point position may indicate a sub-pixel at a 1 / 4 pixel unit. However, the reference sample point position is not limited to this, and the sub-pixel unit may be 1 / 8 pixel unit or smaller.

[0086] According to various embodiments, the number of reference sample locations, equal to the number of motion vectors, can be determined for use in the inter-frame prediction mode of the current block. For example, one reference sample location can be determined when the current block is predicted unidirectionally, and two reference sample locations can be determined when the current block is predicted bidirectionally.

[0087] Inter-frame predictor 110 can determine filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location based on the inter-frame prediction mode of the current block. Inter-frame predictor 110 can apply interpolation filters to the reference samples at integer pixel locations and determine information about the interpolation filters to generate reference samples at sub-pixel locations between the reference samples at integer pixel locations.

[0088] According to various embodiments, the inter-frame predictor 110 may determine at least one reconstructed reference sample to which the filter will be applied based on the location of reference samples within the reconstructed reference frame. According to some embodiments, the at least one reconstructed reference sample may include samples whose locations are within a specific range. According to some embodiments, the at least one reconstructed reference sample may include samples in a reference block referenced by the current block and neighboring samples adjacent to the reference block of the current block. According to some embodiments, the at least one reconstructed reference sample may also include reference samples referenced by neighboring blocks adjacent to the current block. According to some embodiments, the at least one reconstructed reference sample may also include neighboring samples adjacent to reference blocks referenced by neighboring blocks of the current block.

[0089] According to some embodiments, the inter-frame predictor 110 may determine reference samples at integer pixel positions within a specific range based on the reference sample positions as at least one reconstructed reference sample to which the interpolation filter will be applied. According to some embodiments, the at least one reconstructed reference sample may include reference samples at integer pixel positions required to generate reference samples at sub-pixel positions included in a reference block referenced by the current block. According to some embodiments, the at least one reconstructed reference sample may further include reference samples at integer pixel positions required to generate neighboring reference samples at sub-pixel positions adjacent to the reference block of the current block. According to some embodiments, the at least one reconstructed reference sample may further include reference samples at integer pixel positions required to generate reference samples at sub-pixel positions referenced by neighboring blocks adjacent to the current block.

[0090] According to various embodiments, filter information may include at least one of the filter's orientation and size. According to various embodiments, inter-frame predictor 110 may determine to use a separable 2D filter. According to embodiments, inter-frame predictor 110 may determine to apply each of a horizontal direction filter and a vertical direction filter.

[0091] According to various embodiments, the inter-frame predictor 110 can determine the size of the filter by setting the number of filter taps to a specific value. The number of filter taps can represent the number of reference samples at integer pixel locations required to generate reference samples at sub-pixel locations. Here, the specific value can be determined based on the inter-frame prediction mode of the current block.

[0092] According to various embodiments, the inter-frame predictor 110 may determine filter information based on the size of the current block. According to some embodiments, the inter-frame predictor 110 may determine the size of the filter based on the size of the current block.

[0093] According to an embodiment, the inter-frame predictor 110 may determine to use a first filter if the size of the current block is equal to or greater than a specific reference value; otherwise, it may determine to use a second filter. According to an embodiment, the size of the first filter may be larger than the size of the second filter. According to an embodiment, the specific reference value may be determined based on the inter-frame prediction mode of the current block. According to an embodiment, the inter-frame predictor 110 may compare the size of the current block with multiple reference values ​​and determine the filter to be used from among multiple filters based on the comparison results.

[0094] According to an embodiment, the inter-frame predictor 110 can determine horizontal direction filter information based on the horizontal width of the current block, and can determine vertical direction filter information based on the vertical height of the current block. According to an embodiment, when the current block is not square, the inter-frame predictor 110 can determine the horizontal direction filter and the vertical direction filter as filters of different sizes.

[0095] According to an embodiment, the inter-frame predictor 110 can determine horizontal and vertical filter information based on the ratio of the horizontal width to the vertical height of the current block. For example, when the horizontal filter information of the current block is determined, the inter-frame predictor 110 can determine the vertical filter information based on the horizontal filter information and the ratio of the horizontal width to the vertical height of the current block. In another example, when the vertical filter information of the current block is determined, the inter-frame predictor 110 can determine the horizontal filter information based on the vertical filter information and the ratio of the horizontal width to the vertical height of the current block. According to an embodiment, the inter-frame predictor 110 can determine the size of the horizontal filter and the size of the vertical filter to be proportional to the horizontal width and vertical height of the current block.

[0096] According to various embodiments, the inter-frame predictor 110 can determine filter information based on whether the current block is a luma block or a chroma block. According to embodiments, the inter-frame predictor 110 can determine to use a first filter when the current block is a luma block, and determine to use a second filter when the current block is a chroma block. According to embodiments, the size of the first filter can be twice the size of the second filter.

[0097] According to various embodiments, the inter-frame predictor 110 can determine filter information based on whether the inter-frame prediction of the current block is unidirectional or bidirectional. According to embodiments, the inter-frame predictor 110 can determine to use a first filter when the current block is unidirectionally predicted, and determine to use a second filter when the current block is bidirectionally predicted. According to embodiments, the size of the first filter can be twice the size of the second filter.

[0098] According to various embodiments, the inter-frame predictor 110 can determine filter information based on the number of reference blocks referenced by the current block. According to an embodiment, the image decoding device 100 can determine to use a filter with a first number of taps when the number of reference blocks referenced by the current block is less than or equal to a specific reference value; otherwise, it determines to use a filter with a second number of taps less than the first number of taps. According to an embodiment, the inter-frame predictor 110 can compare the number of reference blocks referenced by the current block with multiple reference values ​​and determine the filter to be used from among multiple filters based on the comparison results.

[0099] According to various embodiments, the inter-frame predictor 110 can determine the maximum size of the filter based on the inter-frame prediction mode of the current block. For example, when the inter-frame prediction mode of the current block is OBMC mode, the inter-frame predictor 110 can determine the filter size to be less than or equal to 2 taps. For example, when the inter-frame prediction mode of the current block is DMVR mode, the inter-frame predictor 110 can determine the filter size to be less than or equal to 4 taps. For example, when the inter-frame prediction mode of the current block is affine mode, the inter-frame predictor 110 can determine the filter size to be less than or equal to 4 taps. For example, when the inter-frame prediction mode of the current block is BIO mode, the inter-frame predictor 110 can determine the filter size to be less than or equal to 8 taps.

[0100] According to various embodiments, the inter-frame predictor 110 can determine filtering information such that the memory bandwidth required for inter-frame prediction of the current block is within a specific range. Memory bandwidth can be defined as the number of reference samples retrieved from memory for predicting the current sample. The memory bandwidth can be determined based on the size of the current block and the number of reconstructed reference samples required for inter-frame prediction of the current block.

[0101] According to various embodiments, the inter-frame predictor 110 may determine whether to perform filtering on the at least one reconstructed reference sample based on at least one of the inter-frame prediction mode of the current block, the size of the current block, whether the current block is a luma block or a chroma block, and whether the inter-frame prediction of the current block is unidirectional or bidirectional.

[0102] Inter-frame predictor 110 may perform filtering on the at least one reconstructed reference sample based on filter information. According to various embodiments, inter-frame predictor 110 may generate reference samples at sub-pixel positions by applying an interpolation filter to the reconstructed reference samples at integer pixel positions based on the filter information. According to various embodiments, inter-frame predictor 110 may generate predicted samples corresponding to the samples of the current block by using the reference samples at integer pixel positions and the reference samples at sub-pixel positions.

[0103] Decoder 115 can decode the current block using prediction samples generated via filtering. According to various embodiments, decoder 115 can reconstruct the current block based on the prediction samples of the current block and the residual block of the current block.

[0104] Figure 1b This is a block diagram of an image decoding device 100 according to various embodiments. The image decoding device 100 may be... Figure 1a Image decoding device 100.

[0105] Image decoding apparatus 100 according to various embodiments may include memory 120 and at least one processor 125 for accessing memory 120. Operation of image decoding apparatus 100 according to various embodiments may be performed by a separate processor or under the control of a central processing unit. Furthermore, memory 120 of image decoding apparatus 100 may store data received from external sources and data generated by processor 125.

[0106] The memory 120 of the image decoding apparatus 100 according to various embodiments may include at least one instruction configured to be executable by at least one processor 125. The at least one instruction may be configured, when executed, to cause the at least one processor 125 to perform the following operations: when the current block is inter-frame predicted, determine an inter-frame prediction mode for the current block; based on the inter-frame prediction mode of the current block, determine at least one reference sample location to be referenced by the current block; based on the inter-frame prediction mode of the current block, determine filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location; perform filtering on the at least one reconstructed reference sample based on the filtering information; and decode the current block using prediction samples generated via filtering.

[0107] Figure 2 This is a block diagram of an image decoder 200 according to various embodiments.

[0108] The image decoder 200 according to various embodiments can perform operations by... Figure 1a or Figure 1b The image decoding device 100 processes operations to decode image data.

[0109] Reference Figure 2The entropy decoder 215 parses the encoded image data to be decoded and the encoding information required for decoding from the bitstream 205. The encoded image data is quantized transform coefficients, and the dequantizer 220 and the inverse transform 225 reconstruct the residual data from the quantized transform coefficients.

[0110] Intra-predictor 240 performs intra-prediction for each block. Inter-predictor 235 uses a reference frame obtained from the reconstructed frame buffer 230 to perform inter-prediction for each block. Inter-predictor 235 can be used with... Figure 1a The inter-frame predictor 110 corresponds to this.

[0111] In response to the summation of prediction data and residual data generated by the intra-frame predictor 240 or inter-frame predictor 235 for each block to reconstruct spatial domain data for the current image, the deblocking unit 245 and the sample adaptive offset (SAO) actuator 250 can output a filtered reconstructed image by performing loop filtering on the reconstructed spatial domain data. Furthermore, the reconstructed image stored in the reconstructed image buffer 230 can be output as a reference image.

[0112] In order for the image decoding device 100 to decode image data, the step-by-step operation of the image decoder 200 according to various embodiments can be performed block by block.

[0113] Figure 3 This is a flowchart of an image decoding method 300 according to various embodiments.

[0114] In operation S310, the image decoding device 100 can determine the inter-frame prediction mode of the current block. The image decoding device 100 can receive a bitstream and obtain information about the inter-frame prediction mode of the current block from the bitstream. The information about the inter-frame prediction mode of the current block can be information about the inter-frame prediction mode applied to the current block among a plurality of inter-frame prediction modes. For example, the inter-frame prediction mode can be at least one of the following: merge mode, advanced motion vector prediction (AMVP) mode, inter-frame skip mode, overlap block motion compensation (OBMC) mode, decoder-side motion vector correction (DMVR) mode, affine mode, and bidirectional optical flow (BIO) mode.

[0115] In operation S320, the image decoding device 100 can determine the position of at least one reference sample point to be referenced by the current block based on the inter-frame prediction mode of the current block. The image decoding device 100 can obtain at least one motion vector and at least one reference frame index of the current block based on the inter-frame prediction mode of the current block, and determine the position indicated by the motion vector within the reference frame indicated by the reference frame index as the reference sample point position.

[0116] According to various embodiments, for further accurate prediction, reference sample locations may indicate sub-pixels located between integer pixels. According to various embodiments, the number of reference sample locations, equal to the number of motion vectors, used in the inter-frame prediction mode of the current block can be determined.

[0117] In operation S330, the image decoding device 100 can determine, based on the inter-frame prediction mode of the current block, filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location. The image decoding device 100 can apply an interpolation filter to the reference sample at the integer pixel location and determine information about the interpolation filter to generate reference samples at sub-pixel locations between the reference samples at the integer pixel locations.

[0118] According to various embodiments, the image decoding device 100 can determine at least one reconstructed reference sample to which a filter will be applied based on the location of reference samples within a reconstructed reference frame. According to some embodiments, the at least one reconstructed reference sample may include samples whose locations are within a specific range. According to some embodiments, the at least one reconstructed reference sample may include samples in a reference block referenced by the current block and neighboring samples adjacent to the reference block of the current block. According to some embodiments, the at least one reconstructed reference sample may also include reference samples referenced by neighboring blocks adjacent to the current block. According to some embodiments, the at least one reconstructed reference sample may also include neighboring samples adjacent to reference blocks referenced by neighboring blocks of the current block.

[0119] According to some embodiments, the image decoding device 100 may determine reference samples at integer pixel positions within a specific range as at least one reconstructed reference sample to which an interpolation filter will be applied. According to some embodiments, the at least one reconstructed reference sample may include reference samples at integer pixel positions required to generate reference samples at sub-pixel positions included in a reference block referenced by the current block. According to some embodiments, the at least one reconstructed reference sample may further include reference samples at integer pixel positions required to generate neighboring reference samples at sub-pixel positions adjacent to the reference block of the current block. According to some embodiments, the at least one reconstructed reference sample may further include reference samples at integer pixel positions required to generate reference samples at sub-pixel positions referenced by a neighboring block adjacent to the current block.

[0120] According to various embodiments, filter information may include at least one of the filter's orientation and size. According to various embodiments, image decoding device 100 may determine to use a separable 2D filter. According to embodiments, image decoding device 100 may determine to apply each of a horizontal direction filter and a vertical direction filter.

[0121] According to various embodiments, the image decoding device 100 can determine the size of the filter by determining the number of filter taps to a specific value. Here, the specific value can be determined based on the inter-frame prediction mode of the current block.

[0122] According to various embodiments, the image decoding device 100 can determine filter information based on the size of the current block. According to an embodiment, the image decoding device 100 can determine to use a first filter if the size of the current block is equal to or greater than a specific reference value, otherwise determine to use a second filter. According to an embodiment, the size of the first filter may be larger than the size of the second filter. According to an embodiment, the specific reference value can be determined based on the inter-frame prediction mode of the current block. According to an embodiment, the image decoding device 100 can compare the size of the current block with multiple reference values ​​and determine the filter to be used from among multiple filters based on the comparison results.

[0123] According to an embodiment, the image decoding device 100 can determine horizontal filter information based on the horizontal width of the current block and vertical filter information based on the vertical height of the current block. According to an embodiment, when the current block is not square, the image decoding device 100 can determine the horizontal and vertical filters as filters of different sizes.

[0124] According to an embodiment, the image decoding device 100 can determine horizontal filter information and vertical filter information based on the ratio of the horizontal width to the vertical height of the current block. For example, when the horizontal filter information of the current block is determined, the image decoding device 100 can determine the vertical filter information based on the horizontal filter information and the ratio of the horizontal width to the vertical height of the current block. In another example, when the vertical filter information of the current block is determined, the image decoding device 100 can determine the horizontal filter information based on the vertical filter information and the ratio of the horizontal width to the vertical height of the current block. According to an embodiment, the image decoding device 100 can determine the size of the horizontal filter and the size of the vertical filter to be proportional to the horizontal width and vertical height of the current block.

[0125] According to various embodiments, the image decoding device 100 can determine filter information based on whether the current block is a luma block or a chroma block. According to embodiments, the image decoding device 100 can determine to use a first filter when the current block is a luma block, and determine to use a second filter when the current block is a chroma block. According to embodiments, the size of the first filter can be twice the size of the second filter.

[0126] According to various embodiments, the image decoding device 100 can determine filter information based on whether the inter-frame prediction of the current block is unidirectional or bidirectional. According to embodiments, the image decoding device 100 can determine to use a first filter when the current block is unidirectionally predicted, and determine to use a second filter when the current block is bidirectionally predicted. According to embodiments, the size of the first filter can be twice the size of the second filter.

[0127] According to various embodiments, the image decoding device 100 can determine filter information based on the number of reference blocks referenced by the current block. According to embodiments, the image decoding device 100 can compare the number of reference blocks referenced by the current block with multiple reference values ​​and determine the filter to be used from among multiple filters based on the comparison results.

[0128] According to various embodiments, the image decoding device 100 can determine the maximum size of the filter based on the inter-frame prediction mode of the current block. For example, when the inter-frame prediction mode of the current block is OBMC mode, the image decoding device 100 can determine the filter size to be less than or equal to 2 taps. For example, when the inter-frame prediction mode of the current block is DMVR mode, the image decoding device 100 can determine the filter size to be less than or equal to 4 taps. For example, when the inter-frame prediction mode of the current block is affine mode, the image decoding device 100 can determine the filter size to be less than or equal to 4 taps. For example, when the inter-frame prediction mode of the current block is BIO mode, the image decoding device 100 can determine the filter size to be less than or equal to 8 taps.

[0129] According to various embodiments, the image decoding device 100 can determine filter information such that the memory bandwidth required for inter-frame prediction of the current block is within a specific range. Memory bandwidth can be defined as the number of reference samples retrieved from memory for processing the current samples. The memory bandwidth can be determined based on the size of the current block and the number of reconstructed reference samples required for inter-frame prediction of the current block.

[0130] According to various embodiments, the image decoding device 100 may determine whether to perform filtering on the at least one reconstructed reference sample based on at least one of the inter-frame prediction mode of the current block, the size of the current block, whether the current block is a luma block or a chroma block, and whether the inter-frame prediction of the current block is unidirectional or bidirectional.

[0131] In operation S340, the image decoding device 100 may perform filtering on the at least one reconstructed reference sample based on filter information. According to various embodiments, the image decoding device 100 may generate reference samples at sub-pixel positions by applying an interpolation filter to the reconstructed reference samples at integer pixel positions based on the filter information. According to various embodiments, the image decoding device 100 may generate predicted samples corresponding to the samples of the current block by using the reference samples at integer pixel positions and the reference samples at sub-pixel positions.

[0132] In operation S350, the image decoding device 100 can decode the current block using predicted samples generated via filtering. According to various embodiments, the image decoding device 100 can reconstruct the current block based on the predicted samples of the current block and the residual block of the current block.

[0133] Figure 4a This is a block diagram of an image encoding device 400 according to various embodiments.

[0134] Image encoding device 400 may include an inter-frame predictor 405 and an encoder 410. The inter-frame predictor 405 and encoder 410 may include at least one processor. Furthermore, the inter-frame predictor 405 and encoder 410 may include memory storing instructions to be executed by the at least one processor.

[0135] When the prediction mode of the current block is an inter-frame prediction mode, the inter-frame predictor 405 can determine the inter-frame prediction mode applicable to the current block from among a plurality of inter-frame prediction modes. According to various embodiments, the inter-frame predictor 405 can determine the inter-frame prediction mode applicable to the current block based on the coding efficiency of the inter-frame prediction modes applicable to the current block. For example, the inter-frame prediction modes applicable to the current block can be merge mode, AMVP mode, inter-frame skip mode, OBMC mode, DMVR mode, affine mode, and BIO mode.

[0136] According to various embodiments, the inter-frame predictor 405 can determine the number of reference blocks that will be referenced by the current block. For example, the inter-frame predictor 405 can determine whether the current block will be predicted unidirectionally or bidirectionally.

[0137] Inter-frame predictor 405 may determine the position of at least one reference sample point to be referenced by the current block based on the inter-frame prediction mode of the current block. According to various embodiments, inter-frame predictor 405 may determine a reference block as the most similar block to the current block from at least one reference frame based on the inter-frame prediction mode of the current block, and perform motion prediction to determine a motion vector indicating the positional difference between the current block and the reference block.

[0138] Here, for more accurate motion prediction, reference samples at sub-pixel positions can be generated between reference samples at integer pixel positions in the reference frame, and a motion vector indicating the sub-pixel unit of the reference sample at the sub-pixel position can be determined. For example, reference samples at sub-pixel positions can be generated in 1 / 4 pixel units. However, the reference samples are not limited to this, and reference samples at sub-pixel positions can be generated in 1 / 8 pixel units or smaller.

[0139] According to various embodiments, the inter-frame predictor 405 can determine a position in a reference frame indicated by a motion vector as a reference sample position. According to various embodiments, the reference sample position can indicate the upper-left position of a reference block within the reference frame referenced by the current block. The reference sample position can be determined as coordinates obtained by adding the motion vector component of the current block to the upper-left coordinates of the current block.

[0140] According to various embodiments, the number of reference sample locations, equal to the number of motion vectors, can be determined for use in the inter-frame prediction mode of the current block. For example, when the current block is predicted unidirectionally, one reference sample location can be determined. For example, when the current block is predicted bidirectionally, two reference sample locations can be determined.

[0141] The inter-frame predictor 405 can determine filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location based on the inter-frame prediction mode of the current block. The inter-frame predictor 405 can apply an interpolation filter to the reference sample at the integer pixel location and determine information about the interpolation filter to generate reference samples at sub-pixel locations between the reference samples at the integer pixel locations.

[0142] According to various embodiments, the inter-frame predictor 405 may determine at least one reconstructed reference sample to which the filter will be applied based on the location of reference samples within the reconstructed reference frame. According to some embodiments, the at least one reconstructed reference sample may include samples whose locations are within a specific range. According to some embodiments, the at least one reconstructed reference sample may include samples in a reference block referenced by the current block and neighboring samples adjacent to the reference block of the current block. According to some embodiments, the at least one reconstructed reference sample may also include reference samples referenced by neighboring blocks adjacent to the current block. According to some embodiments, the at least one reconstructed reference sample may also include neighboring samples adjacent to reference blocks referenced by neighboring blocks of the current block.

[0143] According to some embodiments, the inter-frame predictor 405 may determine reference samples at integer pixel positions within a specific range as at least one reconstructed reference sample to which the interpolation filter will be applied. According to some embodiments, the at least one reconstructed reference sample may include reference samples at integer pixel positions required to generate reference samples at sub-pixel positions included in a reference block referenced by the current block. According to some embodiments, the at least one reconstructed reference sample may further include reference samples at integer pixel positions required to generate neighboring reference samples at sub-pixel positions adjacent to the reference block of the current block. According to some embodiments, the at least one reconstructed reference sample may further include reference samples at integer pixel positions required to generate reference samples at sub-pixel positions referenced by neighboring blocks adjacent to the current block.

[0144] According to various embodiments, filter information may include at least one of the filter's orientation and size. According to various embodiments, inter-frame predictor 405 may determine to use a separable 2D filter. According to embodiments, inter-frame predictor 405 may determine to apply each of a horizontal direction filter and a vertical direction filter.

[0145] According to various embodiments, the inter-frame predictor 405 can determine the size of the filter by setting the number of filter taps to a specific value. The number of filter taps can represent the number of reference samples at integer pixel locations required to generate reference samples at sub-pixel locations. Here, the specific value can be determined based on the inter-frame prediction mode of the current block.

[0146] According to various embodiments, the inter-frame predictor 405 may determine filter information based on the size of the current block. According to some embodiments, the inter-frame predictor 405 may determine the size of the filter based on the size of the current block.

[0147] According to an embodiment, the inter-frame predictor 405 may determine to use a first filter if the size of the current block is equal to or greater than a specific reference value, otherwise determine to use a second filter. According to an embodiment, the size of the first filter may be larger than the size of the second filter. According to an embodiment, the specific reference value may be determined based on the inter-frame prediction mode of the current block. According to an embodiment, the inter-frame predictor 405 may compare the size of the current block with multiple reference values ​​and determine the filter to be used from among multiple filters based on the comparison results.

[0148] According to an embodiment, the inter-frame predictor 405 can determine horizontal filter information based on the horizontal width of the current block and vertical filter information based on the vertical height of the current block. According to an embodiment, when the current block is not square, the inter-frame predictor 405 can determine the horizontal and vertical filters to be filters of different sizes. According to an embodiment, when the current block is not square, the inter-frame predictor 405 can determine that the number of taps for the horizontal filter and the number of taps for the vertical filter are different from each other.

[0149] According to an embodiment, the inter-frame predictor 405 can determine horizontal and vertical filter information based on the ratio of the horizontal width to the vertical height of the current block. For example, when the horizontal filter information of the current block is determined, the inter-frame predictor 405 can determine the vertical filter information based on the horizontal filter information and the ratio of the horizontal width to the vertical height of the current block. In another example, when the vertical filter information of the current block is determined, the inter-frame predictor 405 can determine the horizontal filter information based on the vertical filter information and the ratio of the horizontal width to the vertical height of the current block. According to an embodiment, the inter-frame predictor 405 can determine the size of the horizontal filter and the size of the vertical filter to be proportional to the horizontal width and vertical height of the current block.

[0150] According to various embodiments, the inter-frame predictor 405 can determine filter information based on whether the current block is a luma block or a chroma block. According to embodiments, the inter-frame predictor 405 can determine to use a first filter when the current block is a luma block, and determine to use a second filter when the current block is a chroma block. According to embodiments, the size of the first filter can be twice the size of the second filter.

[0151] According to various embodiments, the inter-frame predictor 405 can determine filter information based on whether the inter-frame prediction of the current block is unidirectional or bidirectional. According to embodiments, the inter-frame predictor 405 can determine to use a first filter when the current block is unidirectionally predicted, and determine to use a second filter when the current block is bidirectionally predicted. According to embodiments, the size of the first filter can be twice the size of the second filter.

[0152] According to various embodiments, the inter-frame predictor 405 can determine filter information based on the number of reference blocks referenced by the current block. According to embodiments, the inter-frame predictor 405 can compare the number of reference blocks referenced by the current block with multiple reference values ​​and determine the filter to be used from among multiple filters based on the comparison results.

[0153] According to various embodiments, the inter-frame predictor 405 can determine the maximum size of the filter based on the inter-frame prediction mode of the current block. For example, when the inter-frame prediction mode of the current block is OBMC mode, the inter-frame predictor 405 can determine the filter size to be less than or equal to 2 taps. For example, when the inter-frame prediction mode of the current block is DMVR mode, the inter-frame predictor 405 can determine the filter size to be less than or equal to 4 taps. For example, when the inter-frame prediction mode of the current block is affine mode, the inter-frame predictor 405 can determine the filter size to be less than or equal to 4 taps. For example, when the inter-frame prediction mode of the current block is BIO mode, the inter-frame predictor 405 can determine the filter size to be less than or equal to 8 taps.

[0154] According to various embodiments, the inter-frame predictor 405 can determine filter information such that the memory bandwidth required for inter-frame prediction of the current block is within a specific range. Memory bandwidth can be defined as the number of reference samples retrieved from memory for encoding the current sample. The memory bandwidth can be determined based on the size of the current block and the number of reconstructed reference samples required for inter-frame prediction of the current block.

[0155] According to various embodiments, the inter-frame predictor 405 may determine whether to perform filtering on the at least one reconstructed reference sample based on at least one of the inter-frame prediction mode of the current block, the size of the current block, whether the current block is a luma block or a chroma block, and whether the inter-frame prediction of the current block is unidirectional or bidirectional.

[0156] Inter-frame predictor 405 may perform filtering on the at least one reconstructed reference sample based on filter information. According to various embodiments, inter-frame predictor 405 may generate reference samples at sub-pixel positions by applying an interpolation filter to the reconstructed reference samples at integer pixel positions based on filter information.

[0157] According to various embodiments, the inter-frame predictor 405 may determine at least one reference block most similar to the current block based on the inter-frame prediction mode of the current block, using reference samples at integer pixel positions and reference samples at sub-pixel positions. According to various embodiments, the inter-frame predictor 405 may determine the positional difference between the current block and the reference block as a motion vector.

[0158] According to various embodiments, the inter-frame predictor 405 can generate predicted samples from the samples of the reference block that correspond to the samples of the current block.

[0159] Encoder 410 can encode the current block using predicted samples generated via filtering. According to various embodiments, encoder 410 can obtain and encode the residual value between the sample values ​​of the current block and the predicted sample values ​​of the current block.

[0160] According to various embodiments, encoder 410 may encode information related to at least one motion vector of the current block. According to various embodiments, encoder 410 may encode at least one reference frame index of the current block. According to various embodiments, encoder 410 may encode information about the inter-frame prediction mode of the current block.

[0161] Figure 4b This is a block diagram of an image encoding device 400 according to various embodiments. The image encoding device 400 may be... Figure 4a Image encoding device 400.

[0162] Image encoding apparatus 400 according to various embodiments may include memory 420 and at least one processor 425 for accessing memory 420. Operation of image encoding apparatus 400 according to various embodiments may be performed by a separate processor or under the control of a central processing unit. Furthermore, memory 420 of image encoding apparatus 400 may store data received from external sources and data generated by processor 425.

[0163] The memory 420 of the image encoding apparatus 400 according to various embodiments may include at least one instruction configured to be executable by at least one processor 425. The at least one instruction may be configured to, when executed, cause the at least one processor 425 to perform the following operations: when the current block is inter-frame predicted, determine an inter-frame prediction mode for the current block; based on the inter-frame prediction mode of the current block, determine at least one reference sample location to be referenced by the current block; based on the inter-frame prediction mode of the current block, determine filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location; perform filtering on the at least one reconstructed reference sample based on the filter information; and encode the current block using prediction samples generated via filtering.

[0164] Figure 5 This is a block diagram of an image encoder 500 according to various embodiments.

[0165] The image encoder 500 according to various embodiments can perform operations by... Figure 4a or Figure 4b The image encoding device 400 processes operations to encode image data.

[0166] Intra-frame predictor 520 performs intra-frame prediction for each block in the current image 505, and inter-frame predictor 515 performs inter-frame prediction using the current image 505 and a reference image obtained from the reconstructed image buffer 510 for each block. Figure 5 The inter-frame predictor 520 can be used with Figure 4a The inter-frame predictor 405 corresponds to this.

[0167] Residual data is generated by subtracting the predicted data of each block output from the intra-predictor 520 or inter-predictor 515 from the data of the encoded blocks of the current image 505, and the transformer 525 and quantizer 530 can output quantized transform coefficients for each block by performing transform and quantization on the residual data.

[0168] The dequantizer 545 and inverse transform 550 reconstruct the spatial domain residual data by performing dequantization and inverse transform on the quantized transform coefficients. The reconstructed spatial domain residual data is reconstructed as spatial domain data for the blocks of the current image 505 by adding it to the prediction data for each block output from the intra-predictor 520 or inter-predictor 515. The deblocking unit 555 and SAO executor 560 perform intra-loop filtering on the reconstructed spatial domain data and generate a filtered reconstructed image. The generated reconstructed image is stored in the reconstructed image buffer 510. The reconstructed image stored in the reconstructed image buffer 510 can be used as a reference image for inter-frame prediction of another image. The entropy encoder 535 entropy-encodes the quantized transform coefficients, and the entropy-encoded coefficients can be output as a bitstream 540.

[0169] In order for the image encoding device 400 to encode image data, the step-by-step operation of the image encoder 500 according to various embodiments can be performed block by block.

[0170] Figure 6 This is a flowchart of an image encoding method 600 according to various embodiments.

[0171] During operation S610, the image coding device 400 can determine the inter-frame prediction mode for the current block. According to various embodiments, the image coding device 400 can determine the inter-frame prediction mode for the current block based on the coding efficiency of the inter-frame prediction modes applicable to the current block. For example, the inter-frame prediction modes applicable to the current block can be merge mode, AMVP mode, inter-frame skip mode, OBMC mode, DMVR mode, affine mode, and BIO mode.

[0172] According to various embodiments, the image encoding device 400 may determine the number of reference blocks that will be referenced by the current block. For example, the image encoding device 400 may determine whether the current block will be predicted unidirectionally or bidirectionally.

[0173] In operation S620, the image encoding device 400 may determine the position of at least one reference sample point to be referenced by the current block based on the inter-frame prediction mode of the current block. According to various embodiments, the image encoding device 400 may determine a reference block as the block most similar to the current block from at least one reference frame based on the inter-frame prediction mode of the current block, and perform motion prediction to determine a motion vector indicating the positional difference between the current block and the reference block.

[0174] Here, in order to further improve the accuracy of motion prediction, reference samples at sub-pixel positions between reference samples at integer pixel positions of the reference image can be generated, and motion vectors indicating the sub-pixel units of the determined reference samples can be determined.

[0175] According to various embodiments, the image encoding device 400 can determine the position in a reference frame indicated by a motion vector as a reference sample position. According to various embodiments, the number of reference sample positions, equal to the number of motion vectors, used in the inter-frame prediction mode of the current block can be determined.

[0176] In operation S630, the image encoding device 400 can determine, based on the inter-frame prediction mode of the current block, filter information to be applied to at least one reconstructed reference sample corresponding to the at least one reference sample location. The image encoding device 400 can apply an interpolation filter to the reference sample at the integer pixel location and determine information about the interpolation filter to generate reference samples at sub-pixel locations between the reference samples at the integer pixel locations.

[0177] According to various embodiments, the image encoding device 400 may determine at least one reconstructed reference sample to which a filter will be applied based on the location of reference samples within a reconstructed reference frame. According to some embodiments, the at least one reconstructed reference sample may include samples whose locations are within a specific range. According to some embodiments, the at least one reconstructed reference sample may include samples in a reference block referenced by the current block and neighboring samples adjacent to the reference block of the current block. According to some embodiments, the at least one reconstructed reference sample may also include reference samples referenced by neighboring blocks adjacent to the current block. According to some embodiments, the at least one reconstructed reference sample may also include neighboring samples adjacent to reference blocks referenced by neighboring blocks of the current block.

[0178] According to some embodiments, the image encoding device 400 may determine reference samples at integer pixel positions within a specific range as at least one reconstructed reference sample to which an interpolation filter will be applied. According to some embodiments, the at least one reconstructed reference sample may include reference samples at integer pixel positions required to generate reference samples at sub-pixel positions included in a reference block referenced by the current block. According to some embodiments, the at least one reconstructed reference sample may further include reference samples at integer pixel positions required to generate neighboring reference samples at sub-pixel positions adjacent to the reference block of the current block. According to some embodiments, the at least one reconstructed reference sample may further include reference samples at integer pixel positions required to generate reference samples at sub-pixel positions referenced by a neighboring block adjacent to the current block.

[0179] According to various embodiments, filter information may include at least one of the filter's orientation and size. According to various embodiments, image encoding device 400 may determine to use a separable 2D filter. According to embodiments, image encoding device 400 may determine to apply each of a horizontal direction filter and a vertical direction filter.

[0180] According to various embodiments, the image coding apparatus 400 can determine the size of the filter by determining the number of filter taps to a specific value. Here, the specific value can be determined based on the inter-frame prediction mode of the current block.

[0181] According to various embodiments, the image encoding device 400 can determine filter information based on the size of the current block. According to an embodiment, the image encoding device 400 can determine to use a first filter if the size of the current block is equal to or greater than a specific reference value; otherwise, it determines to use a second filter. According to an embodiment, the size of the first filter may be larger than the size of the second filter. According to an embodiment, the image encoding device 400 can compare the size of the current block with multiple reference values ​​and determine the filter to be used from among multiple filters based on the comparison results.

[0182] According to an embodiment, the image encoding device 400 can determine horizontal filter information based on the horizontal width of the current block and vertical filter information based on the vertical height of the current block. According to an embodiment, when the current block is not square, the image encoding device 400 can determine the horizontal and vertical filters to be filters of different sizes. According to an embodiment, when the current block is not square, the image encoding device 400 can determine that the number of taps for the horizontal filter and the number of taps for the vertical filter are different from each other.

[0183] According to an embodiment, the image encoding device 400 can determine horizontal and vertical filter information based on the ratio of the horizontal width to the vertical height of the current block. For example, when the horizontal filter information of the current block is determined, the image encoding device 400 can determine the vertical filter information based on the horizontal filter information and the ratio of the horizontal width to the vertical height of the current block. In another example, when the vertical filter information of the current block is determined, the image encoding device 400 can determine the horizontal filter information based on the vertical filter information and the ratio of the horizontal width to the vertical height of the current block. According to an embodiment, the image encoding device 400 can determine the size of the horizontal filter and the size of the vertical filter to be proportional to the horizontal width and vertical height of the current block.

[0184] According to various embodiments, the image encoding device 400 can determine filter information based on whether the current block is a luma block or a chroma block. According to embodiments, the image encoding device 400 can determine to use a first filter when the current block is a luma block, and determine to use a second filter when the current block is a chroma block. According to embodiments, the size of the first filter can be twice the size of the second filter.

[0185] According to various embodiments, the image coding device 400 can determine filter information based on whether the inter-frame prediction of the current block is unidirectional or bidirectional. According to embodiments, the image coding device 400 can determine to use a first filter when the current block is unidirectionally predicted, and determine to use a second filter when the current block is bidirectionally predicted. According to embodiments, the size of the first filter can be twice the size of the second filter.

[0186] According to various embodiments, the image encoding device 400 can determine filter information based on the number of reference blocks referenced by the current block. According to embodiments, the image encoding device 400 can compare the number of reference blocks referenced by the current block with multiple reference values ​​and determine the filter to be used from among multiple filters based on the comparison results.

[0187] According to various embodiments, the image coding device 400 can determine the maximum size of the filter based on the inter-frame prediction mode of the current block. For example, when the inter-frame prediction mode of the current block is OBMC mode, the image coding device 400 can determine the filter size to be less than or equal to 2 taps. For example, when the inter-frame prediction mode of the current block is DMVR mode, the image coding device 400 can determine the filter size to be less than or equal to 4 taps. For example, when the inter-frame prediction mode of the current block is affine mode, the image coding device 400 can determine the filter size to be less than or equal to 4 taps. For example, when the inter-frame prediction mode of the current block is BIO mode, the image coding device 400 can determine the filter size to be less than or equal to 8 taps.

[0188] According to various embodiments, the image coding apparatus 400 can determine filter information such that the memory bandwidth required for inter-frame prediction of the current block is within a specific range. Memory bandwidth can be defined as the number of reference samples retrieved from memory for processing the current samples. The memory bandwidth can be determined based on the size of the current block and the number of reconstructed reference samples required for inter-frame prediction of the current block.

[0189] According to various embodiments, the image coding device 400 may determine whether to perform filtering on the at least one reconstructed reference sample based on at least one of the inter-frame prediction mode of the current block, the size of the current block, whether the current block is a luma block or a chroma block, and whether the inter-frame prediction of the current block is unidirectional or bidirectional.

[0190] During operation S640, the image encoding device 400 may perform filtering on the at least one reconstructed reference sample based on filter information. According to various embodiments, the image encoding device 400 may generate reference samples at sub-pixel locations by applying an interpolation filter to the reconstructed reference samples at integer pixel locations based on the filter information.

[0191] According to various embodiments, the image coding device 400 can determine at least one reference block most similar to the current block based on the inter-frame prediction mode of the current block, using reference samples at integer pixel positions and reference samples at sub-pixel positions. According to various embodiments, the image coding device 400 can determine the positional difference between the current block and the reference block as a motion vector. According to various embodiments, the image coding device 400 can generate predicted samples corresponding to the samples of the current block from the samples of the reference block.

[0192] In operation S650, the image encoding device 400 can encode the current block using prediction samples generated via filtering. According to various embodiments, the image encoding device 400 can obtain and encode the residual value between the sample values ​​of the current block and the prediction sample values ​​of the current block. According to various embodiments, the image encoding device 400 can encode information related to at least one motion vector of the current block. According to various embodiments, the image encoding device 400 can encode at least one reference frame index of the current block. According to various embodiments, the image encoding device 400 can encode information about the inter-frame prediction mode of the current block.

[0193] The division of coding units will be described in detail below according to various embodiments of the present disclosure.

[0194] First, a frame can be divided into one or more stripes. A strip can be a sequence of one or more maximum coding units (coding tree units (CTUs)). Conceptually, there exists a maximum coding block (coding tree block (CTB) in contrast to a maximum coding unit (CTU).

[0195] A maximum coding unit (CTB) represents an N×N block containing N×N samples (N is an integer). Each color component can be divided into one or more maximum coding blocks.

[0196] When the image has three sample arrays (sample arrays for the Y, Cr, and Cb components), the Maximum Coding Unit (CTU) includes a maximum coding block for the luma samples, two corresponding maximum coding blocks for the chroma samples, and a syntax structure for encoding the luma and chroma samples. When the image is monochrome, the CTU includes a maximum coding block for the monochrome samples and a syntax structure for encoding the monochrome samples. When the image is encoded in a color plane separated according to the color components, the CTU includes a syntax structure for encoding the image and its samples.

[0197] A maximum code block (CTB) can be divided into M×N code blocks (M and N are integers) containing M×N samples.

[0198] When the image has a sample array for the Y, Cr, and Cb components, the encoding unit (CU) includes an encoding block for the luminance samples, two corresponding encoding blocks for the chrominance samples, and a syntax structure for encoding the luminance and chrominance samples. When the image is a monochrome image, the encoding unit includes an encoding block for the monochrome samples and a syntax structure for encoding the monochrome samples. When the image is encoded in a color plane separated according to the color components, the encoding unit includes a syntax structure for encoding the image and its samples.

[0199] As described above, the maximum coding block and the maximum coding unit are conceptually distinct from each other, and the coding block and the coding unit are also conceptually distinct from each other. That is, the (maximum) coding unit refers to a data structure that includes the (maximum) coding block containing the corresponding samples and a syntax structure corresponding to the (maximum) coding block. However, since those skilled in the art understand that the (maximum) coding unit or the (maximum) coding block refers to a block of a predetermined size that includes a predetermined number of samples, unless otherwise described, the maximum coding block and the maximum coding unit or the coding block and the coding unit will be referred to in the following description without distinguishing between them.

[0200] An image can be divided into maximum coding units (MCUs). The size of each MCU can be determined based on information obtained from the bitstream. Each MCU can be a square of the same size. However, the embodiments are not limited to this.

[0201] For example, information about the maximum size of a luminance-coded block can be obtained from the bitstream. For example, the maximum size of a luminance-coded block, indicated by the information about the maximum size of the luminance-coded block, can be one of 16×16, 32×32, 64×64, 128×128, and 256×256.

[0202] For example, information about the luma block size difference and the maximum size of the luma coding block that can be divided into two can be obtained from the bitstream. The information about the luma block size difference can refer to the size difference between the maximum luma coding unit and the maximum luma coding block that can be divided into two. Therefore, by combining the information about the maximum size of the luma coding block that can be divided into two and the information about the luma block size difference obtained from the bitstream, the size of the maximum luma coding unit can be determined. The size of the maximum chroma coding unit can be determined using the size of the maximum luma coding unit. For example, when the Y:Cb:Cr ratio is 4:2:0 according to the color format, the size of the chroma block can be half the size of the luma block, and the size of the maximum chroma coding unit can be half the size of the maximum luma coding unit.

[0203] According to the embodiment, since information about the maximum size of a luminance coding block that can be divided into two parts is obtained from the bitstream, the maximum size of the luminance coding block that can be divided into two parts can be variably determined. Conversely, the maximum size of a luminance coding block that can be divided into three parts can be fixed. For example, the maximum size of a luminance coding block that can be divided into three parts in an I-strip can be 32×32, and the maximum size of a luminance coding block that can be divided into three parts in a P-strip or B-strip can be 64×64.

[0204] Furthermore, the largest coding unit can be hierarchically divided into coding units based on the partition shape pattern information obtained from the bit stream. At least one of the following can be obtained from the bit stream as partition shape pattern information: information indicating whether a quad partition is performed, information indicating whether a multi-partition is performed, partition direction information, and partition type information.

[0205] For example, information indicating whether a quad partitioning is performed can indicate whether the current coding unit has been quad partitioned (QUAD_SPLIT).

[0206] When the current coding unit is not quad-partitioned, the information indicating whether multi-partitioning is performed can indicate whether the current coding unit is no longer partitioned (NO_SPLIT) or is partitioned into two or three parts.

[0207] When the current coding unit is divided into two or three parts, the division direction information indicates that the current coding unit is divided in one of the horizontal or vertical directions.

[0208] When the current coding unit is divided in the horizontal or vertical direction, the division type information indicates whether the current coding unit is divided into two or three parts.

[0209] The partitioning mode of the current coding unit can be determined based on the partitioning direction and partitioning type information. When the current coding unit is partitioned into two parts horizontally, the partitioning mode can be determined as a two-horizontal partitioning mode (SPLIT_BT_HOR); when the current coding unit is partitioned into three parts horizontally, the partitioning mode can be determined as a three-horizontal partitioning mode (SPLIT_TT_HOR); when the current coding unit is partitioned into two parts vertically, the partitioning mode can be determined as a two-vertical partitioning mode (SPLIT_BT_VER); and when the current coding unit is partitioned into three parts vertically, the partitioning mode can be determined as a three-vertical partitioning mode (SPLIT_TT_VER).

[0210] Image decoding device 100 can obtain partitioning shape pattern information from a binary bit string in a bit stream. The bit stream received by image decoding device 100 may be in the form of fixed-length binary code, unary code, truncated unary code, predetermined binary code, etc. The binary bit string is information in binary numbers. The binary bit string may include at least one bit. Image decoding device 100 can obtain partitioning shape pattern information corresponding to the binary bit string based on partitioning rules. Image decoding device 100 can determine, based on a binary bit string, whether to perform four partitions on the coding unit, whether not to partition the coding unit, the partitioning direction, and the partitioning type.

[0211] A coding unit can be smaller than or equal to the largest coding unit. For example, since the largest coding unit is the coding unit with the largest size, it is one of the coding units. When the partitioning shape pattern information regarding the largest coding unit indicates that partitioning is not performed, the coding units determined within the largest coding unit have the same size as the largest coding unit. When the partitioning shape pattern information regarding the largest coding unit indicates that partitioning is performed, the largest coding unit can be partitioned into coding units. Furthermore, when the partitioning shape pattern information regarding coding units indicates that partitioning is performed, a coding unit can be partitioned into smaller coding units. However, the partitioning of an image is not limited to this, and the distinction between the largest coding unit and coding units may not be made. (Refer to...) Figures 7 to 20 Describe in detail the division of coding units.

[0212] Furthermore, one or more prediction blocks for prediction can be determined from the coding unit. The prediction blocks may be the same as or smaller than the coding unit. Additionally, one or more transform blocks for transform can be determined from the coding unit. The transform blocks may be the same as or smaller than the coding unit.

[0213] The shapes and sizes of the transform blocks and the prediction blocks can be independent of each other.

[0214] In another embodiment, prediction can be performed by using the coding unit as a prediction unit. Furthermore, transformation can be performed by using the coding unit as a transform block.

[0215] Reference Figures 7 to 20 The division of coding units is described in detail. The current block and neighboring blocks in this disclosure can refer to one of the maximum coding unit, coding unit, prediction block, and transform block. Furthermore, the current block of the current coding unit is the block currently being decoded or encoded, or the block currently being divided. Neighboring blocks can be blocks reconstructed before the current block. Neighboring blocks can be spatially or temporally adjacent to the current block. Neighboring blocks can be located at one of the following positions: lower left, left side, upper left, above, upper right, right side, or lower right of the current block.

[0216] Figure 7 The illustration shows a process performed by an image decoding device 100 according to an embodiment to determine at least one coding unit by dividing the current coding unit.

[0217] The block shape may include 4N×4N, 4N×2N, 2N×4N, 4N×N, N×4N, 32N×N, N×32N, 16N×N, N×16N, 8N×N, or N×8N. Here, N can be a positive integer. Block shape information is information indicating at least one of the shape, orientation, aspect ratio, or size of the coding unit.

[0218] The shape of the encoding unit can be square or non-square. When the width and height of the encoding unit are the same (i.e., when the block shape of the encoding unit is 4N×4N), the image decoding device 100 can determine the block shape information of the encoding unit as square. The image decoding device 100 can also determine the shape of the encoding unit as non-square.

[0219] When the width and height of the coding unit are different from each other (i.e., when the block shape of the coding unit is 4N×2N, 2N×4N, 4N×N, N×4N, 32N×N, N×32N, 16N×N, N×16N, 8N×N, or N×8N), the image decoding device 100 can determine the block shape information of the coding unit as a non-square shape. When the shape of the coding unit is non-square, the image decoding device 100 can determine the aspect ratio in the block shape information of the coding unit as at least one of 1:2, 2:1, 1:4, 4:1, 1:8, 8:1, 1:16, 16:1, 1:32, and 32:1. Furthermore, the image decoding device 100 can determine whether the coding unit is horizontal or vertical based on its width and height. Additionally, the image decoding device 100 can determine the size of the coding unit based on at least one of its width, height, or area.

[0220] According to an embodiment, the image decoding device 100 can determine the shape of the coding unit by using block shape information, and can determine the coding unit partitioning method by using partition shape pattern information. That is, the coding unit partitioning method indicated by the partition shape pattern information can be determined based on the block shape indicated by the block shape information used by the image decoding device 100.

[0221] Image decoding device 100 can obtain partition shape pattern information from the bitstream. However, the embodiments are not limited to this, and image decoding device 100 and image encoding device 400 can determine pre-agreed partition shape pattern information based on block shape information. Image decoding device 100 can determine pre-agreed partition shape pattern information for the largest or smallest coding unit. For example, image decoding device 100 can determine the partition shape pattern information for the largest coding unit as a four-part partition. Furthermore, image decoding device 100 can determine the partition shape pattern information for the smallest coding unit as "no partitioning". In particular, image decoding device 100 can determine the size of the largest coding unit as 256×256. Image decoding device 100 can determine the pre-agreed partition shape pattern information as a four-part partition. A four-part partition is a partition shape pattern in which the width and height of the coding unit are both bisected. Image decoding device 100 can obtain a 128×128 coding unit from the 256×256-sized largest coding unit based on the partition shape pattern information. Furthermore, image decoding device 100 can determine the size of the smallest coding unit as 4×4. Image decoding device 100 can obtain partitioning shape pattern information indicating that "no partitioning is performed" for the smallest coding unit.

[0222] According to an embodiment, the image decoding device 100 can use block shape information indicating that the current coding unit has a square shape. For example, the image decoding device 100 can determine whether not to divide the coding unit into square units, whether to divide the coding unit vertically, whether to divide the coding unit horizontally, or whether to divide the square coding unit into four coding units based on the division shape pattern information. (See also...) Figure 7 When the block shape information of the current coding unit 700 indicates a square shape, the decoder 115 may determine that the coding unit 710a with the same size as the current coding unit 700 will not be divided based on the division shape pattern information indicating that division will not be performed, or may determine the coding units 710b, 710c, 710d, 710e or 710f divided based on the division shape pattern information indicating a predetermined division method.

[0223] Reference Figure 7According to an embodiment, the image decoding device 100 can determine two coding units 710b obtained by dividing the current coding unit 700 in the vertical direction based on division shape pattern information indicating division in the vertical direction. The image decoding device 100 can determine two coding units 710c obtained by dividing the current coding unit 700 in the horizontal direction based on division shape pattern information indicating division in the horizontal direction. The image decoding device 100 can determine four coding units 710d obtained by dividing the current coding unit 700 in both the vertical and horizontal directions based on division shape pattern information indicating division in both the vertical and horizontal directions. According to an embodiment, the image decoding device 100 can determine three coding units 710e obtained by dividing the current coding unit 700 in the vertical direction based on division shape pattern information indicating three divisions in the vertical direction. The image decoding device 100 can determine three coding units 710f obtained by dividing the current coding unit 700 in the horizontal direction based on division shape pattern information indicating three divisions in the horizontal direction. However, the method for dividing square coding units is not limited to the above methods, and the division shape pattern information can indicate various methods. The predetermined division method for dividing square coding units will be described in detail below with respect to various embodiments.

[0224] Figure 8 The illustration shows a process performed by an image decoding device 100 according to an embodiment to determine at least one coding unit by dividing non-square coding units.

[0225] According to an embodiment, the image decoding device 100 can use block shape information indicating that the current coding unit has a non-square shape. The image decoding device 100 can determine, based on the partitioning shape pattern information, whether not to partition the non-square current coding unit or whether to partition the non-square current coding unit using a predetermined partitioning method. (Refer to...) Figure 8 When the block shape information of the current encoding unit 800 or 850 indicates a non-square shape, the image decoding device 100 can determine, based on the partitioning shape pattern information indicating that partitioning is not performed, that encoding units 810 or 860 with the same size as the current encoding unit 800 or 850 will not be partitioned, or determine the encoding units 820a and 820b, 830a to 830c, 870a and 870b, or 880a to 880c partitioned based on the partitioning shape pattern information indicating a predetermined partitioning method. The predetermined partitioning method for partitioning non-square encoding units will be described in detail below with respect to various embodiments.

[0226] According to an embodiment, the image decoding device 100 can determine the partitioning method of the coding units by using partitioning shape pattern information, and in this case, the partitioning shape pattern information can indicate the number of one or more coding units generated by partitioning the coding units. (See also...) Figure 8 When the partitioning shape pattern information indicates that the current coding unit 800 or 850 is divided into two coding units, the image decoding device 100 can determine the two coding units 820a and 820b or 870a and 870b included in the current coding unit 800 or 850 by partitioning the current coding unit 800 or 850 based on the partitioning shape pattern information.

[0227] According to an embodiment, when the image decoding device 100 divides a non-square current coding unit 800 or 850 based on the division shape pattern information, the image decoding device 100 may divide the current coding unit by considering the position of the long side of the non-square current coding unit 800 or 850. For example, the image decoding device 100 may consider the shape of the current coding unit 800 or 850 and determine multiple coding units by dividing the long side of the current coding unit 800 or 850.

[0228] According to an embodiment, when the partitioning shape pattern information indicates that the coding unit is divided (tripartitely divided) into an odd number of blocks, the image decoding device 100 can determine the odd number of coding units included in the current coding unit 800 or 850. For example, when the partitioning shape pattern information indicates that the current coding unit 800 or 850 is divided into three coding units, the image decoding device 100 can divide the current coding unit 800 or 850 into three coding units 830a, 830b and 830c or 880a, 880b and 880c.

[0229] According to an embodiment, the aspect ratio of the current coding unit 800 or 850 can be 4:1 or 1:4. When the aspect ratio is 4:1, because the width is longer than the height, the block shape information can be horizontal. When the aspect ratio is 1:4, because the width is shorter than the height, the block shape information can be vertical. The image decoding device 100 can determine to divide the current coding unit into an odd number of blocks based on the division shape pattern information. Furthermore, the image decoding device 100 can determine the division direction of the current coding unit 800 or 850 based on the block shape information of the current coding unit 800 or 850. For example, when the current coding unit 800 is vertical, the image decoding device 100 can determine coding units 830a to 830c by dividing the current coding unit 800 horizontally. Similarly, when the current coding unit 850 is horizontal, the image decoding device 100 can determine coding units 880a to 880c by dividing the current coding unit 850 vertically.

[0230] According to an embodiment, the image decoding device 100 may determine an odd number of coding units included in the current coding unit 800 or 850, and not all determined coding units may have the same size. For example, the size of a predetermined coding unit 830b or 880b in the determined odd number of coding units 830a, 830b, and 830c, or 880a, 880b, and 880c, may be different from the sizes of other coding units 830a and 830c, or 880a and 880c. That is, the coding units determined by dividing the current coding unit 800 or 850 may have multiple sizes, and in some cases, all the odd number of coding units 830a, 830b, and 830c, or 880a, 880b, and 880c, may have different sizes.

[0231] According to an embodiment, when the partitioning shape pattern information indicates that the coding unit should be divided into an odd number of blocks, the image decoding device 100 can determine the odd number of coding units included in the current coding unit 800 or 850, and furthermore, a predetermined restriction can be imposed on at least one of the odd number of coding units generated by partitioning the current coding unit 800 or 850. (Refer to...) Figure 8 The image decoding device 100 may configure the decoding process for the central coding unit 830b or 880b among the three coding units 830a, 830b, and 830c or 880a, 880b, and 880c generated when dividing the current coding unit 800 or 850 to be different from the decoding process for the other coding units 830a and 830c or 880a or 880c. For example, unlike the other coding units 830a and 830c or 880a and 880c, the image decoding device 100 may restrict the coding unit 830b or 880b at the central position to either not be divided or to be divided only a predetermined number of times.

[0232] Figure 9 The illustration shows a process performed by an image decoding device 100 according to an embodiment, based on at least one partitioning encoding unit of block shape information and partitioning shape pattern information.

[0233] According to an embodiment, the image decoding device 100 can determine whether to divide the first coding unit 900 of the square into coding units or not to divide the first coding unit 900 of the square, based on at least one of block shape information and division shape pattern information. According to an embodiment, when the division shape pattern information indicates that the first coding unit 900 is divided in the horizontal direction, the image decoding device 100 can determine the second coding unit 910 by dividing the first coding unit 900 in the horizontal direction. The terms first coding unit, second coding unit, and third coding unit used in the embodiment are terms used to understand the relationship before and after the division of coding units. For example, the second coding unit can be determined by dividing the first coding unit, and the third coding unit can be determined by dividing the second coding unit. It will be understood that the structure of the first coding unit, second coding unit, and third coding unit follows the above description.

[0234] According to an embodiment, the image decoding device 100 may determine, based on the division shape pattern information, whether to divide the determined second coding unit 910 into coding units or not to divide the determined second coding unit 410. (Refer to...) Figure 9 The image decoding device 100 can divide a non-square second coding unit 910, determined by dividing the first coding unit 900, into one or more third coding units 920a, or 920b, 920c, and 920d, based on the division shape pattern information, or it can choose not to divide the non-square second coding unit 910. The image decoding device 100 can obtain the division shape pattern information and can obtain multiple second coding units (e.g., 910) of various shapes by dividing the first coding unit 900 based on the obtained division shape pattern information. Furthermore, based on the division shape pattern information, the second coding unit 910 can be divided using the division method of the first coding unit 900. According to an embodiment, when the first coding unit 900 is divided into second coding units 910 based on the division shape pattern information of the first coding unit 900, the second coding unit 910 can also be divided into third coding units 920a, or 920b, 920c, and 920d based on the division shape pattern information of the second coding unit 910. That is, the coding units can be recursively divided based on the division shape pattern information of each coding unit. Therefore, square coding units can be determined by dividing non-square coding units, and non-square coding units can be determined by recursively dividing square coding units.

[0235] Reference Figure 9The predetermined coding units (e.g., coding units located at the center or square coding units) in an odd number of third coding units 920b, 920c, and 920d, determined by dividing the non-square second coding unit 910, can be recursively divided. According to an embodiment, the square third coding unit 920c in the odd number of third coding units 920b, 920c, and 920d can be divided horizontally into a plurality of fourth coding units. The non-square fourth coding units 930b or 930d in the plurality of fourth coding units 930a, 930b, 930c, and 930d can again be divided into a plurality of coding units. For example, the non-square fourth coding units 930b or 930d can again be divided into an odd number of coding units. Methods for recursively dividing coding units will be described below with respect to various embodiments.

[0236] According to an embodiment, the image decoding device 100 can divide each of the third coding units 920a, or 920b, 920c, and 920d, into coding units based on the division shape pattern information. Furthermore, the image decoding device 100 can determine, based on the division shape pattern information, not to divide the second coding unit 910. According to an embodiment, the image decoding device 100 can divide the non-square second coding unit 910 into an odd number of third coding units 920b, 920c, and 920d. The image decoding device 100 can impose predetermined restrictions on predetermined third coding units among the odd number of third coding units 920b, 920c, and 920d. For example, the image decoding device 100 can restrict the third coding unit 920c at the center position among the odd number of third coding units 920b, 920c, and 920d to no longer be divided or to be divided a set number of times.

[0237] Reference Figure 9 The image decoding device 100 may restrict the third coding unit 920c at the center position of the odd number of third coding units 920b, 920c, and 920d included in the non-square second coding unit 910 to no longer be divided, to be divided using a predetermined division method (e.g., divided into only four coding units or divided using the division method of the second coding unit 910), or to be divided only a predetermined number of times (e.g., divided only n times (where n>0)). However, the restriction on the third coding unit 920c at the center position is not limited to the above examples and may include various restrictions for decoding the third coding unit 920c at the center position differently from the other third coding units 920b and 920d.

[0238] According to an embodiment, the image decoding device 100 can obtain partitioning shape pattern information used to partition the current coding unit from a predetermined position in the current coding unit.

[0239] Figure 10 This illustrates a method performed by an image decoding device 100 according to an embodiment to determine a predetermined coding unit from an odd number of coding units.

[0240] Reference Figure 10 The segmentation shape pattern information of the current encoding unit 1000 can be obtained from a sample at a predetermined position (e.g., sample 1040 at the center position) among a plurality of sample points included in the current encoding unit 1000. However, the predetermined position in the current encoding unit 1000 from which at least one segmentation shape pattern information can be obtained is not limited to... Figure 10 The predetermined position can be the center position within the current encoding unit 1000, and may include various positions (e.g., above, below, left, right, upper left, lower left, upper right, and lower right positions). The image decoding device 100 can obtain division shape pattern information from the predetermined position and can determine whether to divide the current encoding unit into encoding units of various shapes and sizes.

[0241] According to an embodiment, when the current coding unit is divided into a predetermined number of coding units, the image decoding device 100 may select one coding unit from the coding units. As will be described below with respect to various embodiments, various methods may be used to select one of a plurality of coding units.

[0242] According to an embodiment, the image decoding device 100 can divide the current encoding unit into multiple encoding units and determine the encoding unit at a predetermined position.

[0243] According to an embodiment, the image decoding device 100 can use information indicating the positions of an odd number of coding units to determine the coding unit at the center position among the odd number of coding units. (See also...) Figure 10 Image decoding device 100 can determine an odd number of coding units 1020a, 1020b, and 1020c by dividing the current coding unit 1000. Image decoding device 150 can determine the intermediate coding unit 1020b by using information about the positions of the odd number of coding units 1020a, 1020b, and 1020c. For example, image decoding device 100 can determine the coding unit 1020b at the center position by determining the positions of coding units 1020a, 1020b, and 1020c based on information indicating the positions of predetermined samples included in coding units 1020a, 1020b, and 1020c. In detail, the image decoding device 100 can determine the position of the encoding units 1020a, 1020b and 1020c based on the information indicating the position of the upper left sample points 1030a, 1030b and 1030c of the encoding units 1020a, 1020b and 1020c, thereby determining the encoding unit 1020b at the center position.

[0244] According to an embodiment, the information indicating the positions of the top-left sample points 1030a, 1030b, and 1030c included in encoding units 1020a, 1020b, and 1020c respectively may include information about the positions or coordinates of the encoding units 1020a, 1020b, and 1020c in the frame. According to an embodiment, the information indicating the positions of the top-left sample points 1030a, 1030b, and 1030c included in encoding units 1020a, 1020b, and 1020c respectively may include information indicating the width or height of the encoding units 1020a, 1020b, and 1020c included in the current encoding unit 1000, and the width or height may correspond to information indicating the difference between the coordinates of the encoding units 1020a, 1020b, and 1020c in the frame. In other words, the image decoding device 100 can determine the encoding unit 1020b at the center position by directly using information about the position or coordinates of the encoding units 1020a, 1020b and 1020c in the image, or by using information about the width or height corresponding to the difference between the encoding units and their coordinates.

[0245] According to an embodiment, the information indicating the position of the top-left sample 1030a of the upper encoding unit 1020a may include coordinates (xa, ya), the information indicating the position of the top-left sample 1030b of the middle encoding unit 1020b may include coordinates (xb, yb), and the information indicating the position of the top-left sample 1030c of the lower encoding unit 1020c may include coordinates (xc, yc). The image decoding device 100 can determine the middle encoding unit 1020b by using the coordinates of the top-left samples 1030a, 1030b, and 1030c included in the encoding units 1020a, 1020b, and 1020c, respectively. For example, when the coordinates of the top left sample points 1030a, 1030b, and 1030c are sorted in ascending or descending order, the coding unit 1020b, which includes the coordinates (xb, yb) of the sample point 1030b at the center position, can be determined as the coding unit at the center position among the coding units 1020a, 1020b, and 1020c determined by dividing the current coding unit 1000. However, the coordinates indicating the positions of the top-left sample points 1030a, 1030b, and 1030c may include coordinates indicating absolute positions within the frame, or coordinates (dxb, dyb) indicating the relative position of the top-left sample point 1030b of the intermediate encoding unit 1020b relative to the top-left sample point 1030a of the upper encoding unit 1020a, and coordinates (dxc, dyc) indicating the relative position of the top-left sample point 1030c of the lower encoding unit 1020c relative to the top-left sample point 1030a of the upper encoding unit 1020a. The method of determining the encoding unit at a predetermined position by using the coordinates of the sample points included in the encoding unit as information indicating the position of the sample points is not limited to the methods described above, and may include various arithmetic methods capable of using the coordinates of the sample points.

[0246] According to an embodiment, the image decoding device 100 can divide the current encoding unit 1000 into a plurality of encoding units 1020a, 1020b, and 1020c, and can select one of the encoding units 1020a, 1020b, and 1020c based on a predetermined criterion. For example, the image decoding device 100 can select encoding unit 1020b, whose size is different from that of other encoding units, from the encoding units 1020a, 1020b, and 1020c.

[0247] According to an embodiment, the image decoding device 100 can determine the width or height of each of the encoding units 1020a, 1020b, and 1020c using coordinates (xa, ya), (xb, yb), and (xc, yc), wherein coordinates (xa, ya) indicate the position of the upper left sample point 1030a of the upper encoding unit 1020a, coordinates (xb, yb) indicate the position of the upper left sample point 1030b of the middle encoding unit 1020b, and coordinates (xc, yc) indicate the position of the upper left sample point 1030c of the lower encoding unit 1020c. The image decoding device 100 can determine the respective dimensions of the encoding units 1020a, 1020b, and 1020c using coordinates (xa, ya), (xb, yb), and (xc, yc) indicating the positions of the encoding units 1020a, 1020b, and 1020c. According to an embodiment, the image decoding device 100 can determine the width of the upper encoding unit 1020a as the width of the current encoding unit 1000. The image decoding device 100 can determine the height of the upper encoding unit 1020a as yb-ya. According to an embodiment, the image decoding device 100 can determine the width of the middle encoding unit 1020b as the width of the current encoding unit 1000. The image decoding device 100 can determine the height of the middle encoding unit 1020b as yc-yb. According to an embodiment, the image decoding device 100 can determine the width or height of the lower encoding unit 1020c by using the width or height of the current encoding unit 1000 or the width or height of the upper encoding unit 1020a and the middle encoding unit 1020b. The image decoding device 100 can determine encoding units with dimensions different from those of other encoding units based on the determined widths and heights of the encoding units 1020a to 1020c. (Refer to...) Figure 10 The image decoding device 100 can determine an intermediate encoding unit 1020b, which has a size different from that of the upper encoding unit 1020a and the lower encoding unit 1020c, as an encoding unit at a predetermined position. However, the method described above by the image decoding device 100 for determining an encoding unit with a size different from that of other encoding units corresponds only to the example of determining an encoding unit at a predetermined position by using the size of the encoding unit determined based on the coordinates of sample points. Therefore, various methods for determining an encoding unit at a predetermined position by comparing the sizes of encoding units determined based on the coordinates of predetermined sample points can be used.

[0248] However, the position of the sample points considered in determining the position of the coding unit is not limited to the upper left position mentioned above, and information about any position of the sample points included in the coding unit can be used.

[0249] According to an embodiment, the image decoding device 100 may consider the shape of the current coding unit and select a coding unit at a predetermined position from an odd number of coding units determined by dividing the current coding unit. For example, when the current coding unit has a non-square shape with a width greater than its height, the image decoding device 100 may determine a coding unit at a predetermined position along the horizontal direction. That is, the image decoding device 100 may determine one coding unit from the coding units at different positions along the horizontal direction and impose restrictions on that coding unit. When the current coding unit has a non-square shape with a height greater than its width, the image decoding device 100 may determine a coding unit at a predetermined position along the vertical direction. That is, the image decoding device 100 may determine one coding unit from the coding units at different positions along the vertical direction and may impose restrictions on that coding unit.

[0250] According to an embodiment, the image decoding device 100 can use information indicating the positions of each of an even number of coding units to determine the coding unit at a predetermined position among the even number of coding units. The image decoding device 100 can determine the even number of coding units by dividing (bi-dividing) the current coding unit, and can determine the coding unit at the predetermined position by using information about the positions of the even number of coding units. The associated operations are similar to those already described above regarding... Figure 10 The operation corresponding to determining a predetermined position (e.g., the center position) among an odd number of coding units is described in detail, and therefore its detailed description is not provided here.

[0251] According to an embodiment, when a non-square current coding unit is divided into multiple coding units, predetermined information about the coding unit at a predetermined position can be used in the division operation to determine the coding unit at the predetermined position among the multiple coding units. For example, the image decoding device 100 can use at least one of block shape information and division shape pattern information stored in the samples included in the intermediate coding unit to determine the coding unit at the center position among the multiple coding units determined by dividing the current coding unit in the division operation.

[0252] Reference Figure 10The image decoding device 100 can divide the current coding unit 500 into multiple coding units 1020a, 1020b, and 1020c based on the division shape pattern information, and can determine the coding unit 1020b at the center position among the multiple coding units 1020a, 1020b, and 1020c. Furthermore, the image decoding device 100 can determine the coding unit 1020b at the center position by considering the location where the division shape pattern information is obtained. That is, the division shape pattern information of the current coding unit 1000 can be obtained from the sample point 1040 at the center position of the current coding unit 1000, and when the current coding unit 1000 is divided into multiple coding units 1020a, 1020b, and 1020c based on the division shape pattern information, the coding unit 1020b including the sample point 1040 can be determined as the coding unit at the center position. However, the information used to determine the coding unit at the center position is not limited to the division shape pattern information, and various types of information can be used to determine the coding unit at the center position.

[0253] According to an embodiment, predetermined information for identifying the coding unit at a predetermined location can be obtained from predetermined samples included in the coding unit to be determined. (Refer to...) Figure 10 The image decoding device 100 can use the partitioning shape pattern information obtained from samples at predetermined positions in the current coding unit 1000 (e.g., samples at the center of the current coding unit 1000) to determine the coding unit at a predetermined position (e.g., the coding unit at the center of the partitioned coding units) among the plurality of coding units 1020a, 1020b, and 1020c determined by partitioning the current coding unit 1000. That is, the image decoding device 100 can determine the samples at the predetermined positions by considering the block shape of the current coding unit 1000, determine the coding unit 1020b including samples from the plurality of coding units 1020a, 1020b, and 1020c determined by partitioning the current coding unit 1000, and can impose predetermined constraints on the coding unit 1020b. (Refer to...) Figure 10 According to an embodiment, during the decoding operation, the image decoding device 100 can determine a sample 1040 at the center position of the current encoding unit 1000 as a sample from which predetermined information can be obtained, and can impose a predetermined restriction on the encoding unit 1020b including the sample 1040. However, the position of the sample from which predetermined information can be obtained is not limited to the above-mentioned position, and can include any position in the encoding unit 1020b that will be determined as a restricted sample.

[0254] According to an embodiment, the location of a sample point from which predetermined information can be obtained can be determined based on the shape of the current encoding unit 1000. According to an embodiment, block shape information can indicate whether the current encoding unit has a square shape or a non-square shape, and the location of the sample point from which predetermined information can be obtained can be determined based on this shape. For example, the image decoding device 100 can determine a sample point located on a boundary used to halve at least one of the width and height of the current encoding unit as a sample point from which predetermined information can be obtained by using at least one of information about the width and information about the height of the current encoding unit. As another example, when the block shape information of the current encoding unit indicates a non-square shape, the image decoding device 100 can determine one of the sample points adjacent to the boundary used to halve the long side of the current encoding unit as a sample point from which predetermined information can be obtained.

[0255] According to an embodiment, when the current coding unit is divided into multiple coding units, the image decoding device 100 can use partition shape pattern information to determine the coding unit at a predetermined position among the multiple coding units. According to an embodiment, the image decoding device 100 can obtain partition shape pattern information from samples at predetermined positions in the coding units, and divide the multiple coding units by using the partition shape pattern information obtained from samples at predetermined positions in each of the multiple coding units, wherein the multiple coding units are generated by dividing the current coding unit. That is, the coding units can be recursively divided based on the partition shape pattern information obtained from samples at predetermined positions in each coding unit. The above already relates to... Figure 9 The operation of recursively dividing the coding unit is described, and therefore its detailed description will not be provided here.

[0256] According to an embodiment, the image decoding device 100 can determine one or more coding units by dividing the current coding unit, and can determine the order in which the one or more coding units are decoded based on a predetermined block (e.g., the current coding unit).

[0257] Figure 11 This illustrates the order in which the image decoding device 100 processes the plurality of coding units when it determines the plurality of coding units by dividing the current coding unit, according to an embodiment.

[0258] According to an embodiment, the image decoding device 100 may determine second coding units 1110a and 1110b by dividing the first coding unit 1100 in the vertical direction, and determine second coding units 1130a and 1130b by dividing the first coding unit 1100 in the horizontal direction, or determine second coding units 1150a to 1150d by dividing the first coding unit 1100 in both the vertical and horizontal directions, based on the division shape pattern information.

[0259] Reference Figure 11 The image decoding device 100 can determine to process the second coding units 1110a and 1110b determined by dividing the first coding unit 1100 in the vertical direction in a horizontal direction sequence 1110c. The image decoding device 100 can determine to process the second coding units 1130a and 1130b determined by dividing the first coding unit 1100 in the horizontal direction in a vertical direction sequence 1130c. The image decoding device 100 can determine to process the second coding units 1150a to 1150d determined by dividing the first coding unit 1100 in the vertical and horizontal directions according to a predetermined order (e.g., raster scan order or zigzag scan order 1150e), wherein the coding units in one row are processed in the predetermined order, and then the coding units in the next row are processed.

[0260] According to an embodiment, the image decoding device 100 can recursively divide encoding units. (See also...) Figure 11 The image decoding device 100 can determine a plurality of coding units 1110a and 1110b, 1130a and 1130b, or 1150a to 1150d by dividing a first coding unit 1100, and recursively divide each of the determined plurality of coding units 1110a and 1110b, 1130a and 1130b, or 1150a to 1150d. The method of dividing the plurality of coding units 1110a and 1110b, 1130a and 1130b, or 1150a to 1150d can correspond to the method of dividing the first coding unit 1100. Thus, each of the plurality of coding units 1110a and 1110b, 1130a and 1130b, or 1150a to 1150d can be independently divided into a plurality of coding units. (Refer to...) Figure 11 The image decoding device 100 can determine the second coding units 1110a and 1110b by dividing the first coding unit 1100 in the vertical direction, and can determine whether to divide or not divide each of the second coding units 1110a and 1110b independently.

[0261] According to an embodiment, the image decoding device 100 can determine the third coding units 1120a and 1120b by dividing the left second coding unit 1110a in the horizontal direction, and may not divide the right second coding unit 1110b.

[0262] According to an embodiment, the processing order of coding units can be determined based on the operation of dividing coding units. In other words, the processing order of the divided coding units can be determined based on the processing order of the coding units immediately preceding the division. The image decoding device 100 can determine the processing order of the third coding units 1120a and 1120b determined by dividing the left second coding unit 1110a independently of the right second coding unit 1110b. Because the third coding units 1120a and 1120b are determined by dividing the left second coding unit 1110a in the horizontal direction, the third coding units 1120a and 1120b can be processed in the vertical order 1120c. Because the left second coding unit 1110a and the right second coding unit 1110b are processed in the horizontal order 1110c, the right second coding unit 1110b can be processed after the third coding units 1120a and 1120b included in the left second coding unit 1110a are processed in the vertical order 1120c. The operation of determining the processing order of coding units based on the coding units before division is not limited to the above example, and various methods can be used to process the divided coding units and the coding units determined to be of various shapes independently in a predetermined order.

[0263] Figure 12 The illustration shows a process performed by an image decoding device 100 according to an embodiment, in which the current coding unit is determined to be divided into an odd number of coding units when the coding units cannot be processed in a predetermined order.

[0264] According to an embodiment, the image decoding device 100 can determine whether the current coding unit is divided into an odd number of coding units based on the obtained partition shape pattern information. (See also...) Figure 12 The square first coding unit 1200 can be divided into non-square second coding units 1210a and 1210b, and the second coding units 1210a and 1210b can be independently divided into third coding units 1220a and 1220b, and 1220c to 1220e. According to an embodiment, the image decoding device 100 can determine a plurality of third coding units 1220a and 1220b by dividing the left second coding unit 1210a in the horizontal direction, and can divide the right second coding unit 1210b into an odd number of third coding units 1220c to 1220e.

[0265] According to an embodiment, the image decoding device 100 can determine whether any coding unit is divided into an odd number of coding units by determining whether the third coding units 1220a and 1220b and 1220c to 1220e can be processed in a predetermined order. (Refer to...) Figure 12The image decoding device 100 can determine the third coding units 1220a and 1220b, and 1220c to 1220e by recursively dividing the first coding unit 1200. The image decoding device 100 can determine whether any one of the following coding units is divided into an odd number of coding units based on at least one of block shape information and division shape pattern information: the first coding unit 1200, the second coding units 1210a and 1210b, and the third coding units 1220a and 1220b, and 1220c to 1220e. For example, the rightmost second coding unit 1210b of the second coding units 1220a and 1220b can be divided into an odd number of third coding units 1220c, 1220d, and 1220e. The processing order of the plurality of coding units included in the first coding unit 1200 can be a predetermined order (e.g., zigzag scanning order 1230). The image decoding device 100 can determine whether the third coding units 1220c, 1220d, and 1220e, which are determined by dividing the right second coding unit 1210b into an odd number of coding units, meet the conditions for processing in a predetermined order.

[0266] According to an embodiment, the image decoding device 100 can determine whether the third encoding units 1220a and 1220b, and 1220c to 1220e included in the first encoding unit 1200 satisfy a condition for processing in a predetermined order, and this condition relates to whether at least one of the width and height of the second encoding units 1210a and 1210b will be divided in half along the boundary of the third encoding units 1220a and 1220b, and 1220c to 1220e. For example, the third encoding units 1220a and 1220b determined when the height of the non-square-shaped left second encoding unit 1210a is divided in half can satisfy the condition. Because the boundary of the third encoding units 1220c to 1220e determined when the right second encoding unit 1210b is divided into three encoding units fails to divide the width or height of the right second encoding unit 1210b in half, it can be determined that the third encoding units 1220c to 1220e do not satisfy the condition. When the conditions described above are not met, the image decoding device 100 can determine that the scanning order is discontinuous, and based on the determination result, can determine that the right-side second coding unit 1210b will be divided into an odd number of coding units. According to an embodiment, when the coding unit is divided into an odd number of coding units, the image decoding device 100 can impose a predetermined restriction on the coding units at predetermined positions within the divided coding units. The restriction or the predetermined position has already been described above with respect to various embodiments, and therefore its detailed description will not be provided here.

[0267] Figure 13The illustration shows a process performed by an image decoding device 100 according to an embodiment, which involves dividing a first coding unit 1300 to determine at least one coding unit.

[0268] According to an embodiment, the image decoding device 100 can divide the first coding unit 1300 based on the division shape pattern information obtained by a receiver (not shown). The square first coding unit 1300 can be divided into four square coding units, or it can be divided into multiple non-square coding units. For example, referring to… Figure 13 When the partitioning shape pattern information indicates that the first coding unit 1300 should be partitioned into non-square coding units, the image decoding device 100 may partition the first coding unit 1300 into a plurality of non-square coding units. Specifically, when the partitioning shape pattern information indicates that an odd number of coding units should be determined by partitioning the first coding unit 1300 in the horizontal or vertical direction, the image decoding device 100 may partition the square first coding unit 1300 into an odd number of coding units (e.g., second coding units 1310a, 1310b, and 1310c determined by partitioning the square first coding unit 1300 in the vertical direction, or second coding units 1320a, 1320b, and 1320c determined by partitioning the square first coding unit 1300 in the horizontal direction).

[0269] According to an embodiment, the image decoding device 100 can determine whether the second encoding units 1310a, 1310b, 1310c, 1320a, 1320b, and 1320c included in the first encoding unit 1300 satisfy a condition for processing in a predetermined order, and this condition is related to whether at least one of the width and height of the first encoding unit 1300 is divided in half along the boundary of the second encoding units 1310a, 1310b, 1310c, 1320a, 1320b, and 1320c. (Refer to...) Figure 13Because the boundaries of the second coding units 1310a, 1310b, and 1310c, defined by the first coding unit 1300 dividing the square in the vertical direction, do not halve the width of the first coding unit 1300, it can be determined that the first coding unit 1300 does not meet the conditions for processing in a predetermined order. Furthermore, because the boundaries of the second coding units 1320a, 1320b, and 1320c, defined by the first coding unit 1300 dividing the square in the horizontal direction, do not halve the height of the first coding unit 1300, it can be determined that the first coding unit 1300 does not meet the conditions for processing in a predetermined order. When the conditions are not met as described above, the image decoding device 100 can determine that the scanning order is discontinuous, and can determine, based on the determination result, that the first coding unit 1300 will be divided into an odd number of coding units. According to an embodiment, when the coding unit is divided into an odd number of coding units, the image decoding device 100 can impose a predetermined restriction on the coding units at predetermined positions within the divided coding units. The limitations or predetermined positions have been described above with respect to various embodiments, and therefore will not be described in detail here.

[0270] According to an embodiment, the image decoding device 100 can determine coding units of various shapes by dividing a first coding unit.

[0271] Reference Figure 13 The image decoding device 100 can divide the square first coding unit 1300 or the non-square first coding unit 1330 or 1350 into coding units of various shapes.

[0272] Figure 14 The image decoding device 100, when dividing the first encoding unit 1400 according to the embodiment, determines that the second encoding unit, which has a non-square shape, satisfies a predetermined condition, and the shape into which the second encoding unit can be divided is limited.

[0273] According to an embodiment, the image decoding device 100 can determine, based on the division shape pattern information obtained by a receiver (not shown), to divide a square first coding unit 1400 into non-square second coding units 1410a and 1410b or 1420a and 1420b. The second coding units 1410a and 1410b or 1420a and 1420b can be divided independently. Thus, the image decoding device 100 can determine, based on the division shape pattern information of each of the second coding units 1410a and 1410b or 1420a and 1420b, to divide each of the second coding units 1410a and 1410b or 1420a and 1420b into multiple coding units or not to divide each of the second coding units 1410a and 1410b or 1420a and 1420b. According to an embodiment, the image decoding device 100 can determine third coding units 1412a and 1412b by dividing the non-square left second coding unit 1410a, determined by dividing the first coding unit 1400 in the vertical direction, in the horizontal direction. However, when the left second coding unit 1410a is divided in the horizontal direction, the image decoding device 100 can restrict the right second coding unit 1410b to not be divided in the horizontal direction in which the left second coding unit 1410a is divided. When the third coding units 1414a and 1414b are determined by dividing the right second coding unit 1410b in the same direction, since the left second coding unit 1410a and the right second second coding unit 1410b are divided independently in the horizontal direction, the third coding units 1412a and 1412b or 1414a and 1414b can be determined. However, this situation works in the same way as when the image decoding device 100 divides the first encoding unit 1400 into four squares, namely the second encoding units 1430a, 1430b, 1430c and 1430d, based on the division shape pattern information, and may be inefficient in terms of image decoding.

[0274] According to an embodiment, the image decoding device 100 can determine the third coding units 1422a and 1422b or 1424a and 1424b by dividing the non-square second coding units 1420a or 1420b, which are determined by dividing the first coding unit 1400 in the horizontal direction, in the vertical direction. However, when the second coding unit (e.g., the upper second coding unit 1420a) is divided in the vertical direction, for the reasons described above, the image decoding device 100 can restrict another second coding unit (e.g., the lower second coding unit 1420b) to not be divided in the vertical direction in which the upper second coding unit 1420a is divided.

[0275] Figure 15The illustration shows the process performed by the image decoding device 100 according to an embodiment of dividing a square coding unit when the dividing shape pattern information fails to indicate that the square coding unit is divided into four square coding units.

[0276] According to an embodiment, the image decoding device 100 can determine second coding units 1510a and 1510b, or 1520a and 1520b, etc., by dividing the first coding unit 1500 based on division shape pattern information. The division shape pattern information may include information about various methods of dividing the coding units, but may not include information for dividing the coding unit into four square coding units. Based on such division shape pattern information, the image decoding device 100 may not divide the square first coding unit 1500 into four square second coding units 1530a, 1530b, 1530c, and 1530d. The image decoding device 100 can determine non-square second coding units 1510a and 1510b, or 1520a and 1520b, etc., based on the division shape pattern information.

[0277] According to an embodiment, the image decoding device 100 can independently divide non-square second coding units 1510a and 1510b or 1520a and 1520b, etc. Each of the second coding units 1510a and 1510b or 1520a and 1520b, etc., can be recursively divided in a predetermined order, and this division method can correspond to the method of dividing the first coding unit 1500 based on the division shape pattern information.

[0278] For example, the image decoding device 100 can determine the third coding units 1512a and 1512b of the square by dividing the left second coding unit 1510a in the horizontal direction, and can determine the third coding units 1514a and 1514b of the square by dividing the right second coding unit 1510b in the horizontal direction. Furthermore, the image decoding device 100 can determine the third coding units 1516a, 1516b, 1516c, and 1516d of the square by dividing both the left second coding unit 1510a and the right second coding unit 1510b in the horizontal direction. In this case, coding units with the same shape as the second coding units 1530a, 1530b, 1530c, and 1530d of the four squares divided from the first coding unit 1500 can be determined.

[0279] As another example, the image decoding device 100 can determine the third coding units 1522a and 1522b of the square by dividing the upper second coding unit 1520a in the vertical direction, and can determine the third coding units 1524a and 1524b of the square by dividing the lower second coding unit 1520b in the vertical direction. Furthermore, the image decoding device 100 can determine the third coding units 1526a, 1526b, 1526c, and 1526d of the square by dividing both the upper second coding unit 1020a and the lower second coding unit 1520b in the vertical direction. In this case, coding units with the same shape as the second coding units 1530a, 1530b, 1530c, and 1530d of the four squares divided from the first coding unit 1500 can be determined.

[0280] Figure 16 This illustrates that, according to an embodiment, the processing order among multiple coding units can be changed based on the process of dividing coding units.

[0281] According to an embodiment, the image decoding device 100 may divide the first coding unit 1600 based on division shape pattern information. When the block shape indicates a square shape and the division shape information indicates that the first coding unit 1600 is divided in at least one of the horizontal and vertical directions, the image decoding device 100 may determine the second coding units 1610a and 1610b or 1620a and 1620b by dividing the first coding unit 1600. (See also...) Figure 16 The non-square second coding units 1610a and 1610b, or 1620a and 1620b, determined by dividing the first coding unit 1600 only in the horizontal or vertical direction, can be independently divided based on the division shape pattern information of each coding unit. For example, the image decoding device 100 can determine third coding units 1616a, 1616b, 1616c, and 1616d by dividing the second coding units 1610a and 1610b generated by dividing the first coding unit 1600 in the vertical direction in the horizontal direction, and can determine third coding units 1626a, 1626b, 1626c, and 1626d by dividing the second coding units 1620a and 1620b generated by dividing the first coding unit 1600 in the horizontal direction in the vertical direction. The above has already discussed... Figure 15 The operation of dividing the second coding unit 1610a and 1610b or 1620a and 1620b is described, so its detailed description will not be provided here.

[0282] According to an embodiment, the image decoding device 100 can process the encoding units in a predetermined order. The above has already mentioned... Figure 16The operation of processing encoded units in a predetermined order is described, therefore its detailed description will not be provided here. (See reference...) Figure 16 The image decoding device 100 can determine the third encoding units 1616a, 1616b, 1616c, and 1616d, as well as 1626a, 1626b, 1626c, and 1626d, of the four squares by dividing the first encoding unit 1600 into squares. According to an embodiment, the image decoding device 100 can determine the processing order of the third encoding units 1616a, 1616b, 1616c, and 1616d, as well as 1626a, 1626b, 1626c, and 1626d, based on the division method of the first encoding unit 1600.

[0283] According to an embodiment, the image decoding device 100 can determine third coding units 1616a, 1616b, 1616c, and 1616d by dividing the second coding units 1610a and 1610b generated by dividing the first coding unit 1600 in the vertical direction in the horizontal direction, and can process the third coding units 1616a, 1616b, 1616c, and 1616d in the following processing order 1617: first, the third coding units 1616a and 1616c included in the left second coding unit 1610a are processed in the vertical direction, and then the third coding units 1616b and 1616d included in the right second coding unit 1610b are processed in the vertical direction.

[0284] According to an embodiment, the image decoding device 100 can determine third coding units 1626a, 1626b, 1626c, and 1626d by dividing the second coding units 1620a and 1620b generated by dividing the first coding unit 1600 in the horizontal direction in the vertical direction, and can process the third coding units 1626a, 1626b, 1626c, and 1626d in the following processing order 1627: firstly, the third coding units 1626a and 1626b included in the upper second coding unit 1620a are processed in the horizontal direction, and then the third coding units 1626c and 1626d included in the lower second coding unit 1620b are processed in the horizontal direction.

[0285] Reference Figure 16The third coding units 1616a, 1616b, 1616c and 1616d, as well as 1626a, 1626b, 1626c and 1626d of the square can be determined by dividing the second coding units 1610a and 1610b and 1620a and 1620b respectively. Although the second coding units 1610a and 1610b, determined by dividing the first coding unit 1600 in the vertical direction, are different from the second coding units 1620a and 1620b, determined by dividing the first coding unit 1600 in the horizontal direction, the third coding units 1616a, 1616b, 1616c, and 1616d, and the third coding units 1626a, 1626b, 1626c, and 1626d, derived from the second coding units 1610a and 1610b and the second coding units 1620a and 1620b, ultimately show coding units of the same shape derived from the first coding unit 1600. Thus, by recursively dividing the coding units in different ways based on the division shape pattern information, even if the coding units are ultimately determined to have the same shape, the image decoding device 100 can process multiple coding units in different orders.

[0286] Figure 17 The illustration shows the process of determining the depth of a coding unit as the shape and size of the coding unit change when multiple coding units are determined by recursively dividing the coding units according to an embodiment.

[0287] According to an embodiment, the image decoding device 100 can determine the depth of the coding unit based on a predetermined criterion. For example, the predetermined criterion may be the length of the long side of the coding unit. When the length of the long side of the coding unit before partitioning is 2n (n>0) times the length of the long side of the current coding unit after partitioning, the image decoding device 100 can determine that the depth of the current coding unit is increased by n compared to the depth of the coding unit before partitioning. In the following description, the coding unit with the increased depth is referred to as a deeper coding unit.

[0288] Reference Figure 17According to an embodiment, the image decoding device 100 can determine deeper second and third coding units 1702 and 1704 by dividing a first coding unit 1700 of a square based on block shape information indicating the shape of the square (e.g., the block shape information may be represented as "0: SQUARE"). Assuming the size of the first coding unit 1700 of the square is 2N×2N, the second coding unit 1702, determined by dividing the width and height of the first coding unit 1700 by half, can have a size of N×N. Furthermore, the third coding unit 1704, determined by dividing the width and height of the second coding unit 1702 by half, can have a size of N / 2×N / 2. In this case, the width and height of the third coding unit 1704 are 1 / 4 of the width and height of the first coding unit 1700. When the depth of the first coding unit 1700 is D, the depth of the second coding unit 1702, whose width and height are half of the width and height of the first coding unit 1700, can be D+1, and the depth of the third coding unit 1704, whose width and height are one-quarter of the width and height of the first coding unit 1700, can be D+2.

[0289] According to an embodiment, the image decoding device 100 can determine a deeper second coding unit 1712 or 1722 and a third coding unit 1714 or 1724 by dividing a non-square first coding unit 1710 or 1720 based on block shape information indicating a non-square shape (e.g., the block shape information can be represented as "1: NS_VER" indicating a non-square shape with a height greater than its width, or "2: NS_HOR" indicating a non-square shape with a width greater than its height).

[0290] The image decoding device 100 can determine the second coding unit 1702, 1712, or 1722 by dividing the width and height of the first coding unit 1710, which has a size of N×2N. That is, the image decoding device 100 can determine the second coding unit 1702 or the second coding unit 1722 with a size of N×N by dividing the first coding unit 1710 in the horizontal direction, or it can determine the second coding unit 1712 with a size of N / 2×N by dividing the first coding unit 1710 in both the horizontal and vertical directions.

[0291] According to an embodiment, the image decoding device 100 can determine the second encoding unit 1702, 1712, or 1722 by dividing the width and height of the first encoding unit 1720, which has a size of 2N×N. That is, the image decoding device 100 can determine the second encoding unit 1702 or the second encoding unit 1712 with a size of N×N by dividing the first encoding unit 1720 in the vertical direction, or it can determine the second encoding unit 1722 with a size of N×N / 2 by dividing the first encoding unit 1720 in both the horizontal and vertical directions.

[0292] According to an embodiment, the image decoding device 100 can determine a third encoding unit 1704, 1714, or 1724 by dividing the width and height of a second encoding unit 1702 with a size of N×N. That is, the image decoding device 100 can determine a third encoding unit 1704 with a size of N / 2×N / 2, a third encoding unit 1714 with a size of N / 4×N / 2, or a third encoding unit 1724 with a size of N / 2×N / 4 by dividing the second encoding unit 1702 in the vertical and horizontal directions.

[0293] According to an embodiment, the image decoding device 100 can determine the third encoding unit 1704, 1714, or 1724 by dividing the width and height of the second encoding unit 1712, which has a size of N / 2 × N. That is, the image decoding device 100 can determine the third encoding unit 1704 or the third encoding unit 1724 with a size of N / 2 × N / 4 by dividing the second encoding unit 1712 in the horizontal direction, or it can determine the third encoding unit 1714 with a size of N / 4 × N / 2 by dividing the second encoding unit 1712 in both the vertical and horizontal directions.

[0294] According to an embodiment, the image decoding device 100 can determine the third encoding unit 1704, 1714, or 1724 by dividing the width and height of the second encoding unit 1722, which has a size of N×N / 2. That is, the image decoding device 100 can determine the third encoding unit 704 with a size of N / 2×N / 2 or the third encoding unit 1714 with a size of N / 4×N / 2 by dividing the second encoding unit 1722 in the vertical direction, or it can determine the third encoding unit 1724 with a size of N / 2×N / 4 by dividing the second encoding unit 1722 in both the vertical and horizontal directions.

[0295] According to an embodiment, the image decoding device 100 can divide square coding units 1700, 1702, or 1704 in the horizontal or vertical direction. For example, the image decoding device 100 can determine a first coding unit 1710 of size N×2N by dividing a first coding unit 1700 of size 2N×2N in the vertical direction, or it can determine a first coding unit 1720 of size 2N×N by dividing a first coding unit 1700 of size 2N×2N in the horizontal direction. According to an embodiment, when the depth is determined based on the length of the longest side of the coding unit, the depth of the coding unit determined by dividing a first coding unit 1700 of size 2N×2N in the horizontal or vertical direction can be the same as the depth of the first coding unit 1700.

[0296] According to an embodiment, the width and height of the third coding unit 1714 or 1724 can be 1 / 4 of the width and height of the first coding unit 1710 or 1720. When the depth of the first coding unit 1710 or 1720 is D, the depth of the second coding unit 1712 or 1722, whose width and height are 1 / 2 of the width and height of the first coding unit 1710 or 1720, can be D+1, and the depth of the third coding unit 1714 or 1724, whose width and height are 1 / 4 of the width and height of the first coding unit 1710 or 1720, can be D+2.

[0297] Figure 18 The diagram illustrates a depth that can be determined based on the shape and size of the coding unit, and a partial index (PID) used to distinguish the coding unit, according to an embodiment.

[0298] According to an embodiment, the image decoding device 100 can determine second coding units of various shapes by dividing a first coding unit 1800 into squares. (See also...) Figure 18 The image decoding device 100 can determine second coding units 1802a and 1802b, second coding units 1804a and 1804b, and second coding units 1806a, 1806b, 1806c, and 1806d by dividing the first coding unit 1800 in at least one direction, either vertical or horizontal, based on the division shape pattern information. In other words, the image decoding device 100 can determine the second coding units 1802a and 1802b, second coding units 1804a and 1804b, and second coding units 1806a, 1806b, 1806c, and 1806d based on the division shape pattern information of the first coding unit 1800.

[0299] According to an embodiment, the depths of the second coding units 1802a and 1802b, second coding units 1804a and 1804b, and second coding units 1806a, 1806b, 1806c, and 1806d, determined based on the division shape pattern information of the square first coding unit 1800, can be determined based on the length of their longer sides. For example, since the length of the side of the square first coding unit 1800 is equal to the length of the longer side of the non-square second coding units 1802a and 1802b and 1804a and 1804b, the first coding unit 1800 and the non-square second coding units 1802a and 1802b and 1804a and 1804b can have the same depth, such as D. However, when the image decoding device 100 divides the first encoding unit 1800 into four square second encoding units 1806a, 1806b, 1806c and 1806d based on the division shape pattern information, the depth of the second encoding units 1806a, 1806b, 1806c and 1806d can be D+1 deeper than the depth D of the first encoding unit 1800.

[0300] According to an embodiment, the image decoding device 100 can determine a plurality of second coding units 1812a and 1812b, as well as 1814a, 1814b, and 1814c, by dividing a first coding unit 1810 in the horizontal direction with a height greater than its width based on the division shape pattern information. According to an embodiment, the image decoding device 150 can determine a plurality of second coding units 1822a and 1822b, as well as 1824a, 1824b, and 1824c, by dividing a first coding unit 1820 in the vertical direction with a width greater than its height based on the division shape pattern information.

[0301] According to an embodiment, the depths of second coding units 1812a and 1812b, and second coding units 1814a, 1814b and 1814c, or second coding units 1822a and 1822b, and second coding units 1824a, 1824b and 1824c, determined based on the division shape pattern information of the non-square first coding unit 1810 or 1820, can be determined based on the length of their longer sides. For example, since the length of the side of the square second coding units 1812a and 1812b is half the length of the longer side of the non-square first coding unit 1810, which has a height longer than its width, the depth of the square second coding units 1812a and 1812b is D+1 deeper than the depth D of the non-square first coding unit 1810.

[0302] Furthermore, the image decoding device 100 can divide the non-square first coding unit 1810 into an odd number of second coding units 1814a, 1814b, and 1814c based on the division shape pattern information. The odd number of second coding units 1814a, 1814b, and 1814c may include the non-square second coding units 1814a and 1814c and the square second coding unit 1814b. In this case, since the length of the long side of the non-square second coding units 1814a and 1814c and the length of the side of the square second coding unit 1814b are half the length of the long side of the first coding unit 1810, the depth of the second coding units 1814a, 1814b, and 1814c can be D+1 deeper than the depth D of the non-square first coding unit 1810. The image decoding device 100 can determine the depth of the coding unit divided from the first coding unit 1820, which has a width longer than its height, by using the method described above for determining the depth of the coding unit divided from the first coding unit 1810.

[0303] According to an embodiment, when an odd number of the divided coding units do not have equal sizes, the image decoding device 100 can determine the PID used to identify the divided coding units based on the size ratio between the coding units. (See also...) Figure 8 In an odd-numbered set of coding units 1814a, 1814b, and 1814c, the width of the central coding unit 1814b can be equal to the width of the other coding units 1814a and 1814c, and its height can be twice the height of the other coding units 1814a and 1814c. That is, in this case, the central coding unit 1814b may include two other coding units 1814a or 1814c. Therefore, when the PID of the central coding unit 1814b is 1 based on the scan order, the PID of the coding unit 1814c located adjacent to the central coding unit 1814b can be increased by 2 and thus can be 3. That is, there may be discontinuous PID values. According to an embodiment, the image decoding device 100 can determine whether the odd-numbered coding units do not have equal sizes based on whether there is a discontinuity in the PID used to identify the divided coding units.

[0304] According to an embodiment, the image decoding device 100 may determine whether to use a specific partitioning method based on PID values ​​used to identify multiple coding units determined by partitioning the current coding unit. (See also...) Figure 18The image decoding device 100 can determine an even number of coding units 1812a and 1812b or an odd number of coding units 1814a, 1814b, and 1814c by dividing a first coding unit 1810 having a rectangular shape with a height longer than its width. The image decoding device 100 can use a PID to identify each coding unit. According to an embodiment, the PID can be obtained from a sample point at a predetermined position of each coding unit (e.g., the upper left sample point).

[0305] According to an embodiment, the image decoding device 100 can determine the coding unit at a predetermined position among the divided coding units by using a PID for distinguishing coding units. According to an embodiment, when the division shape pattern information of a first coding unit 1810 having a rectangular shape with a height longer than its width indicates that the coding unit should be divided into three coding units, the image decoding device 100 can divide the first coding unit 1810 into three coding units 1814a, 1814b, and 1814c. The image decoding device 100 can assign a PID to each of the three coding units 1814a, 1814b, and 1814c. The image decoding device 100 can compare the PIDs of an odd number of divided coding units to determine the coding unit at the center position among the odd number of divided coding units. The image decoding device 100 can determine the coding unit 1814b, whose PID corresponds to the middle value among the PIDs of the coding units, as the coding unit at the center position among the coding units determined by dividing the first coding unit 1810. According to an embodiment, when the divided coding units do not have equal sizes, the image decoding device 100 can determine a PID for distinguishing the divided coding units based on the size ratio between the coding units. (Refer to...) Figure 18The width of the coding unit 1814b generated by dividing the first coding unit 1810 can be equal to the width of the other coding units 1814a and 1814c, and its height can be twice the height of the other coding units 1814a and 1814c. In this case, when the PID of the coding unit 1814b at the center position is 1, the PID of the coding unit 1814c located adjacent to the coding unit 1814b can be increased by 2 and therefore can be 3. When the PID does not increase uniformly as described above, the image decoding device 100 can determine that the coding unit is divided into a plurality of coding units, wherein the plurality of coding units includes coding units with dimensions different from those of the other coding units. According to an embodiment, when the division shape pattern information indicates that the coding unit is divided into an odd number of coding units, the image decoding device 100 can divide the current coding unit in such a way that the coding unit at a predetermined position in the odd number of coding units (e.g., the coding unit at the center position) has a dimension different from that of the other coding units. In this case, the image decoding device 100 can determine the coding unit at the center position with a different dimension by using the PID of the coding unit. However, the PID of the encoding unit at the predetermined position and its size or position are not limited to the examples above, and various PIDs of the encoding unit as well as various positions and sizes can be used.

[0306] According to an embodiment, the image decoding device 100 may use a predetermined data unit, in which encoding units are recursively divided.

[0307] Figure 19 The illustration shows how multiple encoding units are determined based on multiple predetermined data units included in the screen, according to an embodiment.

[0308] According to an embodiment, a predetermined data unit can be defined as a data unit that recursively divides coding units by using division shape pattern information. That is, the predetermined data unit can correspond to a coding unit used to determine the highest depth of a plurality of coding units divided from the current frame. In the following description, for ease of explanation, the predetermined data unit is referred to as a reference data unit.

[0309] According to an embodiment, the reference data unit may have a predetermined size and a predetermined shape. According to an embodiment, the reference coding unit may include M×N sample points. Here, M and N may be equal to each other and may be integers represented as powers of 2. That is, the reference data unit may have a square shape or a non-square shape and may be divided into an integer number of coding units.

[0310] According to an embodiment, the image decoding device 100 can divide the current frame into multiple reference data units. According to an embodiment, the image decoding device 100 can divide the multiple reference data units from the current frame using the division shape pattern information of each reference data unit. The operation of dividing the reference data units can correspond to a division operation using a quadtree structure.

[0311] According to an embodiment, the image decoding device 100 can pre-determine the minimum allowed size of the reference data units included in the current frame. Therefore, the image decoding device 100 can determine various reference data units with sizes equal to or greater than the minimum size, and can determine one or more coding units by referring to the determined reference data units and using partitioning shape pattern information.

[0312] Reference Figure 19 The image decoding device 100 may use a square reference coding unit 1900 or a non-square reference coding unit 1902. According to an embodiment, the shape and size of the reference coding unit may be determined based on various data units (e.g., sequences, frames, stripes, strip segments, maximum coding units, etc.) that can include one or more reference coding units.

[0313] According to an embodiment, the receiver (not shown) of the image decoding device 100 can obtain from the bitstream at least one of reference coding unit shape information and reference coding unit size information for each of the various data units. The above has already discussed... Figure 15 The operation of dividing the current coding unit 1500 describes the operation of dividing the square reference coding unit 1900 into one or more coding units, and the above has already been discussed... Figure 16 The operation of dividing the current coding unit 1600 describes the operation of dividing the non-square reference coding unit 1902 into one or more coding units. Therefore, its detailed description will not be provided here.

[0314] According to an embodiment, the image decoding device 100 can determine the size and shape of a reference coding unit using a PID (PID) for identifying the size and shape of a reference coding unit, based on some data units predetermined according to predetermined conditions. That is, the receiver (not shown) can obtain from the bitstream only the PID for identifying the size and shape of the reference coding unit for each slice, slice segment, or maximum coding unit, wherein the slice, slice segment, or maximum coding unit is a data unit (e.g., a data unit with a size equal to or smaller than the slice) among various data units (e.g., a sequence, a frame, a slice, a slice segment, a maximum coding unit, etc.) that meets predetermined conditions. The image decoding device 100 can determine the size and shape of the reference data unit for each data unit that meets the predetermined conditions by using the PID. When obtaining and using reference coding unit shape information and reference coding unit size information from the bitstream based on each data unit with a relatively small size, the efficiency of using the bitstream may be low; therefore, only the PID can be obtained and used, instead of directly obtaining the reference coding unit shape information and reference coding unit size information. In this case, at least one of the size and shape of the reference coding unit corresponding to the PID for identifying the size and shape of the reference coding unit can be predetermined. In other words, the image decoding device 100 can determine at least one of the dimensions and shapes of the reference coding unit included in the data unit used as the unit for obtaining the PID by selecting at least one of the dimensions and shapes of the reference coding unit predetermined based on the PID.

[0315] According to embodiments, the image decoding device 100 may use one or more reference coding units included in the maximum coding unit. That is, the maximum coding unit divided from the image may include one or more reference coding units, and the coding unit can be determined by recursively dividing each reference coding unit. According to embodiments, at least one of the width and height of the maximum coding unit may be an integer multiple of at least one of the width and height of the reference coding unit. According to embodiments, the size of the reference coding unit can be obtained by dividing the maximum coding unit n times based on a quadtree structure. That is, according to various embodiments, the image decoding device 100 may determine the reference coding unit by dividing the maximum coding unit n times based on a quadtree structure, and may divide the reference coding unit based on at least one of block shape information and division shape pattern information.

[0316] Figure 20 The diagram illustrates a processing block, according to an embodiment, used as a standard for determining the order of reference coding units included in a screen 2000.

[0317] According to an embodiment, the image decoding device 100 can determine one or more processing blocks divided from the image. A processing block is a data unit divided from the image that includes one or more reference coding units, and the one or more reference coding units included in the processing block can be determined according to a specific order. That is, the order in which the one or more reference coding units determined in each processing block can correspond to one of various types of orders used to determine reference coding units, and can vary depending on the processing block. The order in which the reference coding units determined for each processing block can be one of various orders (e.g., raster scan order, zigzag scan, N-shaped scan, upper right diagonal scan, horizontal scan, and vertical scan), but is not limited to the scan orders mentioned above.

[0318] According to an embodiment, the image decoding device 100 can obtain processing block size information and determine the size of one or more processing blocks included in the frame. The image decoding device 100 can obtain processing block size information from a bitstream and determine the size of one or more processing blocks included in the frame. The size of a processing block can be a predetermined size of a data unit indicated by the processing block size information.

[0319] According to an embodiment, the receiver (not shown) of the image decoding device 100 can obtain processing block size information from the bitstream based on each specific data unit. For example, processing block size information can be obtained from the bitstream according to data units such as images, sequences, frames, stripes, or strip segments. That is, the receiver (not shown) can obtain processing block size information from the bitstream based on each of the various data units, and the image decoding device 100 can determine the size of one or more processing blocks divided from the frame by using the obtained processing block size information. The size of the processing block can be an integer multiple of the size of the reference coding unit.

[0320] According to an embodiment, the image decoding device 100 can determine the sizes of processing blocks 2002 and 2012 included in the image 2000. For example, the image decoding device 100 can determine the size of the processing block based on processing block size information obtained from the bitstream. (Refer to...) Figure 20 According to an embodiment, the image decoding device 100 may determine the width of processing blocks 2002 and 2012 to be four times the width of the reference coding unit, and may determine the height of processing blocks 2002 and 2012 to be four times the height of the reference coding unit. The image decoding device 100 may determine the determination order of one or more reference coding units in one or more processing blocks.

[0321] According to an embodiment, the image decoding device 100 can determine the processing blocks 2002 and 2012 included in the image 2000 based on the size of the processing blocks, and can determine the determination order of one or more reference coding units in the processing blocks 2002 and 2012. According to an embodiment, determining the reference coding unit may include determining the size of the reference coding unit.

[0322] According to an embodiment, the image decoding device 100 can obtain determination order information of one or more reference coding units included in one or more processing blocks from a bitstream, and can determine a determination order for one or more reference coding units based on the obtained determination order information. The determination order information can be defined as used to determine the order or direction of the reference coding units in a processing block. That is, the determination order of the reference coding units can be determined independently for each processing block.

[0323] According to an embodiment, the image decoding device 100 can obtain deterministic order information of reference coding units from the bitstream based on each specific data unit. For example, a receiver (not shown) can obtain deterministic order information of reference coding units from the bitstream based on each data unit (such as an image, sequence, frame, strip, strip fragment, or processing block). Because the deterministic order information of the reference coding units indicates the order of the reference coding units used to determine the processing block, deterministic order information can be obtained for each specific data unit comprising an integer number of processing blocks.

[0324] According to an embodiment, the image decoding device 100 may determine one or more reference coding units based on a determined order.

[0325] According to an embodiment, a receiver (not shown) can obtain the determined order information of reference coding units from the bitstream as information related to processing blocks 2002 and 2012, and the image decoding device 100 can determine the determined order of one or more reference coding units included in processing blocks 2002 and 2012, and determine one or more reference coding units included in the frame 2000 based on the determined order. (Refer to...) Figure 20 The image decoding device 100 can determine the determination order 2004 and 2014 of one or more reference coding units in processing blocks 2002 and 2012, respectively. For example, when obtaining the determination order information of reference coding units for each processing block, determination order information of different types of reference coding units can be obtained for processing blocks 2002 and 2012. When the determination order 2004 of reference coding units in processing block 2002 is a raster scan order, the reference coding units included in processing block 2002 can be determined according to the raster scan order. Conversely, when the determination order 2014 of reference coding units in another processing block 2012 is a reverse raster scan order, the reference coding units included in processing block 2012 can be determined according to the reverse raster scan order.

[0326] According to an embodiment, the image decoding device 100 can decode one or more determined reference coding units. The image decoding device 100 can decode an image based on the reference coding units determined as described above. Methods for decoding the reference coding units may include various image decoding methods.

[0327] According to an embodiment, the image decoding device 100 can obtain block shape information indicating the shape of the current coding unit or partition shape pattern information indicating the partitioning method of the current coding unit from the bitstream, and can use the obtained information. The partition shape pattern information can be included in the bitstream associated with various data units. For example, the image decoding device 100 can use the partition shape pattern information included in the sequence parameter set, picture parameter set, video parameter set, strip header, or strip segment header. Furthermore, the image decoding device 100 can obtain syntax elements corresponding to the block shape information or partition shape pattern information from the bitstream according to each maximum coding unit, each reference coding unit, or each processing block, and can use the obtained syntax elements.

[0328] The method for determining partitioning rules according to embodiments of the present disclosure will be described in detail below.

[0329] Image decoding device 100 can determine the segmentation rules of an image. The segmentation rules can be predetermined between image decoding device 100 and image encoding device 400. Image decoding device 100 can determine the segmentation rules based on information obtained from the bitstream. Image decoding device 100 can determine the segmentation rules based on information obtained from at least one of sequence parameter set, picture parameter set, video parameter set, strip header, and strip segment header. Image decoding device 100 can determine the segmentation rules differently based on frames, stripes, time layers, maximum coding units, or coding units.

[0330] Image decoding device 100 may determine partitioning rules based on block shape information of the coding units. The block shape may include the size, shape, aspect ratio, and orientation of the coding unit. Image encoding device 400 and image decoding device 100 may determine partitioning rules based on pre-determined block shape information of the coding units. However, embodiments are not limited to this. Image decoding device 100 may determine partitioning rules based on information obtained from the bitstream received from image encoding device 400.

[0331] The shape of the encoding unit can be square or non-square. When the width and height of the encoding unit are the same, the image decoding device 100 can determine the shape of the encoding unit as square. Furthermore, when the width and height of the encoding unit are not the same, the image decoding device 100 can determine the shape of the encoding unit as non-square.

[0332] The size of the coding unit can include various sizes, such as 4×4, 8×4, 4×8, 8×8, 16×4, 16×8, and up to 256×256. The size of the coding unit can be classified based on the length of its long side, the length of its short side, or its area. The image decoding device 100 can apply the same classification rules to coding units classified into the same group. For example, the image decoding device 100 can classify coding units with the same long side length as having the same size. Furthermore, the image decoding device 100 can apply the same classification rules to coding units with the same long side length.

[0333] The aspect ratio of the coding unit may include 1:2, 2:1, 1:4, 4:1, 1:8, 8:1, 1:16, or 16:1. Furthermore, the orientation of the coding unit may include a horizontal direction and a vertical direction. A horizontal orientation may indicate that the width of the coding unit is longer than its height. A vertical orientation may indicate that the width of the coding unit is shorter than its height.

[0334] The image decoding device 100 can adaptively determine the partitioning rules based on the size of the coding unit. The image decoding device 100 can determine the permissible partitioning shape patterns based on different coding unit sizes. For example, the image decoding device 100 can determine whether a partition is permissible based on the size of the coding unit. The image decoding device 100 can determine the partitioning direction based on the size of the coding unit. The image decoding device 100 can determine the permissible partitioning type based on the size of the coding unit.

[0335] The partitioning rule determined based on the size of the coding unit can be a partitioning rule predetermined between the image encoding device 400 and the image decoding device 100. Alternatively, the image decoding device 100 can determine the partitioning rule based on information obtained from the bitstream.

[0336] The image decoding device 100 can adaptively determine the partitioning rules based on the position of the coding units in the image.

[0337] Furthermore, the image decoding device 100 can determine partitioning rules such that encoding units generated via different partitioning paths do not have the same block shape. However, the embodiment is not limited to this; encoding units generated via different partitioning paths may have the same block shape. Encoding units generated via different partitioning paths may have different decoding processing orders. As already referred to above... Figure 16 The decoding process order has been described, so its details are not provided here.

[0338] In the following text, reference will be made to Figures 21a to 30Detailed description of image coding and image decoding devices, as well as image coding and image decoding methods, according to various embodiments for adaptively applying filters to reference samples in an inter-frame prediction mode based on the current block.

[0339] Figure 21a and Figure 21b This is a diagram illustrating a method for performing filtering on reference samples for motion compensation during inter-frame prediction according to various embodiments.

[0340] Figure 21a Reference samples at integer pixel positions and reference samples at sub-pixel positions are shown according to various embodiments.

[0341] Interpolation filters can be applied to reference samples at integer pixel locations to generate reference samples at sub-pixel locations located between integer pixels. Interpolation filters can also be applied to neighboring samples at integer pixel locations adjacent to the reference samples at the generated sub-pixel locations. For example, the application of an interpolation filter involves sequentially multiplying the coefficients of the interpolation filter with the sample values ​​at the integer pixel locations and generating a value obtained by summing the multiplication results (or by scaling the sum) as the interpolated sample value. The samples at the integer pixel locations to which the interpolation filter has been applied are arranged in a line surrounding the reference samples at the generated sub-pixel locations, and the interpolation filter can be applied to a number of samples equal to the number of coefficients of the interpolation filter.

[0342] exist Figure 21a In the reference samples shown, samples indicated by uppercase letters of the alphabet such as A, B, C, D, E, F, G, H, I, J, K, and L are samples at integer pixel positions. Samples indicated by lowercase letters of the alphabet such as a, b, c, e, f, and g are samples at sub-pixel positions. Figure 21a The image shows only subpixels at 1 / 4 pixel units, but the samples at the subpixel locations are not limited to this, and samples at the subpixel locations can be generated in 1 / 8 pixel units, 1 / 16 pixel units, or smaller units.

[0343] For example, refer to Figure 21aReference numeral 2100 shows that samples a, b, and c at a sub-pixel location can be generated by applying an interpolation filter to eight reference samples (such as A, B, C, D, E, F, G, and H) at integer pixel locations adjacent to the sub-pixel location. In other words, the interpolation filter used to generate the samples at the sub-pixel location can be an 8-tap filter. Here, the reference samples at integer pixel locations to which the interpolation filter will be applied can be selected from the left and right sides of the sub-pixel location in adjacent order. However, the filter coefficient sets of the interpolation filter used to generate sample a at the sub-pixel location, the filter coefficient sets of the interpolation filter used to generate sample b at the sub-pixel location, and the filter coefficient sets of the interpolation filter used to generate sample c at the sub-pixel location can be different from each other.

[0344] For example, refer to Figure 21a Reference numeral 2110 shows that samples e, g, and f at a sub-pixel location can be generated by applying an interpolation filter to four reference samples (such as I, J, K, and L) at integer pixel locations adjacent to the sub-pixel location. In other words, the interpolation filter used to generate the samples at the sub-pixel location can be a 4-tap filter. The filter coefficient sets of the interpolation filter used to generate sample e at the sub-pixel location, the filter coefficient sets of the interpolation filter used to generate sample f at the sub-pixel location, and the filter coefficient sets of the interpolation filter used to generate sample g at the sub-pixel location can be different from each other.

[0345] According to various embodiments, the interpolation filter used by the image coding device 400 and the image decoding device 100 to generate reference samples for motion compensation (prediction) can be determined based on the inter-frame prediction mode of the current block. According to various embodiments, the orientation or size of the interpolation filter can be determined based on the inter-frame prediction mode of the current block.

[0346] According to various embodiments, the interpolation filter used by the image coding device 400 and the image decoding device 100 to generate reference samples for motion compensation (prediction) can also be determined based on the size of the current block, whether the current block is a luma block or a chroma block, and / or whether the inter-frame prediction mode of the current block is one-way prediction or two-way prediction.

[0347] exist Figure 21a The example shown only uses neighboring reference samples at integer pixel locations in the horizontal direction, but similarly, samples at sub-pixel locations can be generated by applying an interpolation filter to neighboring reference samples at integer pixel locations in the vertical direction.

[0348] Figure 21b This shows an example of the reference samples required for inter-frame prediction of the current block.

[0349] The image decoding device 100 can determine a reference block 2180 from the position of a reference sample point indicated by the motion vector of the current block in a reference frame 2150 indicated by the reference frame index of the current block. For example, a reference block can be determined such that the motion vector of the current block indicates a sample point located at the upper left boundary of the reference block.

[0350] According to various embodiments, when the reference sample point indicated by the motion vector of the current block is a sample point at a sub-pixel location, the reference block 2180 may include the same number of reference sample points at sub-pixel locations as the current block. Neighboring reference sample points at integer pixel locations may be as follows: Figure 21a The reference samples shown are used to generate reference samples at each sub-pixel position in reference block 2180. Therefore, the reference samples required to generate reference block 2180 corresponding to the current block may include samples at integer pixel positions in reference block 2180 and samples 2170 at integer pixel positions adjacent to reference block 2180.

[0351] Figure 21b The illustration shows an example where the current block size is 8×8 and 8-tap filters in the horizontal and vertical directions are used as interpolation filters. In this case, the reference block 2180 determined based on the motion vector of the current block may include reference samples at 8×8 = 64 sub-pixel positions. Eight left and right neighboring reference samples or top and bottom neighboring reference samples at integer pixel positions can be selected for each sub-pixel position, and reference samples at the sub-pixel positions can be generated by applying interpolation filters to the selected integer pixel positions. For example, to generate reference samples at 64 sub-pixel positions, interpolation filters can be applied to reference samples at integer pixel positions in the reference block, specifically to reference samples at integer pixel positions in the 3 columns adjacent to the left of the reference block, the 4 columns adjacent to the right of the reference block, the 3 rows adjacent to the top of the reference block, and the 4 rows adjacent to the bottom of the reference block. In this case, the number of reconstructed reference samples read from the reference frame for inter-frame prediction of the 8×8 current block can be as high as (8+7)×(8+7) = 225.

[0352] Figure 22a The reference samples required for inter-frame prediction of the current block according to various embodiments are shown. Figure 22a It can be a reference Figure 21a and Figure 21b A more generalized example of the described embodiments.

[0353] In the current block 2210, the horizontal width is referred to as W, and the vertical height is referred to as H. The position in the reference frame indicated by the motion vector, as indicated by the reference frame index of the current block 2210, can be determined as the reference sample position 2230. The reference sample position 2230 can be determined by adding the motion vector component of the current block 2210 to the upper-left coordinate of the current block 2210. The reference sample position 2230 can indicate the upper-left position of the reference block 2240 that will be referenced by the current block 2210 during inter-frame prediction.

[0354] The interpolation filter is configured with separable 2D filters (i.e., horizontal and vertical filters), and it is assumed that the filter size is T, i.e., a T-tap filter is used. In this case, in order to generate reference samples at sub-pixel positions in reference block 2240, T neighboring reference samples at integer pixel positions are required, both vertically and horizontally. Therefore, the number of reconstructed reference samples 2220 read from the reconstructed reference image to generate all reference samples included in reference block 2240 can be as high as (W+T-1)×(H+T-1).

[0355] As a result, the average number of reference samples read from the reconstructed reference image stored in memory to predict the current sample can be expressed as Equation 1.

[0356] [Equation 1]

[0357]

[0358] According to various embodiments, Equation 1 can be defined as the memory bandwidth. According to Equation 1, the memory bandwidth increases as the size of the current block (W and H) decreases, and the memory bandwidth also increases as the size of the interpolation filter (i.e., the number of taps T) increases. Therefore, in order to keep the memory bandwidth within a certain level, it may be necessary to reduce the number of filter taps when the size of the current block decreases.

[0359] According to various embodiments, the image decoding device 100 can determine filter information such that the memory bandwidth is within a specific range. According to various embodiments, the image decoding device 100 can adjust the memory bandwidth by determining the filter size based on the size of the current block. According to an embodiment, the image decoding device 100 can determine to use a filter with a first number of taps when the size of the current block is equal to or greater than a specific reference value; otherwise, it determines to use a filter with a second number of taps smaller than the first number of taps. According to an embodiment, the specific reference value can be determined based on the inter-frame prediction mode of the current block. According to an embodiment, the image encoding device 400 or the image decoding device 100 can compare the size of the current block with multiple reference values ​​and determine the filter to be used from a plurality of filters based on the comparison results.

[0360] Figure 22b The reference samples required for inter-frame prediction of the current block when the current block references multiple reference blocks are shown according to various embodiments.

[0361] Figure 22b One embodiment is shown, wherein the horizontal width of the current block 2210 is referred to as W and its vertical height as H, and a separable 2D T-tap filter is shown as... Figure 22a It is used as an interpolation filter, but with Figure 22a Unlike the previous example, the current block 2210 is predicted bidirectionally.

[0362] When the current block is predicted bidirectionally, two motion vectors are determined for inter-frame prediction of the current block. Therefore, two reference sample locations indicated by the motion vectors and two reference blocks corresponding to those reference sample locations can be determined. These two reference sample locations can indicate positions within the same reference frame or positions in different reference frames. In this case, the maximum number of reconstructed reference samples 2220 and 2250 required to generate the reference samples for the reference blocks is 2 × (W + T - 1) × (H + T - 1), where this maximum number is twice that of unidirectional prediction.

[0363] When the current block references N reference blocks, the average number of reference samples to predict the current sample can be calculated as shown in Equation 2.

[0364] [Equation 2]

[0365]

[0366] According to various embodiments, Equation 2 can be defined as the memory bandwidth. According to Equation 2, the memory bandwidth increases with the number of reference blocks used for prediction, and also with the size of the interpolation filter (i.e., the number of taps T). Therefore, in order to keep the memory bandwidth within a certain level, it may be necessary to reduce the number of filter taps as the number of reference blocks increases.

[0367] According to various embodiments, the image decoding device 100 can determine filter information such that the memory bandwidth is within a specific range. According to various embodiments, the image decoding device 100 can determine the filter size based on whether the inter-frame prediction of the current block is unidirectional or bidirectional, thereby adjusting the memory bandwidth. According to embodiments, the image decoding device 100 can determine to use a first filter when the current block is unidirectionally predicted, and determine to use a second filter when the current block is bidirectionally predicted. According to embodiments, the size of the first filter can be twice the size of the second filter.

[0368] According to various embodiments, the image decoding device 100 can adjust the memory bandwidth by determining the filter size based on the number of reference blocks referenced by the current block. According to an embodiment, the image decoding device 100 can determine to use a filter with a first number of taps when the number of reference blocks referenced by the current block is less than or equal to a specific reference value; otherwise, it determines to use a filter with a second number of taps smaller than the first number of taps. According to an embodiment, the image decoding device 100 can compare the number of reference blocks referenced by the current block with multiple reference values ​​and determine the filter to be used from among multiple filters based on the comparison results.

[0369] Figure 23 This illustrates the memory bandwidth required for inter-frame prediction of the current block according to various embodiments.

[0370] Figure 23 Table 2300 shows the results according to Figure 22a or Figure 22b The memory bandwidth is calculated according to Equations 1 and 2 when the horizontal width is referred to as W and the vertical height as H, and a separable 2D T-tap filter is used as an interpolation filter in the embodiment. Various examples are provided for the case 2310 of the luma block and the case 2320 of the chroma block sampled in 4:2:0, based on the size of the current block and the number of taps.

[0371] In the HEVC / H.265 standard, inter-frame prediction is not permitted for 4×4 size blocks targeting luma blocks. Only unidirectional prediction is allowed when the current block size is 4×8 or 8×4. Bidirectional prediction is allowed when the current block size is equal to or greater than 8×8. Therefore, refer to... Figure 23 In HEVC, when the current block size is 8×8 and an 8-tap filter is used for bidirectional prediction, the maximum memory bandwidth required is calculated to be approximately 7.03 according to Equation 2. This value can be defined as the HEVC limit.

[0372] According to various embodiments, the image decoding device 100 can determine the size of the filter such that the memory bandwidth calculated according to Equation 1 or Equation 2 does not exceed a specific reference value. According to an embodiment, when the memory bandwidth of the combination of the current block size, the number of filter taps, and one-way / two-way prediction is less than the specific reference value, the image decoding device 100 can determine that the combination is suitable for inter-frame prediction.

[0373] For example, the specific reference value may be the same as the HEVC limit. Figure 23 The values ​​of memory bandwidth exceeding the HEVC limit are indicated in bold in the tables and other figures.

[0374] In the case 2310 of the luma block, according to the embodiment where the specific reference value is set as the HEVC limit, when the current block size is 4×4, the filter size can be determined to be 4 taps or less in unidirectional and bidirectional prediction. According to another embodiment, when the current block size is 4×4, an 8-tap filter is allowed in unidirectional prediction.

[0375] In the case of chroma blocks 2320, because the number of samples included in the chroma block is 1 / 4 of the number of samples in a luminance block of the same size when the chroma block is sampled at a 4:2:0 ratio, the memory bandwidth is equal to the value obtained by dividing the result of Equation 1 or Equation 2 by 4. According to an embodiment where the specific reference value is set to the HEVC limit, when the current chroma block size is 2×2, the filter size can be determined to be 8 taps or less in unidirectional prediction, and 4 taps or less in bidirectional prediction.

[0376] Figure 24 The diagram illustrates the reference samples required for inter-frame prediction of the current block when using filters of different sizes in the vertical and horizontal directions, according to various embodiments.

[0377] like Figures 22a to 23 As assumed in the previous embodiment, in the current block 2410, the horizontal width is referred to as W and the vertical height as H, and the interpolation filter used to obtain the reference sample is configured with separable 2D filters (i.e., a horizontal filter and a vertical filter). However, unlike the previous embodiment, it is assumed that a T1 tap filter is applied in the horizontal direction and a T2 tap filter is applied in the vertical direction.

[0378] In this case, for each sub-pixel position, T1 horizontal neighboring reference points at the integer pixel position and T2 vertical neighboring reference points at the integer pixel position are needed to generate the reference point at the sub-pixel position. Therefore, the total number of reconstructed reference points 2420 read from the reconstructed reference screen to generate all reference points included in the reference block (shown in dashed lines) corresponding to the current block 2410 is (W+T1-1)×(H+T2-1).

[0379] Therefore, the average number of reference samples (i.e., memory bandwidth) read from the reconstructed reference image stored in memory to predict the current sample can be defined as Equation 3.

[0380] [Equation 3]

[0381]

[0382] Here, N represents the number of reference blocks that will be referenced for inter-frame prediction of the current block. N is 1 during unidirectional prediction and 2 during bidirectional prediction.

[0383] According to an embodiment, the image decoding device 100 can independently determine the size of the horizontal filter and the size of the vertical filter, such that the memory bandwidth calculated by Equation 3 is within a specific range. According to an embodiment, the number of taps T1 of the horizontal filter can be determined based on the horizontal width W, and the number of taps T2 of the vertical filter can be determined based on the vertical height H. According to an embodiment, when the current block is not square, the image decoding device 100 can determine the horizontal filter and the vertical filter as filters of different sizes.

[0384] According to an embodiment, the image decoding device 100 can determine horizontal filter information and vertical filter information based on the ratio of the horizontal width to the vertical height of the current block. For example, when the horizontal filter information of the current block is determined, the image decoding device 100 can determine the vertical filter information based on the horizontal filter information and the ratio of the horizontal width to the vertical height of the current block. In another example, when the vertical filter information of the current block is determined, the image decoding device 100 can determine the horizontal filter information based on the vertical filter information and the ratio of the horizontal width to the vertical height of the current block. According to an embodiment, the image decoding device 100 can determine the size of the horizontal filter and the size of the vertical filter to be proportional to the horizontal width and vertical height of the current block. For example, T1 and T2 can be determined such that the ratio T1:T2 of the number of taps of the horizontal filter to the number of taps of the vertical filter matches the ratio W:H of the horizontal width to the vertical height.

[0385] Figure 25a This illustrates prediction methods for OBMC patterns without using sub-blocks, according to various embodiments.

[0386] In OBMC mode without using sub-blocks, the motion vectors of blocks 2520, 2522, 2524, and 2528 adjacent to the current block 2510 can be used for motion prediction of the current block 2510. Specifically, the motion vector of block 2522, which is adjacent above the current block 2510, can be used to predict samples Y0 and Y1 adjacent to the upper boundary of the current block 2510, and the motion vector of block 2528, which is adjacent below the current block 2510, can be used to predict samples Y2 and Y3 adjacent to the lower boundary of the current block 2510. Similarly, the motion vector of block 2520, which is adjacent to the left of the current block 2510, can be used to predict samples Y0 and Y2 adjacent to the left boundary of the current block 2510, and the motion vector of block 2524, which is adjacent to the right of the current block 2510, can be used to predict samples Y1 and Y3 adjacent to the right boundary of the current block 2510.

[0387] However, when decoding the current block 2510 according to the encoding order of the raster scan, the block 2528 adjacent to the current block 2510 below is not decoded, and in this case, the motion vector of the current block 2510 can be used instead of the motion vector of the block 2528 adjacent to the current block 2510 below.

[0388] Multiple predicted values ​​are obtained by using the motion vector of the current block and the motion vectors of the blocks adjacent to the current block. According to an embodiment, reference samples corresponding to the following can be obtained: the four lower rows R4 and R5 of block 2522 adjacent above the current block 2510, the four right columns R6 and R7 of block 2520 adjacent to the left of the current block 2510, and the four left columns R8 and R9 of block 2524 adjacent to the right of the current block 2510. The final predicted value of the current block can be obtained by performing a weighted average of the multiple predicted values.

[0389] Figure 25b The reference samples required for inter-frame prediction of the current block in OBMC mode without using sub-blocks are shown according to various embodiments.

[0390] like Figure 25a As shown, in the OBMC mode without using sub-blocks, in addition to the reference points of the reference block 2544 corresponding to the current block 2530, reference points corresponding to the four pixels adjacent to each of the top, left, and right sides of the current block 2530 can be further used. Therefore, the OBMC mode without using sub-blocks can be described as performing motion compensation on block 2542, where the current block in block 2542 extends by four pixels along each of the top, left, and right sides.

[0391] Here, when the horizontal width of the current block 2530 is referred to as W and its vertical height as H, and a separable 2D T-tap filter is used as an interpolation filter, the total number of reconstructed reference samples 2540 read from the reconstructed reference frame for inter-frame prediction of the current block 2530 is (W+8+T-1)×(H+4+T-1). Therefore, in OBMC mode without using sub-blocks, the average number of reference samples read from the reconstructed reference frame stored in memory to predict the current sample (i.e., memory bandwidth) can be calculated as in Equation 4.

[0392] [Equation 4]

[0393]

[0394] Here, N represents the number of reference blocks that will be referenced for inter-frame prediction of the current block. N is 1 during unidirectional prediction and 2 during bidirectional prediction.

[0395] Figure 25cThe diagram illustrates the memory bandwidth required for inter-frame prediction of the current block in OBMC mode without using sub-blocks, according to various embodiments. Figure 25c Showing according to Figure 25a or Figure 25b The memory bandwidth of the embodiment is calculated according to Equation 4 when the horizontal width of the current block is called W and its vertical height is called H, and a separable 2D T-tap filter is used as an interpolation filter. Various examples are provided for case 2560 of a luminance block sampled at 4:2:0 and case 2570 of a chrominance block.

[0396] According to various embodiments, the image decoding device 100 can determine the size of the filter such that the memory bandwidth calculated according to Equation 4 does not exceed a specific reference value. According to embodiments, the specific reference value can be determined based on the aforementioned HEVC limit.

[0397] In the case of a luminance block 2560, according to an embodiment where the specific reference value is set to the HEVC limit, when the current block size is 8×8, the filter size can be determined to be 2 taps or less in bidirectional prediction, and the filter size can be determined to be 8 taps or less in unidirectional prediction. When the current block size is 4×4, even when using a 2-tap filter, the memory bandwidth exceeds the HEVC limit, therefore it can be determined that the OBMC mode without using sub-blocks is not used for 4×4 blocks. However, according to another embodiment, when a 2-tap filter is used for unidirectional prediction of a 4×4 block, the increase in memory bandwidth compared to the HEVC limit is small, allowing for a 2-tap filter.

[0398] In the case of chroma blocks 2570, because the number of samples included in the chroma block is 1 / 4 of the number of samples in a luma block of the same size when the chroma block is sampled at a 4:2:0 ratio, the memory bandwidth is equal to the value obtained by dividing the result of Equation 4 by 4. According to an embodiment where the specific reference value is set as the HEVC limit, the filter size can be determined to be 2 taps or less when the current block size is 2×2. According to another embodiment, a 4-tap filter is allowed in unidirectional prediction when the current block size is 2×2.

[0399] Figure 26a This illustrates prediction methods for OBMC patterns using sub-blocks according to various embodiments.

[0400] In OBMC mode using sub-blocks, the current blocks 2610 and 2620, which are predicted inter-frame, can be divided into 4×4 sub-blocks, and motion compensation can be performed on a sub-block basis. Motion vectors of sub-blocks adjacent to the current sub-block can be used for motion prediction of the current sub-block. According to an embodiment, multiple prediction values ​​are obtained by using the motion vectors of the current block and the motion vectors of the sub-blocks adjacent to the current sub-block. Then, a weighted average of the multiple prediction values ​​is performed to obtain the final prediction value of the current sub-block.

[0401] According to various embodiments, motion compensation for sub-block units can be performed at the boundaries of the current block 2610. According to an embodiment, the motion vector of the current block 2610 and the motion vector of the sub-block 2612 adjacent above the current sub-block 2611 can be used for motion prediction of the sub-block 2611 located at the upper boundary of the current block 2610. According to an embodiment, the motion vector of the current block 2610 and the motion vector of the sub-block 2617 adjacent to the left of the current sub-block 2616 can be used for motion prediction of the sub-block 2616 located at the left boundary of the current block 2610. According to an embodiment, the motion vector of the current block 2610, the motion vector of the sub-block 2615 adjacent to the left of the current sub-block 2613, and the motion vector of the sub-block 2614 adjacent above the current sub-block 2613 can all be used for motion prediction of the sub-block 2613 adjacent to both the left and upper boundaries of the current block 2610.

[0402] According to various embodiments, motion compensation for sub-block units can be performed at locations other than the boundaries of the current block 2620. According to embodiments, the motion vector of the current block 2620 and the motion vectors of the sub-blocks 2625, 2626, 2627, and 2628 adjacent to the current sub-block 2622 above, below, to the left, and to the right can all be used for motion prediction of the current sub-block 2622.

[0403] Figure 26b The reference samples required for inter-frame prediction of the current block in OBMC mode using sub-blocks are shown according to various embodiments.

[0404] like Figure 26a As shown, in the OBMC mode using sub-blocks, motion prediction can be performed by using reference samples corresponding to 4×4-sized sub-blocks adjacent to the current block 2630 above, below, left, and right of the current block, in addition to using reference samples from the reference block 2644 corresponding to the current block 2630. Therefore, the OBMC mode using sub-blocks can be viewed as performing motion compensation on block 2642, where the current block is extended by 4 pixels on each of the top, bottom, left, and right sides.

[0405] Here, when the horizontal width of the current block 2630 is referred to as W and its vertical height as H, and a separable 2D T-tap filter is used as an interpolation filter, the total number of reconstructed reference samples 2640 read from the reconstructed reference frame for inter-frame prediction of the current block 2630 is (W+8+T-1)×(H+8+T-1). Therefore, in the OBMC mode using sub-blocks, the average number of reference samples read from the reconstructed reference frame stored in memory to predict the current sample (i.e., memory bandwidth) can be calculated as in Equation 5.

[0406] [Equation 5]

[0407]

[0408] Here, N represents the number of reference blocks that will be referenced for inter-frame prediction of the current block. N is 1 during unidirectional prediction and 2 during bidirectional prediction.

[0409] Figure 26c The memory bandwidth required for inter-frame prediction of the current block in OBMC mode using sub-blocks is shown according to various embodiments. Figure 26c Showing according to Figure 26a or Figure 26b The memory bandwidth of the embodiment is calculated according to Equation 5 when the horizontal width of the current block is called W and its vertical height is called H, and a separable 2D T-tap filter is used as an interpolation filter. Various examples are provided for the case 2660 of a luminance block sampled at 4:2:0 and the case 2670 of a chrominance block.

[0410] According to various embodiments, the image decoding device 100 can determine the size of the filter such that the memory bandwidth calculated according to Equation 5 does not exceed a specific reference value. According to embodiments, the specific reference value can be determined based on the aforementioned HEVC limit.

[0411] In the case of a luminance block 2660, when the current block size is 4×4, the memory bandwidth exceeds the HEVC limit even when using a 2-tap filter. Therefore, according to an embodiment that sets the specific reference value as the HEVC limit, the OBMC mode using sub-blocks can be determined not to be used for 4×4 blocks. According to another embodiment, when the current block size is 4×4, the filter size can be determined to be 2 taps or smaller.

[0412] In the case of chroma blocks 2670, because the number of samples included in the chroma block is 1 / 4 of the number of samples in a luma block of the same size when the chroma block is sampled at a 4:2:0 ratio, the memory bandwidth is equal to the value obtained by dividing the result of Equation 5 by 4. According to another embodiment, when the current block size is 2×2, the filter size can be determined to be 2 taps or less. According to another embodiment, when the current block size is 2×2, even when using a 2-tap filter, the memory bandwidth exceeds the HEVC limit, therefore it can be determined that the OBMC mode using sub-blocks is not used for 2×2 blocks.

[0413] Figures 27a to 28d Methods for predicting DMVR patterns according to various embodiments are shown.

[0414] In DMVR mode, motion vectors used for inter-frame prediction can be adjusted via template matching. Figure 27a This demonstrates a method for adjusting motion vectors in bidirectional prediction.

[0415] In bidirectional prediction, two original prediction blocks 2721 and 2726 are obtained using the two original motion vectors MV0 and MV1 of the current block 2715. Template 2730 is generated by performing a weighted average on the original prediction blocks 2721 and 2726 (step 1). The image decoding device 100 performs template matching by searching for the updated prediction blocks 2722 and 2727 in reference frames 2720 and 2725 that have the smallest difference with template 2730 (step 2). The updated motion vectors MV0' and MV1' indicating the updated prediction blocks 2722 and 2727 can be determined via template matching. The final prediction value of the current block 2715 is obtained from the updated prediction blocks 2722 and 2727.

[0416] According to various embodiments, the number of motion vector candidates used to search for updated prediction blocks 2722 and 2727 in reference frames 2720 and 2725 can be limited during template matching. For example, image decoding device 100 can determine up to M motion vector candidates. Image decoding device 100 can compare template 2730 with a plurality of prediction block candidates indicated by motion vector candidates in the reference frames and determine the prediction block candidate with the smallest difference as the updated prediction blocks 2722 and 2727.

[0417] According to various embodiments, the search area used to search for updated prediction blocks 2722 and 2727 in reference frames 2720 and 2725 can be limited. For example, image decoding device 100 can search for updated prediction blocks 2722 and 2727 in areas that extend L pixels along the top, bottom, left, and right sides of the original prediction blocks 2721 and 2726, respectively.

[0418] According to an embodiment, the image decoding device 100 can search for an updated prediction block in a region extending by one pixel along the top, bottom, left, and right sides of the original prediction block. According to an embodiment, the image decoding device 100 can determine the original motion vector and eight motion vectors obtained by moving the original motion vector by one pixel along the top, bottom, left, right, upper left, lower left, upper right, and lower right sides as motion vector candidates. The image decoding device 100 can compare the template 2730 with the prediction block candidates indicated by the motion vector candidates and determine the prediction block candidate with the smallest difference as the updated prediction blocks 2722 and 2727.

[0419] exist Figure 27a Only bidirectional prediction is shown, but the embodiments are not limited thereto, and those skilled in the art will understand that the DMVR mode can be used by applying the above method even for unidirectional prediction or multiple prediction using three or more references.

[0420] Figure 27b The reference samples required for inter-frame prediction of the current block in DMVR mode are shown according to various embodiments.

[0421] like Figure 27b As shown, the original prediction block 2754 is determined based on the motion vector of the reference frame indicated by the reference frame index from the current block 2740. According to an embodiment, the search area for template matching may be limited to a region 2752 of the original prediction block 2754 extending by 1 pixel along each of the top, bottom, left, and right sides.

[0422] Here, when the horizontal width of the current block 2740 is referred to as W and its vertical height as H, and a separable 2D T-tap filter is used as an interpolation filter, the total number of reconstructed reference samples 2750 read from the reconstructed reference frame for inter-frame prediction of the current block 2740 is (W+2+T-1)×(H+2+T-1). Therefore, the average number of reference samples read from the reconstructed reference frame stored in memory to predict the current sample (i.e., memory bandwidth) can be expressed as Equation 6.

[0423] [Equation 6]

[0424]

[0425] Here, N represents the number of reference blocks that will be referenced for inter-frame prediction of the current block. N is 1 during unidirectional prediction and 2 during bidirectional prediction.

[0426] Figure 27c This illustrates the memory bandwidth required for inter-frame prediction of the current block in DMVR mode according to various embodiments. Figure 27c Showing according to Figure 27a and Figure 27b In the embodiment, when the horizontal width of the current block is W, its vertical height is H, a separable 2D T-tap filter is used as an interpolation filter, and the search region for template matching is a region extending by 1 pixel along the top, bottom, left, and right sides of the original prediction block, the memory bandwidth is calculated according to Equation 6. Various examples of case 2770 for a luminance block sampled at 4:2:0 and case 2780 for a chrominance block are provided.

[0427] According to various embodiments, the image decoding device 100 can determine the size of the filter such that the memory bandwidth calculated according to Equation 6 does not exceed a specific reference value. According to embodiments, the specific reference value can be determined based on the aforementioned HEVC limit.

[0428] In the case 2770 of the luma block, according to an embodiment where the specific reference value is set to the HEVC limit, when the current block size is 8×8, the filter size can be determined to be 4 taps or less in bidirectional prediction, and the filter size can be determined to be 8 taps or less in unidirectional prediction. According to an embodiment where the specific reference value is set to the HEVC limit, when the current block size is 4×4, the filter size can be determined to be 2 taps or less in bidirectional prediction, and the filter size can be determined to be 4 taps or less in unidirectional prediction. According to another embodiment, an 8-tap filter is allowed in bidirectional prediction of an 8×8 block.

[0429] In the case of chroma blocks 2780, because the number of samples included in the chroma block is 1 / 4 of the number of samples in a luma block of the same size when the chroma block is sampled at a 4:2:0 ratio, the memory bandwidth is equal to the value obtained by dividing the result of Equation 6 by 4. According to an embodiment where the specific reference value is set as the HEVC limit, the filter size can be determined to be 4 taps or less when the current block size is 2×2. According to an embodiment, an 8-tap filter is allowed in unidirectional prediction when the current block size is 2×2.

[0430] Figure 28a The reference samples required for inter-frame prediction of the current block in DMVR mode using multiple motion vector candidates are shown according to various embodiments.

[0431] As described above, according to various embodiments, the number of motion vector candidates used to search for updated prediction blocks in the reference frame during template matching in DMVR mode can be limited. According to an embodiment, the image decoding device 100 can determine up to M motion vector candidates, and in this case, M prediction block candidates indicated by the motion vector candidates can be determined (shown in dashed lines).

[0432] The horizontal width of the current block 2810 is denoted as W, and its vertical height is denoted as H. It is assumed that a separable 2D T-tap filter is used as the interpolation filter. In this case, the number of reconstructed reference samples 2820, 2821, and 2830 required to generate reference samples for each of the prediction block candidates is as follows: Figure 22a The value is (W+T-1)×(H+T-1). Therefore, the maximum number of reconstruction reference samples 2820, 2821, and 2830 read from the reconstruction reference frame for inter-frame prediction of the current block 2810 can be indicated as M×(W+T-1)×(H+T-1). As a result, the average number of reference samples read from the reconstruction reference frame stored in memory to predict the current sample (i.e., memory bandwidth) can be expressed as Equation 7.

[0433] [Equation 7]

[0434]

[0435] Here, N represents the number of reference blocks that will be referenced for inter-frame prediction of the current block. N is 1 during unidirectional prediction and 2 during bidirectional prediction.

[0436] According to Equation 7, the memory bandwidth increases with the number of motion vector candidates, and also with the size of the interpolation filter (i.e., the number of taps T). Therefore, to keep the memory bandwidth at a uniform level, it may be necessary to reduce the number of filter taps as the number of motion vector candidates increases.

[0437] Figure 28b The memory bandwidth required for inter-frame prediction of the current block in DMVR mode using multiple motion vector candidates is shown according to various embodiments. Figure 28b Showing according to Figure 27a and Figure 27b The memory bandwidth calculated according to Equation 7 is given in the embodiment where the horizontal width of the current block is W, its vertical height is H, a separable 2D T-tap filter is used as an interpolation filter, and the number of motion vector candidates during template matching is limited to a maximum of M. Table 2850 provides various examples of the case for luma blocks.

[0438] According to various embodiments, the image decoding device 100 can determine the size of the filter such that the memory bandwidth calculated according to Equation 7 does not exceed a specific reference value. According to embodiments, the specific reference value can be determined based on the aforementioned HEVC limit.

[0439] When the number of motion vector candidates is limited to a maximum of three, according to an embodiment where the specific reference value is set as the HEVC limit, the filter size can be determined to be two taps or less during unidirectional prediction when the current block size is 8×8 or smaller. However, during bidirectional prediction, even when using a two-tap filter, the memory bandwidth exceeds the HEVC limit. Therefore, according to an embodiment where the specific reference value is set as the HEVC limit, DMVR modes using multiple motion vector candidates can be determined not to be used in bidirectional prediction. According to another embodiment, a two-tap filter is allowed in bidirectional prediction when the current block size is 8×8 or smaller.

[0440] When the number of motion vector candidates is limited to a maximum of four, according to an embodiment where the specific reference value is set as the HEVC limit, the filter size can be determined to be two taps or less during unidirectional prediction when the current block size is 8×8 or smaller. However, during bidirectional prediction, even when using a two-tap filter, the memory bandwidth exceeds the HEVC limit. Therefore, according to an embodiment where the specific reference value is set as the HEVC limit, DMVR modes using multiple motion vector candidates can be determined not to be used in bidirectional prediction. According to another embodiment, a two-tap filter is allowed in bidirectional prediction when the current block size is 8×8 or smaller.

[0441] When the number of motion vector candidates is 5 or more, according to an embodiment, when the current block size is 4×4, the memory bandwidth exceeds the HEVC limit even when using a 2-tap filter in unidirectional prediction; therefore, the DMVR mode can be determined not to be used for 4×4 blocks. According to another embodiment, when 5 or more motion vector candidates are allowed, a 2-tap filter can be determined to be used for 4×4 blocks.

[0442] Figure 28c The reference samples required for inter-frame prediction of the current block in a search-range-limited DMVR mode, according to various embodiments, are shown.

[0443] As described above, according to various embodiments, the search area used to search for updated prediction blocks in the reference frame during template matching in DMVR mode can be limited. According to an embodiment, the search area for template matching can be limited to a region 2862 of the original prediction block 2864 extending L pixels along each of the top, bottom, left, and right sides.

[0444] Here, when the horizontal width of the current block 2810 is referred to as W and its vertical height as H, and a separable 2D T-tap filter is used as an interpolation filter, the total number of reconstructed reference samples 2860 read from the reconstructed reference frame for inter-frame prediction of the current block 2810 is (W+2L+T-1)×(H+2L+T-1). Therefore, the average number of reference samples read from the reconstructed reference frame stored in memory to predict the current sample (i.e., memory bandwidth) can be expressed as Equation 8.

[0445] [Equation 8]

[0446]

[0447] Here, N represents the number of reference blocks that will be referenced for inter-frame prediction of the current block. N is 1 during unidirectional prediction and 2 during bidirectional prediction.

[0448] According to Equation 8, the memory bandwidth increases with the search region of the prediction block used for searching updates, and also with the size of the interpolation filter (i.e., the number of taps T). Therefore, to keep the memory bandwidth at a uniform level, it may be necessary to reduce the number of filter taps as the search region increases.

[0449] Figure 28d This illustrates the memory bandwidth required for inter-frame prediction of the current block in a search-range-limited DMVR mode, according to various embodiments. Figure 28d Showing according to Figure 27a and Figure 28c The memory bandwidth calculated according to Equation 7 is as follows: when the horizontal width of the current block is W, its vertical height is H, a separable 2D T-tap filter is used as an interpolation filter, and the search region is limited to the region of the original prediction block extending L pixels along the top, bottom, left and right sides during template matching.

[0450] Table 2870 provides various examples of the case for luma blocks. Although the case for chroma blocks is not shown, since the number of samples included in the chroma block when the chroma block is sampled at a 4:2:0 ratio is 1 / 4 of the number of samples in the luma block of the same size, the memory bandwidth is equal to the value obtained by dividing the result of Equation 8 by 4.

[0451] When the search region is limited to areas extending 2 pixels along the top, bottom, left, and right sides of the original prediction block, according to an embodiment where a specific reference value is set as the HEVC limit, when the current block size is 4×4, the filter size can be determined to be 4 taps or less during unidirectional prediction, and 2 taps or less during bidirectional prediction. According to an embodiment where the specific reference value is set as the HEVC limit, when the current block size is 8×8, the filter size can be determined to be 8 taps or less in unidirectional prediction, and 4 taps or less in bidirectional prediction.

[0452] When the search region is limited to an area extending 3 pixels along the top, bottom, left, and right sides of the original prediction block, according to an embodiment that sets the specific reference value as the HEVC limit, the filter size can be determined to be 2 taps or less during unidirectional prediction when the current block size is 4×4. According to an embodiment, when the current block size is 8×8, the filter size can be determined to be 4 taps or less during unidirectional prediction. According to another embodiment, a 4-tap filter is allowed in unidirectional prediction of a 4×4 block.

[0453] However, during bidirectional prediction, even when using a 2-tap filter, the memory bandwidth exceeds the HEVC limit. Therefore, according to an embodiment where the specific reference value is set to the HEVC limit, a DMVR mode where the search region is a region extending 3 or more pixels along the top, bottom, left, and right sides of the original prediction block can be determined not to be used in bidirectional prediction. According to another embodiment, in a DMVR mode where the search region is a region extending 3 or more pixels along the top, bottom, left, and right sides of the original prediction block, a 2-tap filter can be allowed during bidirectional prediction.

[0454] When the search region is an area that extends 4 or more pixels along the top, bottom, left, and right sides of the original prediction block, according to an embodiment that sets the specific reference value as the HEVC limit, the filter size can be determined to be 2 taps or less during unidirectional prediction. According to another embodiment, a 4-tap filter is allowed in unidirectional prediction of an 8×8 block.

[0455] Figure 29a The prediction patterns of affine patterns according to various embodiments are shown.

[0456] The four vertices 2902, 2904, 2906, and 2908 of the current block 2900 correspond to motion vectors 2912, 2914, 2916, and 2918, respectively. When the current block 2900 is affinely transformed using motion vectors 2912, 2914, 2916, and 2918, an affine transformation block 2910 is generated. Samples located at the current block 2900 can be matched with samples of the affine transformation block 2910.

[0457] For example, sample 2924, obtained by performing an affine transformation on sample 2922, which lies on the line connecting vertex 2906 to sample 2920 located at the upper center of block 2900, lies on the line connecting sample 2926 in affine transformation block 2910, indicated by motion vector 2916 of vertex 2906, and sample 2928 in affine transformation block 2910, indicated by motion vector of sample 2920. The position of affine transformation sample 2924 can be determined by motion vector 2930, obtained by performing linear interpolation on motion vectors 2912, 2914, 2916, and 2918 based on the position of sample 2922. Similarly, other samples of the current block 2900 can be affine transformed to match the samples of affine transformation block 2910. See reference... Figure 29a As mentioned above, inter-frame prediction of all samples in a block can be performed using motion vectors generated for affine transformation.

[0458] Figure 29b This illustrates the memory bandwidth required for inter-frame prediction of the current block in affine mode according to various embodiments.

[0459] For reference Figure 29a As stated above, because affine transformation samples are generated using the motion vector of the current block, they can be compared with the reference... Figure 22a The same method is used to indicate the maximum number of reconstructed reference samples required to generate the affine transform samples of the affine transform block. Therefore, the average number of reference samples read from at least one reconstructed reference frame stored in memory (i.e., memory bandwidth) is the same as in Equation 2.

[0460] Figure 29b Table 2940 shows the results according to Figure 29a and Figure 22a The memory bandwidth is calculated according to Equations 1 and 2 when the horizontal width is referred to as W and the vertical height as H, and a separable 2D T-tap filter is used as an interpolation filter in the embodiment. Various examples based on the number of taps are provided for the cases of 4×4 luma blocks sampled at 4:2:0 and 2×2 chroma blocks.

[0461] In the case of a luma block, according to an embodiment where a specific reference value is set as the HEVC limit, when the current block size is 4×4, the filter size can be determined to be 4 taps or less in unidirectional and bidirectional prediction. According to another embodiment, when the current block size is 4×4, an 8-tap filter is permissible in unidirectional prediction.

[0462] In the case of chroma blocks, because the number of samples included in the chroma block is 1 / 4 of the number of samples in a luma block of the same size when the chroma block is sampled at a 4:2:0 ratio, the memory bandwidth is equal to the value obtained by dividing the result of Equation 1 or Equation 2 by 4. According to an embodiment where the specific reference value is set to the HEVC limit, when the current chroma block size is 2×2, the filter size can be determined to be 8 taps or less in unidirectional prediction, and 4 taps or less in bidirectional prediction.

[0463] Figure 29c The reference samples required for inter-frame prediction of the current block in affine mode are shown according to various embodiments.

[0464] like Figure 29a As shown, the affine transformation block is determined based on the motion vector of the current block 2950. Because affine transformation includes not only translation and rotation but also scaling / shrinking, the size of the reference sample 2964 corresponding to the affine transformation block in the reference frame can be larger than the size of the current block 2950. Compared to the size of the current block 2950, ​​the size of the reference sample 2964 corresponding to the affine transformation block can indicate the scaling / shrinking ratio in the affine transformation. For example, in affine mode inter-frame prediction of a 4×4 current block, when the size of the reference sample 2964 corresponding to the affine transformation block is 8×8, the affine transformation block can be obtained by scaling the current block up to twice its size.

[0465] According to an embodiment, the maximum magnification / scaling ratio of the affine transformation can be limited to less than or equal to a specific value. According to an embodiment, the size of the reference sample 2964 corresponding to the affine transformation block can be limited to less than or equal to the size of the current block 2950, ​​which is extended by 2L pixels in both horizontal width and vertical height. According to an embodiment, the reference sample 2964 corresponding to the affine transformation block can be searched in a region where the block 2966, indicated by the motion vector of the current block 2950, ​​is extended by L pixels along the top, bottom, left, and right sides respectively. For example, when a maximum magnification / scaling of 1.5x is possible in the affine transformation during inter-frame prediction in affine mode for a 4×4-sized current block, L can be limited to 1 because the maximum size of the reference sample 2964 corresponding to the affine transformation block is 6×6.

[0466] According to an embodiment, a P-tap distortion filter may be additionally applied to reference samples of a reference image to generate each affine transform sample of an affine transform block.

[0467] Figure 29c The diagram illustrates that when the horizontal width of the current block 2950 is denoted as W, and its vertical height is denoted as H, the affine transformation block is determined in a region 2964 extending L pixels along the top, bottom, left, and right sides of block 2966 indicated by the motion vector of the current block 2950, ​​a separable 2D T-tap filter is used as an interpolation filter, and a P-tap warping filter is additionally applied, the reconstructed reference samples 2960 for inter-frame prediction of the current block 2950 are read from the reconstructed reference frame. In this case, the total number of reconstructed reference samples 2960 is (W+2L+P-1+T-1)×(H+2L+P-1+T-1). Therefore, the average number of reference samples (i.e., memory bandwidth) read from the reconstructed reference frame stored in memory to predict the current sample can be expressed as Equation 9.

[0468] [Equation 9]

[0469]

[0470] Here, N represents the number of reference blocks that will be referenced for inter-frame prediction of the current block. N is 1 during unidirectional prediction and 2 during bidirectional prediction.

[0471] Figure 29d This illustrates the memory bandwidth required for inter-frame prediction of the current block in affine mode according to various embodiments. Figure 29d The memory bandwidth is shown in Equation 9 when the horizontal width of the current block is called W, its vertical height is called H, the affine transformation block is determined in the region 2964 of the block 2966 indicated by the motion vector of the current block, which is extended by L pixels along the top, bottom, left and right sides respectively, a separable 2D T-tap filter is used as an interpolation filter, and a P-tap distortion filter is applied in addition.

[0472] Table 2970 provides various examples of the case for luma blocks. Although the case for chroma blocks is not shown, since the number of samples included in the chroma block when the chroma block is sampled at a 4:2:0 ratio is 1 / 4 of the number of samples in the luma block of the same size, the memory bandwidth is equal to the value obtained by dividing the result of Equation 9 by 4.

[0473] According to an embodiment that sets a specific reference value as the HEVC limit, when the current block size is 8×8, bidirectional prediction can use the following affine mode: in this affine mode, it is possible to use a 5-tap twist filter and up to 1.5x amplification (i.e., L=1). Alternatively, it is possible to use the following affine mode: in this affine mode, it is possible to use a 3-tap twist filter and up to 2x amplification (i.e., L=2).

[0474] According to an embodiment where the specific reference value is set as the HEVC limit, when the current block size is 4×4, unidirectional prediction can use the following affine mode: in this affine mode, it is possible to use a 5-tap twisted filter and up to 1.5x amplification (i.e., L = 1), or in this affine mode, it is possible to use a 3-tap twisted filter and up to 2x amplification (i.e., L = 2). However, in the illustrated combination of bidirectional prediction, since all memory bandwidth exceeds the HEVC limit, it is determined that the affine mode utilizing the twisted filter will not be used in bidirectional prediction. According to another embodiment, when the current block size is 4×4, bidirectional prediction can use the following affine mode: in this affine mode, it is possible to use a 3-tap twisted filter and up to 1.25x amplification (i.e., L = 0.5).

[0475] Figure 30 The prediction patterns of BIO patterns according to various embodiments are shown.

[0476] In BIO mode, the motion vector of the current block can be adjusted for each sample point 3015 in the current frame. Figure 30 This demonstrates a method for adjusting motion vectors in bidirectional prediction.

[0477] The reference block corresponding to the current block can be determined using the motion vectors (MVx0, MVy0) and (MVx1, MVy1) of the current block. Furthermore, reference samples 3022 and 3032 corresponding to the current sample 3015 can be determined within the reference block. The motion vector (v) of the current sample 3015 is estimated based on the spatial / temporal changes in the values ​​of reference samples 3022 and 3032. x ,v y The final predicted samples 3026 and 3036 are obtained by using the motion vector of the current block and the motion vector of the current sample.

[0478] The spatial change in the values ​​of reference samples 3022 and 3032 can be calculated based on the difference between the values ​​of reference samples 3022 and 3032 and the values ​​of samples adjacent to reference samples 3022 and 3032. According to an embodiment, an interpolation filter can be applied to samples at integer pixel positions adjacent to the reference samples to obtain the values ​​of samples at sub-pixel positions adjacent to the reference samples. According to an embodiment, the interpolation filtering used to calculate the spatial change in the values ​​of the reference samples and the interpolation filtering used for motion compensation can be performed simultaneously.

[0479] When the size of the interpolation filter used to calculate the spatial change in the values ​​of the reference samples is no larger than the size of the interpolation filter used for motion compensation, the number of reconstructed reference samples required to generate all reference samples in the reference block in BIO mode can be comparable to the reference sample size. Figure 22a and Figure 22b The number of descriptions is the same. Therefore, in this case, the BIO mode does not require additional memory bandwidth, and the required memory bandwidth can be calculated according to Equation 2.

[0480] The memory bandwidth required for inter-frame prediction in BIO mode can be referenced. Figure 23 However, in this case, the number of taps T of the filter can represent the larger of the size of the interpolation filter used to calculate the spatial change in the value of the reference sample and the size of the interpolation filter used for motion compensation.

[0481] Based on the above references Figures 21a to 30 In the described embodiments, image decoding device 100 determines filter information, but the embodiments are not limited thereto. According to various embodiments, filter information based on the inter-frame prediction mode of the current block, the size of the current block, whether the current block is a luma block or a chroma block, and whether the current block is unidirectionally or bidirectionally predicted can be predetermined and stored in the memories 120 and 420 of image encoding device 400 and image decoding device 100, respectively. According to various embodiments, image encoding device 400 can determine filter information based on the inter-frame prediction mode of the current block, the size of the current block, whether the current block is a luma block or a chroma block, and whether the current block is unidirectionally or bidirectionally predicted, and send the filter information to image decoding device 100 by including the filter information in a bitstream. Image decoding device 100 can extract the filter information from the bitstream. According to various embodiments, both image encoding device 400 and image decoding device 100 can determine filter information based on the inter-frame prediction mode of the current block, the size of the current block, whether the current block is a luma block or a chroma block, and whether the current block is unidirectionally or bidirectionally predicted.

[0482] According to Figure 1 to Figure 30The described video coding technique, based on tree-structured coding units, encodes image data in the spatial domain for each coding unit within the tree structure. When decoding is performed for each maximum coding unit according to the tree-structured coding unit video decoding technique, the image data in the spatial domain is reconstructed. Therefore, images and video as a sequence of images can be reconstructed. The reconstructed video can be reproduced by a playback device, stored in a storage medium, or transmitted via a network.

[0483] Furthermore, the embodiments described above can be written as programs executable on a computer and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium.

[0484] While this disclosure has been described in conjunction with specific preferred embodiments, other inventive methods, substitutions, modifications, and variations thereof will be apparent to those skilled in the art from the foregoing description. In other words, the claims are intended to cover all such substitutions, modifications, and variations thereof. Therefore, everything described in this specification and the accompanying drawings should be interpreted in an illustrative and non-limiting sense.

Claims

1. A video decoding method, comprising: When the inter-frame prediction mode of the current block is affine mode: Determine a first interpolation filter among a plurality of interpolation filters that corresponds to the size of a sub-block included in the current block, wherein the number of filter coefficients of the first interpolation filter is eight; The first interpolation filter is applied to the first reference sample to generate the first predicted sample of the sub-block; Perform an inverse transformation on the current block to generate a first residual block; and The first reconstruction block of the current block is obtained using the first residual block and a first prediction block that includes the first prediction samples of the sub-blocks, and When the inter-frame prediction mode of the current block is the decoder-side motion vector correction DMVR mode: Determine a second interpolation filter from a plurality of interpolation filters that corresponds to the DMVR mode, wherein the number of filter coefficients of the second interpolation filter is two; The second interpolation filter is applied to the second reference sample to generate the second prediction sample; Perform an inverse transformation on the current block to generate a second residual block; and The second reconstruction block of the current block is obtained using the second residual block and the second prediction block that includes the second prediction sample.

2. A video decoding device, comprising: At least one processor; as well as memory, The memory stores at least one instruction configured to be executable by the at least one processor, and The at least one instruction is configured to, when executed, cause the at least one processor to perform the following operations: When the inter-frame prediction mode of the current block is affine mode: Determine a first interpolation filter among a plurality of interpolation filters that corresponds to the size of a sub-block included in the current block, wherein the number of filter coefficients of the first interpolation filter is eight; The first interpolation filter is applied to the first reference sample to generate the first predicted sample of the sub-block; Perform an inverse transformation on the current block to generate a first residual block; and The first reconstruction block of the current block is obtained using the first residual block and a first prediction block that includes the first prediction samples of the sub-blocks, and When the inter-frame prediction mode of the current block is the decoder-side motion vector correction DMVR mode: Determine a second interpolation filter from a plurality of interpolation filters that corresponds to the DMVR mode, wherein the number of filter coefficients of the second interpolation filter is two; The second interpolation filter is applied to the second reference sample to generate the second prediction sample; Perform an inverse transformation on the current block to generate a second residual block; and The second reconstruction block of the current block is obtained using the second residual block and the second prediction block including the second prediction sample.

3. A video encoding method, comprising: When the inter-frame prediction mode of the current block is affine mode: Determine a first interpolation filter among a plurality of interpolation filters that corresponds to the size of a sub-block included in the current block, wherein the number of filter coefficients of the first interpolation filter is eight; The first interpolation filter is applied to the first reference sample to generate the first predicted sample of the sub-block; The first residual block of the current block is obtained by using the first predicted sample points of the sub-block; Perform a transformation on the first residual block of the current block to generate the first transformation coefficients of the current block; and The first transform coefficients of the current block are encoded into a first bit stream, and When the inter-frame prediction mode of the current block is the decoder-side motion vector correction DMVR mode: Determine a second interpolation filter from a plurality of interpolation filters that corresponds to the DMVR mode, wherein the number of filter coefficients of the second interpolation filter is two; The second interpolation filter is applied to the second reference sample to generate the second predicted sample; and The second residual block of the current block is obtained using the second prediction sample; Perform a transformation on the second residual block of the current block to generate the second transformation coefficients of the current block; and The second transform coefficients of the current block are encoded into a second bit stream.

4. A method for transmitting a bit stream, the method comprising: Perform the video encoding method according to claim 3 to generate the bitstream; as well as Transmit the bit stream.

Images