Image encoding / decoding method and device, and recording medium storing bitstream

Abstract

An image encoding method is provided. The image encoding method comprises the steps of: determining a first division structure of a current block; determining a second division structure of the current block on the basis of the first division structure; and encoding the second division structure of the current block, wherein an initial division structure for determining the second division structure can be derived from the first division structure.

Description

Video encoding method, device and recording medium storing bitstream

[0001] The present invention relates to a video encoding method, a device, and a recording medium storing a bitstream. Specifically, the present invention relates to a video encoding method based on a multi-pass coding method.

[0002] Recently, the demand for high-resolution, high-quality images, such as UHD (Ultra High Definition) images, is increasing across various application fields. As image data becomes higher in resolution and quality, the relative amount of data increases compared to conventional image data. Therefore, transmitting image data using existing media such as wired or wireless broadband lines or storing it using existing storage media leads to increased transmission and storage costs. To address these issues arising from the increasing resolution and quality of image data, high-efficiency image encoding / decoding technologies for higher-resolution and higher-quality images are required.

[0003] Existing multi-pass coding methods have limitations such as high complexity of the coding process and long coding times.

[0004] The purpose of the present invention is to provide a video encoding method and device with improved encoding efficiency.

[0005] In addition, the present invention aims to provide a recording medium storing a bitstream generated by an image encoding method or device according to the present invention.

[0006] In addition, the present invention aims to provide an improved multi-pass coding method to solve the problems of the existing multi-pass coding as described above.

[0007] A video encoding method according to one embodiment of the present invention includes a step of determining a first division structure of a current block, a step of determining a second division structure of the current block based on the first division structure, and a step of encoding the second division structure of the current block, wherein an initial division structure for determining the second division structure is derived from the first division structure.

[0008] In the above image encoding method, the first division structure includes a photo tree structure, a binary tree structure, and a ternary tree structure, and the initial division structure may be a photo tree structure of the first division structure.

[0009] In the above image encoding method, the initial segmentation structure may be a segmentation structure of a predefined segmentation depth of the first segmentation structure.

[0010] In the above image encoding method, the initial segmentation structure may be a segmentation structure composed of an area larger than a size defined in the first segmentation structure.

[0011] In the above image encoding method, the step of determining the second division structure of the current block based on the first division structure includes the step of determining whether to merge a plurality of sub-blocks divided according to the first division structure, and when it is determined that the plurality of sub-blocks divided according to the first division structure are merged, the initial division structure can be derived as a structure in which a plurality of sub-blocks divided according to the first division structure are merged.

[0012] In the above image encoding method, the step of determining whether to merge a plurality of sub-blocks divided according to the first division structure can be performed in an area within a predefined size.

[0013] In the above image encoding method, the step of determining whether to merge a plurality of sub-blocks divided according to the first division structure can be performed in a division structure of a predefined division depth.

[0014] In the above video encoding method, the step of determining the second division structure of the current block based on the first division structure includes the step of determining whether to further divide a plurality of sub-blocks divided according to the first division structure, and when it is determined that the plurality of sub-blocks divided according to the first division structure are further divided, the initial division structure can be derived as a structure in which the plurality of sub-blocks divided according to the first division structure are additionally divided.

[0015] In the above image encoding method, the step of determining whether to further divide a plurality of sub-blocks divided according to the first division structure can be performed in an area within a predefined size.

[0016] In the above image encoding method, the step of determining whether to further divide a plurality of sub-blocks divided according to the first division structure can be performed in a division structure of a predefined division depth.

[0017] In the above image encoding method, the second division structure can be determined by performing at least one of a ternary division method and a binary division method from the initial division structure.

[0018] A non-transitory computer-readable recording medium according to one embodiment of the present invention can store a bitstream generated by the image encoding method.

[0019] A bitstream transmission method according to one embodiment of the present invention can transmit a bitstream generated by the image encoding method.

[0020] A video encoding method according to one embodiment of the present invention includes the steps of performing a first encoding pass for encoding a current picture, performing a second encoding pass for encoding the current picture based on a frequency of at least one piece of encoding information determined in the first encoding pass, and generating a bitstream based on the at least one piece of encoding information determined in the second encoding pass, wherein the at least one piece of encoding information determined in the first encoding pass may be encoded at a higher level in the second encoding pass based on a frequency of occurrence.

[0021] A non-transitory computer-readable recording medium according to one embodiment of the present invention can store a bitstream generated by the image encoding method.

[0022] A bitstream transmission method according to one embodiment of the present invention can transmit a bitstream generated by the image encoding method.

[0023] A video encoding method according to one embodiment of the present invention may include the steps of performing a first encoding pass for encoding an original video, performing a second encoding pass for encoding a left-right symmetrical video of the original video, performing a third encoding pass for encoding a top-to-bottom symmetrical video of the original video, performing a fourth encoding pass for encoding top-to-bottom symmetrical and left-to-right symmetrical videos of the original video, and generating a bitstream using an encoding pass having a lowest encoding cost among a first encoding cost of the first encoding pass, a second encoding cost of the second encoding pass, a third encoding cost of the third encoding pass, and a fourth encoding cost of the fourth encoding pass.

[0024] In the above image encoding method, the original image may be any one of a picture, a slice, a tile, and a coding tree unit.

[0025] A non-transitory computer-readable recording medium according to one embodiment of the present invention can store a bitstream generated by the image encoding method.

[0026] A bitstream transmission method according to one embodiment of the present invention can transmit a bitstream generated by the image encoding method.

[0027] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0028] According to the present invention, a video encoding method and device with improved encoding efficiency can be provided.

[0029] In addition, according to the present invention, the efficiency of a multi-pass coding method can be improved.

[0030] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0031] Figure 1 is a block diagram showing the configuration according to one embodiment of an encoding device to which the present invention is applied.

[0032] Figure 2 is a block diagram showing the configuration according to one embodiment of a decryption device to which the present invention is applied.

[0033] FIG. 3 is a diagram schematically showing a video coding system to which the present invention can be applied.

[0034] FIG. 4 is a diagram for explaining a coding tree unit (CTU) division structure and a coding unit (CU) division structure determination method according to one embodiment of the present invention.

[0035] FIG. 5 is a diagram for explaining a method for determining a segmentation structure using photo tree-based multi-pass coding according to one embodiment of the present invention.

[0036] FIG. 6 is a diagram for explaining a method for determining a segmentation structure using depth condition-based multi-pass coding according to one embodiment of the present invention.

[0037] FIG. 7 is a diagram for explaining a method for determining a segmentation structure using size condition-based multi-pass coding according to one embodiment of the present invention.

[0038] FIG. 8 is a diagram for explaining a method for determining a partition structure using merge-based multi-pass coding according to one embodiment of the present invention.

[0039] FIG. 9 is a diagram for explaining a method for determining a partition structure using additional partition-based multi-pass coding according to one embodiment of the present invention.

[0040] FIG. 10 is a diagram for explaining a multi-type tree splitting method in a second coding pass according to one embodiment of the present invention.

[0041] Figure 11 is a flowchart illustrating an image encoding method according to one embodiment of the present invention.

[0042] Figure 12 is a flowchart illustrating an image encoding method according to one embodiment of the present invention.

[0043] Figure 13 is a flowchart illustrating an image encoding method according to one embodiment of the present invention.

[0044] FIG. 14 is a drawing exemplifying a content streaming system to which an embodiment according to the present invention can be applied.

[0045] A video encoding method according to one embodiment of the present invention includes a step of determining a first division structure of a current block, a step of determining a second division structure of the current block based on the first division structure, and a step of encoding the second division structure of the current block, wherein an initial division structure for determining the second division structure is derived from the first division structure.

[0046] The present invention is susceptible to various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and substitutes falling within the spirit and scope of the present invention. In the drawings, similar reference numerals designate the same or similar functions throughout. The shape and size of elements in the drawings may be provided by way of example only for clarity. The detailed description of the exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments, while different from each other, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention. Furthermore, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be modified without departing from the spirit and scope of the embodiment. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the illustrative embodiments, if properly described, is defined only by the appended claims, along with the full scope equivalents to which such claims are entitled.

[0047] In the present invention, terms such as first, second, etc. may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term "and / or" includes a combination of multiple related described items or any of multiple related described items.

[0048] The components shown in the embodiments of the present invention are independently depicted to represent different characteristic functions, and do not mean that each component is composed of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two components among each component may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separate embodiments of each component are also included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

[0049] The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In addition, some components of the present invention are not essential components that perform essential functions in the present invention and may be optional components merely for performance enhancement. The present invention may be implemented by including only components essential to realizing the essence of the present invention, excluding components used only for performance enhancement, and a structure including only essential components, excluding optional components used only for performance enhancement, is also within the scope of the present invention.

[0050] In embodiments, the term "at least one" may mean one of a number greater than or equal to 1, such as 1, 2, 3, and 4. In embodiments, the term "a plurality of" may mean one of a number greater than or equal to 2, such as 2, 3, and 4.

[0051] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of a related known configuration or function may obscure the gist of this specification, the detailed description will be omitted. The same reference numerals will be used for identical components in the drawings, and duplicate descriptions of identical components will be omitted.

[0052] Glossary of Terms

[0053] Hereinafter, “video” may mean a single picture constituting a video, or may refer to the video itself. For example, “encoding and / or decoding of a video” may mean “encoding and / or decoding of a video,” or may mean “encoding and / or decoding of one of the videos constituting the video.”

[0054] Hereinafter, the terms "video" and "movie" may be used interchangeably and have the same meaning. Furthermore, the target image may be an encoding target image, which is the target of encoding, and / or a decoding target image, which is the target of decoding. Furthermore, the target image may be an input image input to an encoding device, or an input image input to a decoding device. Here, the target image may have the same meaning as the current image.

[0055] Hereinafter, the terms encoder and image encoding device may be used interchangeably and have the same meaning.

[0056] Hereinafter, the terms decoder and image decoding device may be used interchangeably and have the same meaning.

[0057] Hereinafter, “image”, “picture”, “frame” and “screen” may be used with the same meaning and may be used interchangeably.

[0058] Hereinafter, the term "target block" may refer to an encoding target block, which is the target of encoding, and / or a decoding target block, which is the target of decoding. Furthermore, the target block may refer to a current block, which is the target of current encoding and / or decoding. For example, the terms "target block" and "current block" may be used interchangeably and have the same meaning.

[0059] Hereinafter, "block" and "unit" may be used with the same meaning and may be used interchangeably. In addition, "unit" may mean including a luminance component block and a corresponding chroma component block to distinguish it from a block. For example, a coding tree unit (CTU) may be composed of one luma component (Y) coding tree block (CTB) and two chroma component (Cb, Cr) coding tree blocks associated with it.

[0060] Hereinafter, the terms “sample,” “pixel,” and “pixel” may be used interchangeably and have the same meaning. Here, a sample may represent a basic unit that constitutes a block.

[0061] Hereinafter, “inter” and “between screens” may be used interchangeably and have the same meaning.

[0062] Hereinafter, “intra” and “within screen” may be used interchangeably and have the same meaning.

[0063]

[0064] Figure 1 is a block diagram showing the configuration according to one embodiment of an encoding device to which the present invention is applied.

[0065] The encoding device (100) may be an encoder, a video encoding device, or an image encoding device. A video may include one or more images. The encoding device (100) may sequentially encode one or more images.

[0066] Referring to FIG. 1, the encoding device (100) may include an image segmentation unit (110), an intra prediction unit (120), a motion prediction unit (121), a motion compensation unit (122), a switch (115), a subtractor (113), a transformation unit (130), a quantization unit (140), an entropy encoding unit (150), an inverse quantization unit (160), an inverse transformation unit (170), an adder (117), a filter unit (180), and a reference picture buffer (190).

[0067] Additionally, the encoding device (100) can generate a bitstream including encoded information through encoding an input image and output the generated bitstream. The generated bitstream can be stored in a computer-readable recording medium or can be streamed via a wired / wireless transmission medium.

[0068] The video segmentation unit (110) can segment the input video into various forms to increase the efficiency of video encoding / decoding. That is, the input video is composed of multiple pictures, and one picture can be hierarchically segmented and processed for compression efficiency, parallel processing, etc. For example, one picture can be segmented into one or more tiles or slices, which can then be segmented into multiple Coding Tree Units (CTUs). Alternatively, one picture can first be segmented into multiple sub-pictures defined as groups of rectangular slices, and each sub-picture can then be segmented into the tiles / slices. Here, the sub-pictures can be utilized to support the function of partially independently encoding / decoding and transmitting the picture. Since multiple sub-pictures can each be individually restored, there is an advantage of easy editing in applications that configure multi-channel input into a single picture. In addition, tiles can be segmented horizontally to generate bricks. Here, a brick can be utilized as the basic unit of intra-picture parallel processing. In addition, one CTU can be recursively split into a quadtree (QT), and the terminal node of the split can be defined as a coding unit (CU). The CU can be split into a prediction unit (PU) and a transformation unit (TU), and prediction and splitting can be performed. Meanwhile, the CU can be utilized as a prediction unit and / or a transformation unit itself. Here, for flexible splitting, each CTU can be recursively split into a multi-type tree (MTT) as well as a quadtree (QT). Splitting of a CTU into a multi-type tree can start from the terminal node of a QT, and the MTT can be composed of a binary tree (BT) and a triple tree (TT).For example, the MTT structure can be divided into vertical binary split mode (SPLIT_BT_VER), horizontal binary split mode (SPLIT_BT_HOR), vertical ternary split mode (SPLIT_TT_VER), and horizontal ternary split mode (SPLIT_TT_HOR). In addition, the minimum block size (MinQTSize) of the quad tree of the luminance block during splitting can be set to 16x16, the maximum block size (MaxBtSize) of the binary tree can be set to 128x128, and the maximum block size (MaxTtSize) of the triple tree can be set to 64x64. In addition, the minimum block size (MinBtSize) of the binary tree and the minimum block size (MinTtSize) of the triple tree can be set to 4x4, and the maximum depth (MaxMttDepth) of the multi-type tree can be set to 4. Additionally, to improve the encoding efficiency of the I slice, a dual tree can be applied that uses different CTU partition structures for luminance and chrominance components. On the other hand, in the P and B slices, the luminance and chrominance CTBs (Coding Tree Blocks) within the CTU can be partitioned into a single tree that shares the coding tree structure.

[0069] The encoding device (100) may perform encoding on the input image in intra mode and / or inter mode. Alternatively, the encoding device (100) may perform encoding on the input image in a third mode (e.g., IBC mode, Palette mode, etc.) other than the intra mode and inter mode. However, if the third mode has functional characteristics similar to the intra mode or inter mode, it may be classified as intra mode or inter mode for convenience of explanation. In the present invention, the third mode will be classified and described separately only when a specific description is required.

[0070] When the intra mode is used as the prediction mode, the switch (115) can be switched to intra, and when the inter mode is used as the prediction mode, the switch (115) can be switched to inter. Here, the intra mode can mean an intra-screen prediction mode, and the inter mode can mean an inter-screen prediction mode. The encoding device (100) can generate a prediction block for an input block of an input image. In addition, after the prediction block is generated, the encoding device (100) can encode a residual block using a residual of the input block and the prediction block. The input image can be referred to as a current image that is currently a target of encoding. The input block can be referred to as a current block that is currently a target of encoding or an encoding target block.

[0071] When the prediction mode is intra mode, the intra prediction unit (120) can use samples of blocks already encoded / decoded around the current block as reference samples. The intra prediction unit (120) can perform spatial prediction on the current block using the reference samples, and can generate prediction samples for the input block through spatial prediction. Here, intra prediction can mean prediction within the screen.

[0072] As an intra prediction method, non-directional prediction modes such as DC mode and Planar mode, as well as directional prediction modes (e.g., 65 directions) can be applied. Here, the intra prediction method can be expressed as an intra prediction mode or an intra-screen prediction mode.

[0073] When the prediction mode is inter mode, the motion prediction unit (121) can search for an area that best matches the input block from the reference image during the motion prediction process and derive a motion vector using the searched area. At this time, the area can be used as a search area. The reference image can be stored in the reference picture buffer (190). Here, when encoding / decoding for the reference image is processed, it can be stored in the reference picture buffer (190).

[0074] The motion compensation unit (122) can generate a prediction block for the current block by performing motion compensation using a motion vector. Here, inter prediction may mean inter-screen prediction or motion compensation.

[0075] The above motion prediction unit (121) and motion compensation unit (122) can generate a prediction block by applying an interpolation filter to a portion of an area within a reference image when the value of the motion vector does not have an integer value. In order to perform inter-screen prediction or motion compensation, it is possible to determine whether the motion prediction and motion compensation method of the prediction unit included in the corresponding encoding unit is one of Skip Mode, Merge Mode, Advanced Motion Vector Prediction (AMVP) mode, and Intra Block Copy (IBC) mode based on the encoding unit, and perform inter-screen prediction or motion compensation according to each mode.

[0076] In addition, based on the above inter-screen prediction method, the AFFINE mode of sub-PU based prediction, the SbTMVP (Subblock-based Temporal Motion Vector Prediction) mode, and the MMVD (Merge with MVD) mode and the GPM (Geometric Partitioning Mode) mode of PU based prediction can be applied. In addition, in order to improve the performance of each mode, the HMVP (History based MVP), the PAMVP (Pairwise Average MVP), the CIIP (Combined Intra / Inter Prediction), the AMVR (Adaptive Motion Vector Resolution), the BDOF (Bi-Directional Optical-Flow), the BCW (Bi-predictive with CU Weights), the LIC (Local Illumination Compensation), the TM (Template Matching), and the OBMC (Overlapped Block Motion Compensation) can be applied.

[0077] Among these, AFFINE mode is a technology that is used in both AMVP and MERGE modes and also has high encoding efficiency. In the existing video coding standard, since MC (Motion Compensation) is performed by considering only the parallel translation of the block, there was a disadvantage in that it could not properly compensate for motions that occur in reality, such as zoom in / out and rotation. To supplement this, a 4-parameter affine motion model using two control point motion vectors (CPMV) and a 6-parameter affine motion model using three control point motion vectors can be applied to inter prediction. Here, CPMV is a vector representing the affine motion model of one of the upper left, upper right, and lower left of the current block.

[0078] The subtractor (113) can generate a residual block using the difference between the input block and the predicted block. The residual block may also be referred to as a residual signal. The residual signal may refer to the difference between the original signal and the predicted signal. Alternatively, the residual signal may be a signal generated by transforming, quantizing, or transforming and quantizing the difference between the original signal and the predicted signal. The residual block may be a residual signal in block units.

[0079] The transform unit (130) can perform a transform on the residual block to generate a transform coefficient and output the generated transform coefficient. Here, the transform coefficient may be a coefficient value generated by performing a transform on the residual block. When the transform skip mode is applied, the transform unit (130) may also skip the transform on the residual block.

[0080] Quantized levels can be generated by applying quantization to transform coefficients or residual signals. In the following embodiments, quantized levels may also be referred to as transform coefficients.

[0081] For example, a 4x4 luminance residual block generated through within-screen prediction can be transformed using a basis vector based on DST (Discrete Sine Transform), and the remaining residual blocks can be transformed using a basis vector based on DCT (Discrete Cosine Transform). In addition, through RQT (Residual Quad Tree) technology, the transform block is divided into a quad tree shape for one block, and after performing transformation and quantization on each transform block divided through RQT, a coded block flag (cbf) can be transmitted to increase encoding efficiency when all coefficients become 0.

[0082] Another alternative is to apply Multiple Transform Selection (MTS) technology, which selectively performs transformation using multiple transformation bases. That is, instead of dividing CUs into TUs via RQT, a Sub-block Transform (SBT) technology can perform a function similar to TU division. Specifically, SBT is applied only to inter-screen prediction blocks, and unlike RQT, it can divide the current block into ½ or ¼ blocks vertically or horizontally, and then perform transformation on only one of the blocks. For example, in a vertically divided block, the transformation can be performed on the leftmost or rightmost block, and in a horizontally divided block, the transformation can be performed on the topmost or bottommost block.

[0083] Additionally, LFNST (Low Frequency Non-Separable Transform), a secondary transform technique that further transforms the residual signal converted to the frequency domain through DCT or DST, can be applied. LFNST additionally performs a transform on the low-frequency region of 4x4 or 8x8 in the upper left, which allows the residual coefficients to be concentrated in the upper left.

[0084] The quantization unit (140) can generate a quantized level by quantizing a transform coefficient or residual signal according to a quantization parameter (QP), and can output the generated quantized level. At this time, the quantization unit (140) can quantize the transform coefficient using a quantization matrix.

[0085] For example, a quantizer with QP values ​​of 0 to 51 can be used. Alternatively, if the image size is larger and high encoding efficiency is required, a QP of 0 to 63 can be used. In addition, a Dependent Quantization (DQ) method that uses two quantizers instead of a single quantizer can be applied. DQ performs quantization using two quantizers (e.g., Q0 and Q1), but even without signaling information about the use of a specific quantizer, the quantizer to be used for the next transform coefficient can be selected based on the current state through a state transition model.

[0086] The entropy encoding unit (150) can generate a bitstream by performing entropy encoding according to a probability distribution on values ​​produced by the quantization unit (140) or coding parameter values ​​produced during the encoding process, and can output the bitstream. The entropy encoding unit (150) can perform entropy encoding on information about image samples and information for decoding the image. For example, the information for decoding the image can include syntax elements, etc.

[0087] When entropy encoding is applied, a small number of bits are allocated to symbols with a high occurrence probability, and a large number of bits are allocated to symbols with a low occurrence probability, thereby representing the symbols, whereby the size of the bit string for the symbols to be encoded can be reduced. The entropy encoding unit (150) can use an encoding method such as exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), or Context-Adaptive Binary Arithmetic Coding (CABAC) for entropy encoding. For example, the entropy encoding unit (150) can perform entropy encoding using a Variable Length Coding / Code (VLC) table. In addition, the entropy encoding unit (150) may perform arithmetic encoding using the binarization method, probability model, and context model derived from the binarization method of the target symbol and the probability model of the target symbol / bin.

[0088] In this regard, when applying CABAC, the table probability update method can be changed to a simple formula-based table update method to reduce the size of the probability table stored in the decryption device. Furthermore, two different probability models can be used to obtain more accurate symbol probability values.

[0089] The entropy encoding unit (150) can change a two-dimensional block form coefficient into a one-dimensional vector form through a transform coefficient scanning method to encode a transform coefficient level (quantized level).

[0090] Coding parameters may include not only information (flags, indexes, etc.) encoded in an encoding device (100) and signaled to a decoding device (200), such as syntax elements, but also information derived during an encoding process or a decoding process, and may mean information necessary when encoding or decoding an image.

[0091] Here, signaling a flag or index may mean that the encoder entropy encodes the flag or index and includes it in the bitstream, and that the decoder entropy decodes the flag or index from the bitstream.

[0092] The encoded current image can be used as a reference image for other images to be processed later. Accordingly, the encoding device (100) can reconstruct or decode the encoded current image again and store the reconstructed or decoded image as a reference image in the reference picture buffer (190).

[0093] The quantized level can be dequantized in the dequantization unit (160) and inversely transformed in the inverse transformation unit (170). The dequantized and / or inversely transformed coefficients can be combined with a prediction block through an adder (117), and a reconstructed block can be generated by combining the dequantized and / or inversely transformed coefficients and the prediction block. Here, the dequantized and / or inversely transformed coefficients refer to coefficients on which at least one of dequantization and inverse transformation has been performed, and may refer to a reconstructed residual block. The dequantization unit (160) and the inverse transformation unit (170) can be performed in the reverse process of the quantization unit (140) and the transformation unit (130).

[0094] The restoration block may pass through a filter unit (180). The filter unit (180) may apply a deblocking filter, a sample adaptive offset (SAO), an adaptive loop filter (ALF), a bilateral filter (BIF), a Luma Mapping with Chroma Scaling (LMCS), etc. as a filtering technique, in whole or in part, to the restoration sample, restoration block, or restoration image. The filter unit (180) may also be referred to as an in-loop filter. In this case, the in-loop filter is also used as a name excluding LMCS.

[0095] A deblocking filter can remove block distortion that occurs at the boundaries between blocks. Whether to apply a deblocking filter to the current block can be determined based on the samples contained in several columns or rows within the block. When applying a deblocking filter to a block, different filters can be applied depending on the required deblocking filtering strength.

[0096] Sample adaptive offset can be used to compensate for encoding errors by adding an appropriate offset value to sample values. Sample adaptive offset can compensate for the offset from the original image on a sample-by-sample basis for deblocked images. This can be done by dividing the samples contained in the image into a fixed number of regions, determining the regions to be offset, and applying the offset to those regions. Alternatively, the offset can be applied by considering the edge information of each sample.

[0097] Bilateral filter (BIF) can also compensate for the offset from the original image on a sample-by-sample basis for the deblocked image.

[0098] An adaptive loop filter can perform filtering based on a comparison between a reconstructed image and the original image. By dividing the samples contained in the image into predetermined groups and determining the filter to be applied to each group, filtering can be performed differentially for each group. Information regarding whether to apply an adaptive loop filter can be signaled for each coding unit (CU), and the shape and filter coefficients of the adaptive loop filter applied to each block can vary.

[0099] In LMCS (Luma Mapping with Chroma Scaling), luma mapping (LM) refers to remapping luminance values ​​through a piece-wise linear model, and chroma scaling (CS) refers to a technique that scales the residual values ​​of chrominance components according to the average luminance value of the prediction signal. In particular, LMCS can be utilized as an HDR correction technique that reflects the characteristics of HDR (High Dynamic Range) images.

[0100] The restored block or restored image that has passed through the filter unit (180) may be stored in the reference picture buffer (190). The restored block that has passed through the filter unit (180) may be a part of the reference image. In other words, the reference image may be a restored image composed of restored blocks that have passed through the filter unit (180). The stored reference image may be used for inter-screen prediction or motion compensation thereafter.

[0101] Figure 2 is a block diagram showing the configuration according to one embodiment of a decryption device to which the present invention is applied.

[0102] The decoding device (200) may be a decoder, a video decoding device, or an image decoding device.

[0103] Referring to FIG. 2, the decoding device (200) may include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transformation unit (230), an intra prediction unit (240), a motion compensation unit (250), an adder (201), a switch (203), a filter unit (260), and a reference picture buffer (270).

[0104] The decoding device (200) can receive a bitstream output from the encoding device (100). The decoding device (200) can receive a bitstream stored in a computer-readable recording medium, or a bitstream streamed through a wired / wireless transmission medium. The decoding device (200) can perform decoding on the bitstream in intra mode or inter mode. In addition, the decoding device (200) can generate a restored image or a decoded image through decoding, and can output the restored image or the decoded image.

[0105] If the prediction mode used for decryption is intra mode, the switch (203) can be switched to intra. If the prediction mode used for decryption is inter mode, the switch (203) can be switched to inter.

[0106] The decoding device (200) can decode the input bitstream to obtain a reconstructed residual block and generate a prediction block. Once the reconstructed residual block and the prediction block are obtained, the decoding device (200) can generate a reconstructed block to be decoded by adding the reconstructed residual block and the prediction block. The block to be decoded may be referred to as a current block.

[0107] The entropy decoding unit (210) can generate symbols by performing entropy decoding according to a probability distribution for the bitstream. The generated symbols may include symbols in the form of quantized levels. Here, the entropy decoding method may be the reverse process of the entropy encoding method described above.

[0108] The entropy decoding unit (210) can change a one-dimensional vector-shaped coefficient into a two-dimensional block-shaped coefficient through a transform coefficient scanning method to decode a transform coefficient level (quantized level).

[0109] The quantized level can be inversely quantized in the inverse quantization unit (220) and inversely transformed in the inverse transformation unit (230). The quantized level can be generated as a restored residual block as a result of performing inverse quantization and / or inverse transformation. At this time, the inverse quantization unit (220) can apply a quantization matrix to the quantized level. The inverse quantization unit (220) and inverse transformation unit (230) applied to the decoding device can apply the same technology as the inverse quantization unit (160) and inverse transformation unit (170) applied to the encoding device described above.

[0110] When intra mode is used, the intra prediction unit (240) can generate a predicted block by performing spatial prediction on the current block using sample values ​​of already decoded blocks surrounding the block to be decoded. The intra prediction unit (240) applied to the decoding device can apply the same technology as the intra prediction unit (120) applied to the encoding device described above.

[0111] When the inter mode is used, the motion compensation unit (250) can generate a prediction block by performing motion compensation using a motion vector and a reference image stored in the reference picture buffer (270) on the current block. The motion compensation unit (250) can generate a prediction block by applying an interpolation filter to a portion of the reference image when the value of the motion vector does not have an integer value. In order to perform motion compensation, it is possible to determine whether the motion compensation method of the prediction unit included in the corresponding encoding unit is skip mode, merge mode, AMVP mode, or current picture reference mode based on the encoding unit, and motion compensation can be performed according to each mode. The motion compensation unit (250) applied to the decoding device can apply the same technology as the motion compensation unit (122) applied to the encoding device described above.

[0112] The adder (201) can add the restored residual block and the predicted block to generate a restored block. The filter unit (260) can apply at least one of an Inverse-LMCS, a deblocking filter, a sample adaptive offset, and an adaptive loop filter to the restored block or restored image. The filter unit (260) applied to the decoding device can apply the same filtering technology as that applied to the filter unit (180) applied to the encoding device described above.

[0113] The filter unit (260) can output a restored image. The restored block or restored image can be stored in the reference picture buffer (270) and used for inter prediction. The restored block that has passed through the filter unit (260) can be a part of the reference image. In other words, the reference image can be a restored image composed of restored blocks that have passed through the filter unit (260). The stored reference image can be used for inter-screen prediction or motion compensation thereafter.

[0114] FIG. 3 is a diagram schematically showing a video coding system to which the present invention can be applied.

[0115] A video coding system according to one embodiment may include an encoding device (10) and a decoding device (20). The encoding device (10) may transmit encoded video and / or image information or data to the decoding device (20) in the form of a file or streaming via a digital storage medium or a network.

[0116] An encoding device (10) according to one embodiment may include a video source generation unit (11), an encoding unit (12), and a transmission unit (13). A decoding device (20) according to one embodiment may include a reception unit (21), a decoding unit (22), and a rendering unit (23). The encoding unit (12) may be referred to as a video / image encoding unit, and the decoding unit (22) may be referred to as a video / image decoding unit. The transmission unit (13) may be included in the encoding unit (12). The reception unit (21) may be included in the decoding unit (22). The rendering unit (23) may include a display unit, and the display unit may be configured as a separate device or an external component.

[0117] The video source generation unit (11) can obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source generation unit (11) can include a video / image capture device and / or a video / image generation device. The video / image capture device can include, for example, one or more cameras, a video / image archive including previously captured video / images, etc. The video / image generation device can include, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate video / images. For example, a virtual video / image can be generated through a computer, etc., in which case the video / image capture process can be replaced with a process of generating related data.

[0118] The encoding unit (12) can encode the input video / image. The encoding unit (12) can perform a series of procedures such as prediction, transformation, and quantization for compression and encoding efficiency. The encoding unit (12) can output encoded data (encoded video / image information) in the form of a bitstream. The detailed configuration of the encoding unit (12) can also be configured in the same manner as the encoding device (100) of FIG. 1 described above.

[0119] The transmission unit (13) can transmit encoded video / image information or data output in the form of a bitstream to the reception unit (21) of the decoding device (20) via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) can include an element for generating a media file through a predetermined file format and can include an element for transmission via a broadcasting / communication network. The reception unit (21) can extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit (22).

[0120] The decoding unit (22) can decode video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding unit (12). The detailed configuration of the decoding unit (22) can also be configured in the same manner as the decoding device (200) of FIG. 2 described above.

[0121] The rendering unit (23) can render the decrypted video / image. The rendered video / image can be displayed through the display unit.

[0122]

[0123] Multi-pass coding is a method of efficiently coding images by performing coding two or more times, unlike single-pass coding, which codes images once.

[0124] Specifically, in the first coding pass, coding is performed based on rate-distortion cost optimization for the image. Then, in the second coding pass, optimized coding parameters are set based on information about the image obtained from the first coding pass, and coding is performed on the image. In this way, in multi-pass coding, coding in the (N+1)th coding pass is performed based on information about the image obtained in the coding performance of the Nth coding pass. Here, the Nth coding pass means the Nth step of performing coding in multi-pass coding, and N is an integer greater than or equal to 1.

[0125] The existing multi-pass coding method has the advantage of being able to precisely control the bit rate compared to the single-pass coding method, but has the disadvantage of having a complex coding process and taking a long coding time.

[0126] This specification proposes an improved multi-pass coding method to complement existing multi-pass coding. Unlike existing multi-pass coding, the improved multi-pass coding method simplifies the coding process, thereby reducing encoding complexity.

[0127] Meanwhile, in this specification, coding may have the same meaning as encoding. For example, performing coding in the first pass may mean performing encoding in the first pass.

[0128]

[0129] Before describing the improved multi-pass coding method, a method for determining a block division structure according to an embodiment of the present invention will be described with reference to FIG. 4.

[0130] FIG. 4 is a diagram for explaining a coding tree unit (CTU) division structure and a coding unit (CU) division structure determination method according to one embodiment of the present invention.

[0131] In this specification, a coding tree unit partition structure means a structure in which a picture is partitioned into coding tree units, and a coding unit partition structure means a structure in which a coding tree unit is partitioned into coding units.

[0132] In Fig. 4, the Mth picture (M th Picture, 400) means the Mth picture on which encoding is performed. Here, M is an integer greater than or equal to 1. And W picture (401) and H picture (402) represent the width and height of the picture, respectively.

[0133] The coding tree unit partitioning structure (CTU partitioning, 403) in Fig. 4 refers to a structure in which the M-th picture is partitioned into coding tree units. Referring to Fig. 4, the M-th picture can be partitioned into 12 coding tree units having the same width and height. In Fig. 4, the size of each coding tree unit is nxn, where n represents the number of pixels.

[0134] Additionally, each coding tree unit can be divided into coding units (CUs). The division of the coding tree units can be performed independently for each coding tree unit. Accordingly, the size and shape of the coding units within each coding tree unit can be determined in various ways.

[0135] In Fig. 4, the Kth coding tree unit (K th CTU, 404) refers to the Kth coding tree unit where encoding is performed. Here, K is an integer greater than or equal to 1.

[0136] The coding unit partitioning structure (CU partitioning, 405) in FIG. 4 represents a structure in which the Kth coding tree unit (404) is divided into coding units.

[0137] Referring to Fig. 4, the Kth coding tree unit can be partitioned using a quad tree (QT) partitioning method. The quad tree structure refers to a tree structure in which four subordinate nodes are connected to one upper node, and the quad tree partitioning method can refer to a method in image segmentation in which one upper block is partitioned into four subordinate blocks. In this case, the width and height of the subordinate blocks can be the same.

[0138] And, each sub-block divided by the photo tree division method can be divided by the photo tree division method or the multi-type tree division method. Here, the multi-type tree division method means the ternary tree (TT) division method and the binary tree (BT) division method.

[0139] A ternary tree structure refers to a tree structure in which three subordinate nodes are connected to one upper node, and a ternary tree partitioning method may refer to a method in image segmentation in which one upper block is partitioned into three subordinate blocks. In this case, the width or height ratio of the three subordinate blocks may be 1:2:1.

[0140] A binary tree structure refers to a tree structure in which two subordinate nodes are connected to one upper node, and a binary tree partitioning method can refer to a method in image partitioning in which one upper block is partitioned into two subordinate blocks. At this time, the width and height of the subordinate blocks can be the same.

[0141] Meanwhile, the photo tree partitioning method can be performed recursively within an allowable size. Specifically, if the current block is a sub-block partitioned from a parent block by the photo tree partitioning method, and the current block satisfies a partitioning condition, the current block can be partitioned into four sub-blocks by the photo tree partitioning method. The partitioning condition may refer to a partitionable depth and a partitionable size, among other factors.

[0142] Meanwhile, if a multi-type tree partitioning method is performed on an upper block and it is partitioned into sub-blocks, each sub-block can be partitioned only by the multi-type tree partitioning method and cannot be partitioned by the photo-tree partitioning method. At this time, the multi-type tree partitioning method can be performed recursively within an allowable size or an allowable depth. Specifically, if the current block is a sub-block partitioned by the multi-type tree partitioning method and the current block satisfies a partitioning condition, the current block can be partitioned into sub-blocks by the multi-type tree partitioning method. At this time, the partitioning condition may mean a condition that is satisfied when the current block is less than a predetermined partitioning depth, or a condition that is satisfied when the current block is greater than a predetermined size.

[0143] In Fig. 4, the Kth coding tree unit (404) can be divided into four sub-blocks of the same size by a photo tree division method.

[0144] In addition, each sub-block may not be split, or may undergo additional splitting to determine the coding unit splitting structure. Referring to Fig. 4, except for the second sub-block of the coding tree unit, additional splitting is performed on the remaining sub-blocks to determine the coding unit splitting structure (405).

[0145] Meanwhile, in Fig. 4, the M-th picture is divided into 12 coding tree units of the same size, but this is only one example, and the picture can be divided into L coding tree units of the same size. Here, L is a positive integer greater than or equal to 1.

[0146] Meanwhile, the splitting method and coding unit splitting structure (405) of the Kth coding tree unit in FIG. 4 are examples, and the splitting of the coding tree unit can be performed using at least one of a binary tree splitting method, a ternary tree splitting method, and a photo tree splitting method. The encoder can determine the splitting structure by performing the splitting so that the rate-distortion cost (RD cost) value is the smallest.

[0147] Meanwhile, in the coding tree partitioning structure, the maximum size of a coding unit may be equal to the size of the coding tree unit. In addition, in the coding tree partitioning structure, the coding tree unit may be the top node.

[0148] Meanwhile, in this specification, sub-block and sub-block may have the same meaning.

[0149]

[0150] In multi-pass coding, coding may be performed based on rate-distortion cost optimization in the first coding pass, and a coding unit partition structure may be determined. Then, in the second coding pass, coding may be performed based on information about the partition structure obtained in the first coding pass, and a coding unit partition structure may be determined. In this way, in multi-pass coding, coding may be performed in the (N+1)th coding pass based on information about the partition structure determined in the Nth coding pass, and a coding unit partition structure may be determined. Here, the Nth coding pass refers to the Nth step in which coding is performed in multi-pass coding, and N is an integer greater than or equal to 1.

[0151] Meanwhile, the fact that coding is performed in the second coding pass based on information about the segmentation structure obtained in the first coding pass may mean that coding is performed after the initial segmentation structure in the second coding pass is determined based on information about the segmentation structure obtained in the first coding pass. Here, the initial segmentation structure may refer to the structure that serves as the starting point for segmentation in the coding pass.

[0152] Meanwhile, in this specification, the first division structure may mean a division structure determined in the first coding pass, and the second division structure may mean a division structure determined in the second coding pass.

[0153]

[0154] Hereinafter, a method for determining a block division structure based on improved multi-pass coding according to one embodiment of the present invention will be described with reference to FIGS. 5 to 10.

[0155] FIG. 5 is a diagram illustrating a method for determining a partition structure using photo tree-based multi-pass coding according to one embodiment of the present invention. The photo tree-based multi-pass coding method may refer to a method in which the initial partition structure is determined as a photo tree structure when coding is performed in a coding pass following the first coding pass in multi-pass coding. Here, the initial partition structure may refer to a structure that serves as the starting point of the partition.

[0156] In photo tree-based multi-pass coding, in the second coding pass, segmentation can be performed using only information about the photo tree segmentation structure of the coding unit segmentation structure determined in the first coding pass. Accordingly, the photo tree structure of the first segmentation structure is determined as the initial segmentation structure of the second segmentation structure, and in the second coding pass, segmentation can be performed starting from the initial segmentation structure of the second segmentation structure to determine the second segmentation structure.

[0157] In Fig. 5, the first coding pass CU partitioning structure (First coding pass CU partitioning, 500) refers to the first partitioning structure as the coding unit partitioning structure determined in the first coding pass. And, the second coding pass initial CU partitioning structure (Initial CU partitioning for second coding pass, 501) refers to the initial partitioning structure of the second partitioning structure.

[0158] In Fig. 5, C0, C1, C2, and C3 of the first coding pass CU partitioning structure (500) represent respective regions in which the coding tree unit is partitioned using the photo tree partitioning method. Referring to Fig. 5, partitioning was performed for C0 using the photo tree partitioning method, and no additional partitioning was performed for C1. Splitting was performed for C2 using the multi-type tree partitioning method, and splitting was performed for C3 using the photo tree partitioning method and the multi-type tree partitioning method.

[0159] In Fig. 5, the areas shaded in gray in the first coding pass CU partitioning structure (500) represent areas where partitioning was performed using the multi-type tree partitioning method. That is, C2 is a region where partitioning was performed using the multi-type tree partitioning method, and some areas of C3 are regions where partitioning was performed using the multi-type tree partitioning method. Areas shaded in white in the first coding pass CU partitioning structure (500), excluding the areas shaded in gray, represent areas where partitioning was performed using only the photo tree partitioning method.

[0160] Referring to FIG. 5, the division structure of the area indicated in white and the photo tree structure of the area indicated in gray shades in the first coding pass CU division structure (500) are determined as the second coding pass initial coding CU division structure (501). That is, the photo tree structure of the first division structure is determined as the initial division structure of the second division structure.

[0161] And, when the initial division structure of the second division structure is determined, in the second coding pass, additional division can be performed according to the coding unit division structure determination method described in FIG. 4, with the initial division structure as the starting structure for division.

[0162] Meanwhile, the first coding pass CU division structure of FIG. 5 is an example, and any first division structure can be determined according to the coding unit division structure determination method described in FIG. 4.

[0163] Meanwhile, in FIG. 5, only information about the photo tree partition structure of the first partition structure is used in the second coding pass, but this is only an example, and in the second coding pass, partitioning may be performed using only information about the photo tree structure and binary tree structure of the first partition structure, or using only information about the photo tree structure and ternary tree structure of the first partition structure.

[0164] Meanwhile, in Fig. 5, division is performed using only information about the photo tree division structure of the first division structure in the second coding pass, but this is only an example. In photo tree-based multi-pass coding, when coding is performed in the N+1th coding pass, the coding unit division structure can be determined using only information about the photo tree division structure of the coding unit division structure determined in the Nth coding pass. In this case, N is an integer greater than or equal to 1.

[0165]

[0166] FIG. 6 is a diagram illustrating a method for determining a segmentation structure using depth-condition-based multi-pass coding according to one embodiment of the present invention. The depth-condition-based multi-pass coding method may refer to a method in which, when coding is performed in a coding pass following the first coding pass, an initial segmentation structure is determined based on the segmentation depth. The initial segmentation structure may refer to a structure that serves as the starting point of the segmentation.

[0167] In depth condition-based multi-pass coding, the depth condition may mean a criterion by which the depth value of a segmentation structure is determined by comparing it with a predetermined threshold value.

[0168] In depth condition-based multi-pass coding, in the second coding pass, segmentation can be performed using only information about segmentation structures that satisfy a predefined depth condition in the coding unit segmentation structure determined in the first coding pass. Accordingly, a segmentation structure of a predefined segmentation depth of the first segmentation structure is determined as an initial segmentation structure of the second segmentation structure, and in the second coding pass, segmentation can be performed starting from the initial segmentation structure of the second segmentation structure to determine the second segmentation structure.

[0169] Hereinafter, a method for determining a block partition structure using depth condition-based multi-pass coding when the depth condition is 2 is described with reference to FIG. 6. Here, the depth condition of 2 may mean a condition that is satisfied when the depth value of the partition structure is equal to or less than the threshold value 2.

[0170] In Fig. 6, the first coding pass CU partitioning structure (First coding pass CU partitioning, 600) refers to the first partitioning structure, which is the coding unit partitioning structure determined in the first coding pass. And, the second coding pass initial CU partitioning structure (Initial CU partitioning for second coding pass, 601) refers to the initial partitioning structure of the second partitioning structure.

[0171] In Fig. 6, in the first coding pass CU division structure (600), C0, C1, C2, and C3 represent respective regions into which the coding tree unit is divided.

[0172] Referring to Fig. 6, C0 was partitioned using the photo tree partitioning method, and the depth of the partition structure of the C0 region is 2. C1 was not additionally partitioned, so the depth of the partition structure is 1.

[0173] In Fig. 6, the area shaded in gray in the first coding pass CU partition structure (600) indicates an area where partitioning was performed with a partition depth of 3 or more. In C2, partitioning is performed up to a partition structure with a partition depth of 5 using the multi-type tree partition method. In C3, partitioning is performed up to a partition structure with a partition depth of 4 using the photo tree partition method and the multi-type tree partition method. In the first coding pass CU partition structure (600), the area shaded in white, excluding the area shaded in gray, indicates an area where partitioning was performed with a partition depth of 2 or less.

[0174] Referring to FIG. 6, in the first coding pass CU partition structure (600), the partition structure in the area indicated in white and the partition structure with a partition depth of 2 or less in the area indicated in gray shade are determined as the second coding pass initial coding CU partition structure (601). That is, in the first partition structure, the partition structure with a partition depth equal to or less than 2 is determined as the initial partition structure of the second partition structure.

[0175] And, when the initial division structure of the second division structure is determined, in the second coding pass, additional division can be performed according to the coding unit division structure determination method described in FIG. 4, with the initial division structure as the starting structure for division.

[0176] Meanwhile, the first coding pass CU division structure of FIG. 6 is an example, and any first division structure can be determined according to the coding unit division structure determination method described in FIG. 4.

[0177] Meanwhile, in Fig. 6, the depth condition is 2, but this is just one example, and the depth condition may be D. In this case, D is an integer greater than or equal to 0 and may be equal to or less than the maximum depth.

[0178] Meanwhile, in Fig. 6, in the second coding pass, only information about the partition structure that satisfies the depth condition predefined in the first partition structure is used to perform the partitioning, but this is just one example. In depth condition-based multi-pass coding, when coding is performed in the (N+1)th coding pass, only information about the partition structure that satisfies the depth condition predefined in the coding unit partition structure determined in the (N)th coding pass can be used to determine the coding unit partitioning structure. In this case, N is an integer greater than or equal to 1.

[0179] Meanwhile, in Fig. 6, the depth condition is a condition that is satisfied when the depth value of the segmented structure is equal to or less than a predetermined threshold value. However, this is an example, and the depth condition may mean a case where the depth of the segmented structure is less than a predetermined threshold value. As another example, the depth condition may mean a case where the depth value of the segmented structure is greater than a predetermined threshold value.

[0180]

[0181] FIG. 7 is a diagram illustrating a method for determining a partition structure using size-condition-based multi-pass coding according to one embodiment of the present invention. The size-condition-based multi-pass coding method may refer to a method in which, when coding is performed in a coding pass following the first coding pass, the initial partition structure is determined based on the size of the block. The initial partition structure may refer to a structure that serves as the starting point of the partition.

[0182] In size condition-based multi-pass coding, the size condition may mean a criterion by which the size of a block is determined by comparing it with a predetermined size.

[0183] In size condition-based multi-pass coding, in the second coding pass, segmentation can be performed using only information about segmentation structures that satisfy predefined size conditions in the coding unit segmentation structure determined in the first coding pass.

[0184] In Fig. 7, the first coding pass CU partitioning structure (First coding pass CU partitioning, 700) refers to the first partitioning structure, which is the coding unit partitioning structure determined in the first coding pass. And, the second coding pass initial CU partitioning structure (Initial CU partitioning for second coding pass, 701) refers to the initial partitioning structure of the second partitioning structure.

[0185] In Figure 7, the width of the coding tree unit (W CTU , 702) and the height of the coding tree unit (H CTU , 703) are n, respectively. Here, n means the number of pixels.

[0186] Hereinafter, with reference to FIG. 7, a method for determining a block partition structure using size condition-based multi-pass coding when the size condition is n / 4 x n / 4 will be described. Here, the size condition of n / 4 x n / 4 may mean a condition that is satisfied when the size of a block in the first partition structure is equal to or greater than n / 4 x n / 4. Accordingly, a partition structure of an area larger than a pre-defined size of the first partition structure may be determined as an initial partition structure of the second partition structure, and in the second coding pass, partitioning may be performed starting from the initial partition structure of the second partition structure to determine the second partition structure.

[0187] In Fig. 7, in the first coding pass CU division structure (700), C0, C1, C2, and C3 represent respective regions into which the coding tree unit is divided.

[0188] In the first coding pass CU partitioning structure (700) in Fig. 7, the area shaded in gray indicates an area where partitioning is performed so that the block size is smaller than n / 4 x n / 4. In the first coding pass CU partitioning structure (700), the area shaded in white, excluding the area shaded in gray, indicates an area where the size of the partitioned block is all equal to or greater than n / 4 x n / 4.

[0189] Referring to FIG. 7, in the first coding pass CU partition structure (700), the partition structure of the area indicated in white and the partition structure of the area indicated in gray shaded area whose size of the partitioned block is equal to or greater than n / 4 xn / 4 are determined as the second coding pass initial coding CU partition structure (701). That is, in the first partition structure, the partition structure composed of the area whose size is equal to or greater than n / 4 xn / 4 is determined as the initial partition structure of the second partition structure.

[0190] And, when the initial division structure of the second division structure is determined, in the second coding pass, additional division can be performed according to the coding unit division structure determination method described in FIG. 4, with the initial division structure as the starting structure for division.

[0191] Meanwhile, the first coding pass CU division structure of FIG. 7 is an example, and any first division structure can be determined according to the coding unit division structure determination method described in FIG. 4.

[0192] Meanwhile, in Fig. 7, the size condition is n / 4 x n / 4, but this is just one example, and the size condition can be S x S. In this case, S is a positive integer.

[0193] Meanwhile, in Fig. 7, the size condition considers both the width and height of the block, but this is only one example, and either the width or height of the block may be considered. For example, the size condition may be satisfied when the block height is greater than or equal to a predetermined threshold value. Alternatively, the size condition may be satisfied when the block width is greater than or equal to a predetermined threshold value.

[0194] Meanwhile, in Fig. 7, the splitting is performed using only information about the split structure that satisfies the size condition predefined in the first split structure in the second coding pass, but this is only an example. In size condition-based multi-pass coding, when coding is performed in the (N+1)th coding pass, the coding unit splitting structure can be determined using only information about the split structure that satisfies the size condition predefined in the coding unit splitting structure determined in the (N)th coding pass. In this case, N is an integer greater than or equal to 1.

[0195] Meanwhile, in FIG. 7, the size condition is a condition that is satisfied when the block size is equal to or greater than a predetermined size. However, this is only one example; the size condition may mean that the block size is greater than the predetermined size. As another example, the size condition may mean that the block size is smaller than the predetermined size. As another example, the size condition may mean that the block size is equal to the predetermined size. As another example, the size condition may mean that the block size is equal to or less than the predetermined size.

[0196]

[0197] FIG. 8 is a diagram illustrating a method for determining a partition structure using merge-based multi-pass coding according to one embodiment of the present invention. The merge-based multi-pass coding method may refer to a method in which, when coding is performed in a coding pass following the first coding pass, an initial partition structure is derived as a structure that is merged in addition to the partition structure derived in the previous pass. The initial partition structure may refer to a structure that serves as the starting point of the partition.

[0198] In Fig. 8, the first coding pass CU partitioning structure (First coding pass CU partitioning, 800) refers to the first partitioning structure as the coding unit partitioning structure determined in the first coding pass. In addition, in the first coding pass CU partitioning structure (800), C0, C1, C2, and C3 refer to respective regions into which the coding tree unit is partitioned.

[0199] First, the encoder can determine whether to merge multiple sub-blocks divided according to the first division structure.

[0200] Referring to Fig. 8, C2-1-1 in the C2 region is a coding unit divided by performing a ternary tree division method. Here, the encoder determines whether to merge C2-1-1, and if it is determined that C2-1-1 is merged, merging (801) of C2-1-1, C2-1-2, and C2-1-3 can be performed.

[0201] And, a merge can be performed to derive a merged block (Combined block, 802) structure. That is, a structure in which multiple sub-blocks divided according to the first coding pass CU division structure (800) are merged can be derived.

[0202] And, the merged structure can be determined as the initial split structure of the second coding pass.

[0203] And, once the initial division structure of the second division structure is determined, additional division can be performed in the second coding pass according to the method for determining the coding unit division structure described in FIG. 4, with the initial division structure as the starting structure for division.

[0204] Meanwhile, if it is determined that C2-1-1 is not merged, no merging is performed on the first coding pass CU partition structure (800), and the first coding pass CU partition structure (800) can be determined as the initial partition structure of the second coding pass.

[0205] Meanwhile, in Fig. 8, the initial division structure of the second division structure is derived as a structure in which additional merging is performed from the first division structure, but this is only one example, and the initial division structure of the second division structure can be derived as a structure in which additional merging is performed from the photo tree structure of the first division structure.

[0206] Meanwhile, in Fig. 8, the initial division structure of the second division structure is derived as a structure in which additional merging is performed on the first division structure, but this is only one example, and the initial division structure of the second division structure can be derived as a structure in which additional merging is performed on the structure of the predefined division depth of the first division structure. Here, the method for determining the structure of the predefined division depth of the first division structure can be the same as the method for determining the initial division structure of the depth condition-based multi-pass coding described above in Fig. 6.

[0207] Meanwhile, in Fig. 8, the initial division structure of the second division structure is derived as a structure in which additional merging is performed on the first division structure, but this is only one example, and the initial division structure of the second division structure can be derived as a structure in which additional merging is performed on a division structure composed of an area larger than a pre-defined size in the first division structure. Here, the method for determining the division structure composed of an area larger than a pre-defined size in the first division structure may be the same as the method for determining the initial division structure of the size condition-based multi-pass coding described above in Fig. 7.

[0208] Meanwhile, in FIG. 8, if it is determined that C2-1-1 is not merged, merging is not performed in the first coding pass CU split structure (800), but this is just one example. If it is determined whether C2-1-1 and C2-1-2 are merged, it is possible to additionally determine whether C2-1-3 is merged. In addition, if it is determined that C2-1-3 is merged, the initial split structure of the second split structure can be derived as a structure in which merging is performed on C2-1-1, C2-1-2, and C2-1-3, and if it is determined that C2-1-3 is not merged, the initial split structure of the second split structure can be derived as a structure in which C2-1-1 and C2-1-2 are merged and C2-1-3 is not merged.

[0209] Meanwhile, in Fig. 8, the second coding pass uses a partition structure in which additional merging is performed on the first partition structure as the initial partition structure. This is an example, and in a method for determining a block partition structure using merge-based multi-pass coding, additional merging may be performed on the coding unit partition structure determined in the Nth coding pass to determine the initial partition structure of the N+1th coding pass. In this case, N is an integer greater than or equal to 1.

[0210]

[0211] Meanwhile, in a block structure determination method using merge-based multi-pass coding, merging of sub-blocks can be performed within an allowable depth range.

[0212] Specifically, whether to merge is determined only in a partition structure of a predefined depth, and if it is out of this range, whether to merge is not determined and the merge may not be performed.

[0213] Referring to Fig. 8, if the split depth of the merged block (802) is equal to or less than the predefined depth, whether to merge is not determined and merging may not be performed. If merging is not performed, a structure including the structures of C2-1-1, C2-1-2, and C2-1-3 of the first coding pass CU split structure (800) may be determined as the initial split structure of the second coding pass.

[0214] Meanwhile, the predefined depth can be equal to the depth of the coding tree unit. Furthermore, the predefined depth can be set via a global parameter.

[0215] Additionally, the predefined depth can be set for each region or based on information collected through the first pass. This information may include information on motion complexity, scene complexity, scene changes, coding parameters determined through the first pass, syntax information, the size of the coding unit block, and the size of the motion vector.

[0216] Additionally, the predefined depth can be set at a higher level, where the higher level can include a Sequence parameter set (SPS), a Picture parameter set (PPS), a Picture header (PH), and a Slice header (SH).

[0217] Additionally, the predefined depth can be defined by the encoder / decoder's promise.

[0218] Meanwhile, in FIG. 8, if the split depth of the merged block (802) is equal to or less than the predefined depth, whether to merge is not determined and merging is not performed. However, this is an example, and if the split depth is less than the predefined depth, whether to merge is not determined and merging may not be performed.

[0219]

[0220] Meanwhile, in a block structure determination method using merge-based multi-pass coding, merging of sub-blocks can be performed within the allowable block size range.

[0221] Specifically, whether or not to merge is determined only in an area within a predefined size, and if it is outside this size, whether or not to merge may not be determined and the merge may not be performed.

[0222] For example, if the width of the subblock is 2 u and the width of the merged block is 2 u+r In cases where the height of a sub-block is greater than 2, merging may not be performed without determining whether to merge the sub-blocks. As another example, if the height of a sub-block is 2 v and the width of the merged block is 2 v+r In larger cases, merging may not be performed without determining whether to merge sub-blocks. As another example, if the size of the sub-block is 2 u x 2 v and the size of the merged block is 2 u+r x 2 v+r In larger cases, merging may not be performed without determining whether to merge sub-blocks. In the above examples, u, v, and r are integers greater than or equal to 0.

[0223] Referring to FIG. 8, if the size of the merged block (802) is larger than the predefined size, whether to merge is not determined and merging may not be performed. If merging is not performed, the first coding pass CU partition structure (800) may be determined as the initial partition structure of the second coding pass.

[0224] Meanwhile, the predefined size may be equal to the size of the coding tree unit. Furthermore, the predefined size may be set via a global parameter.

[0225] Additionally, the predefined size can be set for each region or based on information collected through the first pass. This information can include information on motion complexity, scene complexity, scene changes, coding parameters determined through the first pass, syntax information, the size of the coding unit block, and the size of the motion vector.

[0226] Additionally, the predefined size can be set at a higher level, where the higher level can include a Sequence parameter set (SPS), a Picture parameter set (PPS), a Picture header (PH), and a Slice header (SH).

[0227] Additionally, the predefined size can be defined by the encoder / decoder's promise.

[0228] Meanwhile, in FIG. 8, if the size of the merged block (802) is larger than a predefined size, whether to merge is not determined and merging is not performed. However, this is an example, and if the size is larger than or equal to a predefined size, whether to merge may not be determined and merging may not be performed.

[0229]

[0230] FIG. 9 is a diagram illustrating a method for determining a partition structure using additional partition-based multi-pass coding according to one embodiment of the present invention. The additional partition-based multi-pass coding method may refer to a method in which, when coding is performed in a coding pass after the first coding pass, an initial partition structure is derived as a structure in which additional partitioning is performed on the partition structure derived in the previous pass. The initial partition structure may refer to a structure that serves as a starting point for partitioning.

[0231] In Fig. 9, the first coding pass CU partitioning structure (First coding pass CU partitioning, 900) refers to the first partitioning structure as the coding unit partitioning structure determined in the first coding pass. In addition, in the first coding pass CU partitioning structure (900), C0, C1, C2, and C3 refer to respective regions into which the coding tree unit is partitioned.

[0232] First, the encoder can determine whether to further split the plurality of sub-blocks split according to the first split structure.

[0233] Referring to Fig. 9, C2-1-1 in the C2 region is a coding unit divided by performing a ternary tree division method.

[0234] First, the encoder determines whether to further split C2-1-1, and if it is determined that C2-1-1 is further split, then additional splitting of C2-1-1 is performed, and if it is determined that C2-1-1 is not split, then additional splitting may not be performed.

[0235] In addition, the encoder can determine whether to perform additional division of C2-1-2. If it is determined that C2-1-2 is additionally divided, additional division of C2-1-2 may be performed, and if it is determined that no additional division is performed, additional division may not be performed.

[0236] In addition, the encoder can determine whether to perform additional division of C2-1-3. If it is determined that C2-1-3 is additionally divided, additional division of C2-1-3 may be performed, and if it is determined that C2-1-3 is not divided, additional division of C2-1-3 may not be performed.

[0237] For example, Fig. 9 is a case where it is determined that C2-1-1 and C2-1-2 are additionally divided, and C2-1-3 is not additionally divided.

[0238] In addition, additional division can be performed to derive a divided block (902) structure. That is, a structure in which multiple sub-blocks divided according to the first coding pass CU division structure (900) are additionally divided can be derived.

[0239] Additionally, the segmented structure can be determined as the initial segmented structure of the second coding pass.

[0240] And, when the initial division structure of the second division structure is determined, in the second coding pass, additional division can be performed according to the coding unit division structure determination method described in FIG. 4, with the initial division structure as the starting structure for division.

[0241] Meanwhile, if it is determined that no additional division will be performed, no additional division is performed on the first coding pass CU division structure (900), and the first coding pass CU division structure (900) can be determined as the initial division structure of the second coding pass.

[0242] Meanwhile, in Fig. 9, the initial division structure of the second division structure is derived as a structure in which division is additionally performed from the first division structure, but this is only one example, and the initial division structure of the second division structure can be derived as a structure in which division is additionally performed from the photo tree structure of the first division structure.

[0243] Meanwhile, in Fig. 9, the initial partition structure of the second partition structure is derived as a structure in which additional partitioning is performed on the first partition structure, but this is only one example, and the initial partition structure of the second partition structure can be derived as a structure in which additional partitioning is performed on the partition structure of the structure of the predefined partition depth of the first partition structure. Here, the method for determining the structure of the predefined partition depth of the first partition structure can be the same as the method for determining the initial partition structure of the depth condition-based multi-pass coding described above in Fig. 6.

[0244] Meanwhile, in Fig. 9, the initial division structure of the second division structure is derived as a structure in which division is additionally performed from the first division structure, but this is only one example, and the initial division structure of the second division structure can be derived as a structure in which division is additionally performed from a division structure composed of an area larger than a pre-defined size in the first division structure. Here, the method for determining the division structure composed of an area larger than a pre-defined size in the first division structure may be the same as the method for determining the initial division structure of the size condition-based multi-pass coding described above in Fig. 7.

[0245] Meanwhile, in Fig. 9, the second coding pass uses a partition structure in which additional partitioning is performed on the first partition structure as the initial partition structure. This is an example, and in a method for determining a block partition structure using additional partition-based multi-pass coding, additional partitioning is performed on the coding unit partition structure determined in the Nth coding pass, so that the initial partition structure of the N+1th coding pass can be determined. In this case, N is an integer greater than or equal to 1.

[0246]

[0247] Meanwhile, in a block structure determination method using additional partition-based multi-pass coding, the partitioning of sub-blocks can be performed within an allowable depth range.

[0248] Specifically, whether to perform additional division is determined only in a division structure of a predefined depth, and if it is out of this range, whether to perform additional division is not determined and additional division may not be performed.

[0249] Referring to FIG. 9, if the split depth of the additionally split block (902) is equal to or greater than the predefined depth, whether or not to perform additional splitting is not determined and additional splitting may not be performed. If additional splitting is not performed, a structure including the structures of C2-1-1, C2-1-2, and C2-1-3 of the first coding pass CU split structure (900) may be determined as the initial split structure of the second coding pass.

[0250] Meanwhile, the predefined depth can be set via a global parameter.

[0251] Additionally, the predefined depth can be set for each region or based on information collected through the first pass. This information may include information on motion complexity, scene complexity, scene changes, coding parameters determined through the first pass, syntax information, the size of the coding unit block, and the size of the motion vector.

[0252] Additionally, the predefined depth can be set at a higher level, where the higher level can include a Sequence parameter set (SPS), a Picture parameter set (PPS), a Picture header (PH), and a Slice header (SH).

[0253] Additionally, the predefined depth can be defined by the encoder / decoder's promise.

[0254] Meanwhile, in FIG. 9, if the split depth of the additionally split block (902) is equal to or greater than the predefined depth, whether to split additionally is not determined and no additional splitting is performed. However, this is an example, and if the split depth is greater than the predefined depth, whether to split additionally is not determined and no additional splitting may be performed.

[0255]

[0256] Meanwhile, in a block structure determination method using additional division-based multi-pass coding, additional division of sub-blocks can be performed within the allowable block size range.

[0257] Specifically, whether to further divide is determined only in an area within a predefined size, and if it is outside this size, whether to further divide is not determined and further division may not be performed.

[0258] For example, if the width of the subblock is 2 uand the width of the additionally divided block is 2 u-r If the height of the sub-block is less than 2, it is not determined whether to further split the sub-block and no further splitting may be performed. As another example, if the height of the sub-block is 2 v and the width of the additionally divided block is 2 v-r If the size of the sub-block is smaller than 2, it is not determined whether to further split the sub-block and no further splitting may be performed. As another example, if the size of the sub-block is 2 u x 2 v And the size of the additionally divided block is 2 u-r x 2 v-r In smaller cases, it is not determined whether to further split the sub-block and no further splitting may be performed. In the above examples, u, v, and r are integers greater than or equal to 0.

[0259] Referring to FIG. 9, if the size of the additionally divided block (902) is smaller than the predefined size, whether or not to perform additional division is not determined and additional division may not be performed. If additional division is not performed, the first coding pass CU division structure (900) may be determined as the initial division structure of the second coding pass.

[0260] Additionally, the predefined size can be set via a global parameter.

[0261] Additionally, the predefined size can be set for each region or based on information collected through the first pass. This information can include information on motion complexity, scene complexity, scene changes, coding parameters determined through the first pass, syntax information, the size of the coding unit block, and the size of the motion vector.

[0262] Additionally, the predefined size can be set at a higher level, where the higher level can include a Sequence parameter set (SPS), a Picture parameter set (PPS), a Picture header (PH), and a Slice header (SH).

[0263] Additionally, the predefined size can be defined by the encoder / decoder's promise.

[0264] Meanwhile, in FIG. 9, whether to perform additional division is not determined and additional division is not performed only when the size of the additionally divided block (902) is smaller than the predefined size. However, this is an example, and whether to perform additional division may not be determined and additional division may not be performed when the size is smaller than or equal to the predefined size.

[0265]

[0266] FIG. 10 is a diagram for explaining a multi-type tree splitting method in a second coding pass according to one embodiment of the present invention.

[0267] According to one embodiment, when partitioning is performed starting from an initial partition structure in a second coding pass, the partitioning method may be limited to a multi-type tree partitioning method. Specifically, the second partitioning structure may be determined by performing at least one of a binary tree partitioning method and a ternary tree partitioning method starting from the initial partitioning structure.

[0268] First, partitioning may be performed in the first coding pass to determine the first partitioning structure. In Fig. 10, the first coding pass CU partitioning structure (First coding pass CU partitioning, 1000) refers to the first partitioning structure.

[0269] In addition, the initial division structure of the second division structure, which is the division structure of the second coding pass, may be determined as the first division structure. Here, the initial division structure may refer to the structure that serves as the starting point for the division of the second coding pass.

[0270] In addition, in the second coding pass, a partitioning process may be performed starting from the initial partitioning structure to determine a second partitioning structure. At this time, the partitioning process may be performed using at least one of a binary tree partitioning method and a ternary tree partitioning method.

[0271] Meanwhile, the binary tree partitioning method and the ternary tree partitioning method can be performed as described above in FIG. 4.

[0272] Meanwhile, if a multi-type tree partitioning method is performed on an upper block and it is partitioned into sub-blocks, each sub-block can be partitioned only by the multi-type tree partitioning method and cannot be partitioned by the photo-tree partitioning method. At this time, the multi-type tree partitioning method can be performed recursively within an allowable size or an allowable depth. Specifically, if the current block is a sub-block partitioned by the multi-type tree partitioning method and the current block satisfies a partitioning condition, the current block can be partitioned into sub-blocks by the multi-type tree partitioning method. At this time, the partitioning condition may mean a condition that is satisfied when the current block is less than a predetermined partitioning depth, or a condition that is satisfied when the current block is greater than a predetermined size.

[0273] In Fig. 10, the final CU partitioning structure for the second coding pass (1001) refers to the second partitioning structure determined by performing only the multi-type partitioning method on the initial partitioning structure. The gray-shaded area of ​​the final CU partitioning structure for the second coding pass (1001) refers to the area where the multi-type tree partitioning was performed on the initial partitioning structure.

[0274] Meanwhile, in Fig. 10, the first partition structure becomes the initial partition structure of the second partition structure and the partitioning is performed. However, this is only one example, and the photo tree structure of the first partition structure may become the initial partition structure of the second partition structure and the partitioning may be performed. In this case, the method for determining the initial partition structure of the second partition structure may be the same as the method for determining the initial partition structure in the photo tree-based multi-pass coding method described above in Fig. 5.

[0275]

[0276] FIG. 11 is a flowchart illustrating an image encoding method according to one embodiment of the present invention. Specifically, FIG. 11 is a flowchart illustrating a method for determining a block division structure based on enhanced multi-pass coding in an image encoding method. The division structure determination method of FIG. 11 can be performed by an encoding device.

[0277] The video encoding device can determine the first division structure of the current block (S1100).

[0278] And, the image encoding device can determine the second division structure of the current block based on the first division structure (S1110).

[0279] Meanwhile, the initial division structure for determining the second division structure can be derived from the first division structure.

[0280] Meanwhile, the first partition structure includes a photo tree structure, a binary tree structure, and a ternary tree structure, and the initial partition structure may be a photo tree structure of the first partition structure.

[0281] Meanwhile, the initial division structure may be a division structure of a predefined division depth of the first division structure.

[0282] Meanwhile, the initial division structure may be a division structure composed of an area larger than the size defined in the first division structure.

[0283] Meanwhile, the step of determining the second division structure of the current block based on the first division structure includes the step of determining whether to merge a plurality of sub-blocks divided according to the first division structure, and when it is determined that the plurality of sub-blocks divided according to the first division structure are merged, the initial division structure can be derived as a structure in which a plurality of sub-blocks divided according to the first division structure are merged.

[0284] Meanwhile, the step of determining whether to merge a plurality of sub-blocks divided according to the first division structure may be performed in an area within a predefined size.

[0285] Meanwhile, the step of determining whether to merge a plurality of sub-blocks divided according to the first division structure can be performed in a division structure of a predefined division depth.

[0286] Meanwhile, the step of determining the second division structure of the current block based on the first division structure includes the step of determining whether to further divide the plurality of sub-blocks divided according to the first division structure, and if it is determined that the plurality of sub-blocks divided according to the first division structure are further divided, the initial division structure can be derived as a structure in which the plurality of sub-blocks divided according to the first division structure are further divided.

[0287] Meanwhile, the step of determining whether to further divide a plurality of sub-blocks divided according to the first division structure may be performed in an area within a predefined size.

[0288] Meanwhile, the step of determining whether to further divide a plurality of sub-blocks divided according to the first division structure may be performed in a division structure of a predefined division depth.

[0289] Meanwhile, the second division structure can be determined by performing at least one of a ternary division method and a binary division method from the initial division structure.

[0290] Meanwhile, the second division structure can be determined by performing at least one of a ternary division method and a binary division method from the initial division structure.

[0291] And, the image encoding device can encode the second division structure of the current block (S1120).

[0292] Meanwhile, a bitstream can be generated by a video encoding method including the steps of FIG. 11. The bitstream can be stored in a non-transitory computer-readable recording medium and can also be transmitted (or streamed).

[0293]

[0294] Meanwhile, in a multi-pass coding according to one embodiment of the present invention, the encoder may perform encoding in a first coding pass, and perform encoding in a second coding pass based on the occurrence frequency of at least one piece of encoding information determined in the first coding pass. In this case, the encoding information determined in the first coding pass may be encoded at a higher level in the second coding pass based on the occurrence frequency.

[0295] Additionally, the encoder can generate a bitstream based on at least one piece of encoding information determined in the second coding pass. Accordingly, bits can be saved.

[0296] For example, the first coding pass may result in no or very few coding units being encoded as intra-picture predictions in inter-picture pictures or inter-screen slices. In other words, the frequency of occurrence of information encoded by performing intra-picture prediction in the first coding pass may be low.

[0297] In this case, the syntax (non_intra_flag) for whether to perform intra-screen prediction in the second coding pass may be defined at the picture level or slice level, and the value of the syntax may be set to 1. Here, the syntax for whether to perform intra-screen prediction may be non_intra_flag, and a value of the syntax of 1 may mean not performing intra-screen prediction.

[0298] And, in the second coding pass, the value of the corresponding syntax at the coding tree unit level can be considered as 1.

[0299] That is, in the second coding pass, all coding units contained in the corresponding picture or slice can be encoded only through inter-screen prediction. Therefore, even coding units that were encoded through intra-screen prediction in the first coding pass can be encoded through inter-screen prediction in the second coding pass.

[0300] Meanwhile, the above-described example relates to multi-pass coding based on the occurrence frequency of encoded information by performing prediction within the screen, but encoding may be performed in a second coding pass based on the occurrence frequency of at least one encoded piece of information in the first coding pass in the same manner. Here, the encoding information of the first coding pass may include syntax regarding whether a coding unit is vertically split and syntax regarding whether a coding unit is binary split. In addition, the syntax regarding whether a coding unit is vertically split may be mtt_split_cu_vertical_flag, and the syntax regarding whether a coding unit is binary split may be mtt_split_cu_binary_flag.

[0301] Meanwhile, in the above-described example, encoding is performed in the second coding pass based on the occurrence frequency of the encoding information determined in the first coding pass, but in the same manner, encoding can be performed in the (N+1)th coding pass based on the occurrence frequency of the encoding information determined in the Nth coding pass. In this case, N is an integer greater than or equal to 1.

[0302]

[0303] As another example, the number of coding units encoded in skip mode in the coding tree unit may be zero or very small as a result of encoding in the first coding pass. In other words, the frequency of occurrence of information encoded in skip mode in the first coding pass may be low.

[0304] In this case, the syntax regarding whether skip mode is used in the second coding pass may be defined at the coding tree unit level, and the value of the syntax may be set to 0. Here, the syntax regarding whether skip mode is used may be cu_skip_flag, and a value of the syntax of 0 may mean that encoding is performed in a mode other than skip mode.

[0305] And, in the second coding pass, the value of the corresponding syntax at the coding unit level can be considered as 0.

[0306] That is, in the second coding pass, all coding units included in the corresponding coding tree unit can be encoded in a mode other than skip mode. Accordingly, even coding units that were encoded in skip mode in the first coding pass can be encoded in a mode other than skip mode in the second coding pass.

[0307] Meanwhile, the above-described example relates to multi-pass coding based on the occurrence frequency of information encoded in skip mode, but encoding may be performed in a second coding pass based on the occurrence frequency of at least one piece of encoded information in the first coding pass in the same manner. Here, the encoding information of the first coding pass may include syntax regarding a prediction mode, syntax regarding whether matrix-based intra prediction is used, syntax regarding whether palette mode is used, syntax regarding whether affine mode is used, etc.

[0308] Additionally, the syntax for the prediction mode can be pred_mode_flag, the syntax for whether to use matrix-based intra prediction can be intra_mip_flag, the syntax for whether to use palette mode can be pred_mode_plt_flag, and the syntax for whether to use affine mode can be inter_affine_flag.

[0309] Meanwhile, in the above example, the syntax is defined at the coding tree unit level in the second coding pass, but the syntax can also be defined at the picture, slice, and tile levels in the same way.

[0310] Meanwhile, in the above-described example, encoding is performed in the second coding pass based on the occurrence frequency of the encoding information determined in the first coding pass, but in the same manner, encoding can be performed in the (N+1)th coding pass based on the occurrence frequency of the encoding information determined in the Nth coding pass. In this case, N is an integer greater than or equal to 1.

[0311]

[0312] Figure 12 is a flowchart illustrating an image encoding method according to one embodiment of the present invention. Specifically, Figure 12 is a flowchart illustrating an improved multi-pass coding method based on the frequency of encoding information in the image encoding method. The multi-pass coding method of Figure 12 can be performed by an encoding device.

[0313] The video encoding device can perform a first encoding pass for encoding the current picture (S1200).

[0314] And, the image encoding device can perform a second encoding pass for encoding the current picture based on the frequency of at least one piece of encoding information determined in the first encoding pass (S1210).

[0315] And, the image encoding device can generate a bitstream based on at least one piece of encoding information determined in the second encoding pass (S1220).

[0316] Meanwhile, at least one piece of encoding information determined in the first encoding pass may be encoded at a higher level in the second encoding pass according to the occurrence frequency.

[0317] Meanwhile, a bitstream generated by a video encoding method including the steps of FIG. 12 can be stored in a non-transitory computer-readable recording medium and can also be transmitted (or streamed).

[0318]

[0319] Meanwhile, due to the nature of encoding technology, even the same image can produce different results depending on the encoding order. Therefore, the encoder can achieve efficient encoding through multi-pass coding, as described below.

[0320] In a multi-pass coding according to one embodiment of the present invention, the encoder may perform a first coding pass for encoding an original image. At this time, the encoding cost of the original image may be stored in 1_pass_cost.

[0321] Additionally, the encoder may perform a second coding pass to encode a mirrored image of the original image. At this time, the encoding cost of the mirrored image may be stored in 2_pass_cost.

[0322] Additionally, the encoder may perform a third coding pass to encode a vertically symmetrical image of the original image. At this time, the encoding cost of the vertically symmetrical image may be stored in 3_pass_cost.

[0323] Additionally, the encoder may perform a fourth coding pass to encode vertically symmetric and horizontally symmetric images of the original image. At this time, the encoding cost of the vertically symmetric and horizontally symmetric images may be stored in 4_pass_cost.

[0324] And, the encoder can generate a bitstream with a coding pass having a cost that has the smallest value among the coding cost of the original image, the coding cost of the bi-symmetrical image, the coding cost of the vertically symmetrical image, and the coding cost of the vertically symmetrical and bi-symmetrical images. For example, if the coding cost of the vertically symmetrical image is the smallest, the encoder can generate a bitstream with the third coding pass.

[0325] Meanwhile, the aforementioned original image may be any one of a picture, slice, tile, and coding tree unit.

[0326] Meanwhile, the above example performs up to the fourth coding pass using four symmetric methods, but this is only one example, and up to the Nth coding pass can be performed using N symmetric methods, where N is any positive integer.

[0327] Meanwhile, the order of images in which encoding is performed in the encoding pass of the above-described example is one example, and the order of images in which encoding is performed in the encoding pass can also be arbitrarily determined.

[0328] Meanwhile, in order for the decoder to be able to reference normally, the images must all be stored in their original form in the buffer of the decoder.

[0329]

[0330] Figure 13 is a flowchart illustrating an image encoding method according to one embodiment of the present invention. Specifically, Figure 13 is a diagram for explaining an improved multi-pass coding method based on the symmetry of an original image in the image encoding method. The multi-pass coding method of Figure 13 can be performed by an encoding device.

[0331] The image encoding device can perform a first encoding pass for encoding the original image (S1300).

[0332] And, the image encoding device can perform a second encoding pass for encoding a left-right symmetrical image of the original image (S1310).

[0333] And, the image encoding device can perform a third encoding pass for encoding a vertically symmetrical image of the original image (S1320).

[0334] And, the image encoding device can perform a fourth encoding pass for encoding the vertically symmetrical and left-right symmetrical images of the original image (S1330).

[0335] And, the video encoding device can generate a bitstream with an encoding pass having the lowest encoding cost among the first encoding cost of the first encoding pass, the second encoding cost of the second encoding pass, the third encoding cost of the third encoding pass, and the fourth encoding cost of the fourth encoding pass (S1340).

[0336] Meanwhile, the original image may be any one of a picture, a slice, a tile, and a coding tree unit.

[0337] Meanwhile, a bitstream generated by a video encoding method including the steps of FIG. 13 can be stored in a non-transitory computer-readable recording medium and can also be transmitted (or streamed).

[0338]

[0339] FIG. 14 is a drawing exemplifying a content streaming system to which an embodiment according to the present invention can be applied.

[0340] As illustrated in FIG. 14, a content streaming system to which an embodiment of the present invention is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0341] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and CCTVs into digital data, generates a bitstream, and transmits it to the streaming server. Alternatively, if multimedia input devices such as smartphones, cameras, and CCTVs directly generate bitstreams, the encoding server may be omitted.

[0342] The above bitstream can be generated by an image encoding method and / or an image encoding device to which an embodiment of the present invention is applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0343] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server can act as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server may control commands / responses between each device within the content streaming system.

[0344] The streaming server can receive content from a media repository and / or encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0345] Examples of the user devices may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital TVs, desktop computers, digital signage, etc.

[0346] Each server within the above content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0347]

[0348] The above embodiments can be performed in the same or corresponding manner in an encoding device and a decoding device. In addition, an image can be encoded / decoded using at least one or a combination of at least one of the above embodiments.

[0349] The order in which the above embodiments are applied may be different in the encoding device and the decoding device. Alternatively, the order in which the above embodiments are applied may be the same in the encoding device and the decoding device.

[0350] The above embodiments can be performed for each of the luminance and chrominance signals. Alternatively, the above embodiments can be performed identically for the luminance and chrominance signals.

[0351] In the above embodiments, the methods are described based on a flowchart as a series of steps or units. However, the present invention is not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps described above. Furthermore, those skilled in the art will understand that the steps depicted in the flowchart are not exclusive, and that other steps may be included, or one or more steps in the flowchart may be deleted without affecting the scope of the present invention.

[0352] The above embodiments may be implemented in the form of program commands that can be executed by various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the computer-readable recording medium may be those specifically designed and constructed for the present invention, or may be known and usable by those skilled in the art of computer software.

[0353] The bitstream generated by the encoding method according to the above embodiment can be stored in a non-transitory computer-readable recording medium. In addition, the bitstream stored in the non-transitory computer-readable recording medium can be decoded by the decoding method according to the above embodiment.

[0354] Here, examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0355] Although the present invention has been described above with specific details such as specific components and limited examples and drawings, these are provided only to help a more general understanding of the present invention, and the present invention is not limited to the above examples, and those with ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from this description.

[0356] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and all things that are modified equally or equivalently to the following claims as well as the claims are considered to fall within the scope of the idea of ​​the present invention.

[0357]

[0358] The present invention can be used in a device for encoding / decoding an image and a recording medium storing a bitstream.

Claims

1. In the video encoding method, A step of determining the first division structure of the current block; A step of determining a second division structure of the current block based on the first division structure; and comprising a step of encoding a second division structure of the current block; An image encoding method, characterized in that the initial division structure for determining the second division structure is derived from the first division structure.

2. In paragraph 1, The above first partition structure includes a photo tree structure, a binary tree structure, and a ternary tree structure, An image encoding method, characterized in that the initial division structure is a photo tree structure of the first division structure.

3. In paragraph 1, An image encoding method, characterized in that the initial segmentation structure is a segmentation structure having a predefined segmentation depth of the first segmentation structure.

4. In paragraph 1, An image encoding method, characterized in that the initial division structure is a division structure composed of an area larger than a size defined in the first division structure.

5. In paragraph 1, The step of determining the second division structure of the current block based on the first division structure is: A step of determining whether to merge a plurality of sub-blocks divided according to the first division structure is included, A video encoding method, characterized in that when it is determined that a plurality of sub-blocks divided according to the first division structure are merged, the initial division structure is derived as a structure in which a plurality of sub-blocks divided according to the first division structure are merged.

6. In paragraph 5, A video encoding method, characterized in that the step of determining whether to merge a plurality of sub-blocks divided according to the first division structure is performed in an area within a predefined size.

7. In paragraph 5, A video encoding method, characterized in that the step of determining whether to merge a plurality of sub-blocks divided according to the first division structure is performed in a division structure having a predefined division depth.

8. In paragraph 1, The step of determining the second division structure of the current block based on the first division structure is: A step of determining whether to further divide a plurality of sub-blocks divided according to the first division structure is included, A video encoding method, characterized in that when additional division of a plurality of sub-blocks divided according to the first division structure is determined, the initial division structure is derived as a structure in which a plurality of sub-blocks divided according to the first division structure are additionally divided.

9. In paragraph 8, A video encoding method, characterized in that the step of determining whether to further divide a plurality of sub-blocks divided according to the first division structure is performed in an area within a predefined size.

10. In paragraph 8, A video encoding method, characterized in that the step of determining whether to further divide a plurality of sub-blocks divided according to the first division structure is performed in a division structure having a predefined division depth.

11. In paragraph 1, An image encoding method, characterized in that the second division structure is determined by performing at least one of a ternary division method and a binary division method from the initial division structure.

12. In the video encoding method, A step of performing a first encoding pass for encoding the current picture; A step of performing a second encoding pass for encoding the current picture based on the frequency of at least one piece of encoding information determined in the first encoding pass; and Comprising a step of generating a bitstream based on at least one encoding information determined in the second encoding pass, An image encoding method, characterized in that at least one piece of encoding information determined in the first encoding pass is encoded at a higher level in the second encoding pass according to occurrence frequency.

13. In a video encoding method, A step of performing a first encoding pass for encoding the original image; A step of performing a second encoding pass for encoding a left-right symmetrical image of the original image; A step of performing a third encoding pass for encoding a top-bottom symmetrical image of the original image; A step of performing a fourth encoding pass for encoding a top-bottom symmetrical and left-right symmetrical image of the original image; and A video encoding method comprising the step of generating a bitstream with an encoding pass having a lowest encoding cost among a first encoding cost of the first encoding pass, a second encoding cost of the second encoding pass, a third encoding cost of the third encoding pass, and a fourth encoding cost of the fourth encoding pass.

14. In paragraph 13, A method for encoding an image, wherein the original image is any one of a picture, a slice, a tile, and a coding tree unit.

Images