Image encoding method and apparatus, and recording medium storing bitstream

Abstract

An image decoding method, according to the present disclosure, comprises the steps of: generating an inter prediction block of a current block from a reference picture; generating an intra prediction block of the current block on the basis of one or more intra prediction mode candidates; and generating a final prediction block of the current block by performing a weighted sum of the inter prediction block and the intra prediction block, wherein the one or more intra prediction mode candidates include a first intra prediction mode candidate derived on the basis of the inter prediction block, and the inter prediction block may be generated on the basis of motion information of an inter prediction mode candidate of the current block.

Description

Video encoding method and device, and recording medium storing bitstream

[0001] The present invention relates to a video encoding method and device, and a recording medium storing a bitstream, and more particularly, to a video encoding method and device for high-speed encoding, and a recording medium storing a bitstream.

[0002] Virtual reality (VR) and augmented reality (AR) technologies are evolving to provide users with more immersive and natural experiences. To deliver natural and immersive omnidirectional video, supporting various degrees of freedom (DOF), such as 3DoF, 3DoF+, and 6DoF, is crucial. Accordingly, active research is underway on the Multi-View Immersive Video (MIV) encoding standard, which provides compression standards for encoding / decoding and rendering in 3DoF, 3DoF+, and 6DoF environments.

[0003] 3DoF can refer to a technology that supports degrees of freedom for three-directional movements (pitch, roll, yaw). 3DoF+ can refer to a technology that supports motion parallax for very limited movements, such as slight head movements in a seated environment, using multiple images including omnidirectional scenes and multiple virtual viewpoint images synthesized based on these images. 6DoF can refer to a technology that supports degrees of freedom for six-directional movements (left-right movement, up-and-down movement, forward-backward movement, pitch, roll, yaw).

[0004] The MIV encoding standard technology aims to provide an interface for viewing immersive videos and reproduce high-quality viewport videos, along with the compression efficiency of traditional video coding.

[0005] The present disclosure aims to increase the number of intra prediction mode candidates by deriving intra prediction mode candidates based on motion information of inter prediction mode.

[0006] The present disclosure aims to improve encoding / decoding performance by considering various directional modes.

[0007] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0008] According to the present disclosure, a video decoding method and device include the steps of: generating an inter prediction block of a current block from a reference picture; generating an intra prediction block of the current block based on one or more intra prediction mode candidates; and generating a final prediction block of the current block by weighting the inter prediction block and the intra prediction block, wherein the one or more intra prediction mode candidates include a first intra prediction mode candidate derived based on the inter prediction block, and the inter prediction block can be generated based on motion information of the inter prediction mode candidate of the current block.

[0009] In the video decoding method and device according to the present disclosure, the one or more intra prediction mode candidates can be derived by applying a filter capable of producing at least one of a slope or a directionality to the inter prediction block.

[0010] In the video decoding method and device according to the present disclosure, the one or more intra prediction mode candidates can be derived by applying a filter to the entire area or a part of the inter prediction block.

[0011] In the video decoding method and device according to the present disclosure, the first intra prediction mode candidate may be an intra prediction mode having an angle that is the same as or most similar to an angle formed by a slope of the inter prediction block.

[0012] In the video decoding method and device according to the present disclosure, the one or more intra prediction mode candidates further include a second intra prediction mode candidate and a third intra prediction mode candidate, and the third intra prediction mode candidate can be derived by performing an average or weighted sum operation on the first intra prediction mode candidate and the second intra prediction mode candidate.

[0013] In the video encoding method and device according to the present disclosure, the one or more intra prediction mode candidates further include a fourth intra prediction mode candidate which is a default mode, and the default mode may be a planar mode.

[0014] In the video decoding method and device according to the present disclosure, when the filter is applied to the entire region of the inter prediction block, the filter may have a size smaller than the inter prediction block, and the first intra prediction mode candidate may be the one having the highest amplitude on a histogram obtained by repeatedly applying the filter to the inter prediction block.

[0015] In the image decoding method and device according to the present disclosure, the filters may be at least two different ones.

[0016] According to the present disclosure, a video encoding method and device include the steps of: generating an inter prediction block of a current block from a reference picture; generating an intra prediction block of the current block based on one or more intra prediction mode candidates; and generating a final prediction block of the current block by weighting the inter prediction block and the intra prediction block, wherein the one or more intra prediction mode candidates include a first intra prediction mode candidate derived based on the inter prediction block, and the inter prediction block can be generated based on motion information of the inter prediction mode candidate of the current block.

[0017] In the present disclosure, a recording medium for storing a bitstream generated by the image encoding method may be provided.

[0018] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0019] According to the present disclosure, there is an effect of increasing the number of intra prediction mode candidates by deriving intra prediction mode candidates based on motion information of inter prediction mode.

[0020] According to the present disclosure, there is an effect of improving encoding / decoding performance by considering modes in various directions.

[0021] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0022] FIG. 1 is a block diagram showing the configuration of an image encoding method and device according to one embodiment of the present invention.

[0023] FIG. 2 is a block diagram showing the configuration of an image decoding method and device according to one embodiment of the present invention.

[0024] Figure 3 illustrates the positions of the left and upper blocks for determining weights when calculating the weighted sum to generate a prediction block for the current block.

[0025] FIG. 4 is a flowchart of an image encoding / decoding method based on a predetermined intra prediction mode derived based on motion information of an inter prediction block, as an embodiment to which the present invention is applied.

[0026] FIG. 5 is a block diagram of an image encoding device that performs an image encoding method based on a predetermined intra prediction mode derived based on motion information of an inter prediction block, as an embodiment to which the present invention is applied.

[0027] FIG. 6 is a block diagram of an image decoding device that performs an image decoding method based on a predetermined intra prediction mode derived based on motion information of an inter prediction block, as an embodiment to which the present invention is applied.

[0028] FIG. 7 is a diagram illustrating an example of generating an inter prediction block as an embodiment to which the present invention is applied.

[0029] FIG. 8 illustrates an example of an inter prediction candidate list for generating an inter prediction block according to an embodiment of the present invention.

[0030] FIG. 9 is a diagram showing an example of generating an intra prediction mode candidate list by applying a Sobel filter as an embodiment to which the present invention is applied.

[0031] FIG. 10 illustrates an immersive video encoding / decoding device according to one embodiment of the present invention.

[0032] FIG. 11 is a drawing showing the differences between an atlas image and a natural image according to one embodiment of the present invention.

[0033] The present disclosure is susceptible to various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, but rather to include all modifications, equivalents, and substitutes falling within the spirit and scope of the present disclosure. In the drawings, similar reference numerals designate the same or similar functions throughout. The shapes and sizes of elements in the drawings may be exaggerated for clarity. The detailed description of the exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments, while different from each other, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure. Furthermore, it should be understood that the positions or arrangements of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the exemplary embodiments, if properly described, is defined only by the appended claims, along with the full scope equivalents to which such claims are entitled.

[0034] While terms such as "first" and "second" may be used herein to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0035] When a component of the present disclosure is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening therein. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components intervening therein.

[0036] The components shown in the embodiments of the present disclosure are independently depicted to represent different characteristic functions, and do not imply that each component is composed of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two components among each component may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separate embodiments of each component are also included in the scope of the present disclosure as long as they do not deviate from the essence of the present disclosure.

[0037] The terminology used in this disclosure is only used to describe specific embodiments and is not intended to limit the present disclosure. The singular expression includes plural expressions unless the context clearly indicates otherwise. In this disclosure, it should be understood that terms such as "comprise" or "have" are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. In other words, the description of a specific configuration in this disclosure as "comprising" does not exclude configurations other than the specified configuration, and means that additional configurations may be included in the scope of the implementation or technical idea of ​​the present disclosure.

[0038] Some components of the present disclosure may not be essential components that perform essential functions of the present disclosure, but may be optional components merely for performance enhancement. The present disclosure may be implemented by including only components essential to implementing the essence of the present disclosure, excluding components used solely for performance enhancement. A structure that includes only essential components, excluding optional components used solely for performance enhancement, is also within the scope of the present disclosure.

[0039] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if a detailed description of a related known configuration or function is judged to obscure the gist of this specification, the detailed description will be omitted. The same reference numerals will be used for identical components in the drawings, and duplicate descriptions of the same components will be omitted.

[0040] First, the terms used in this application are briefly explained as follows.

[0041] Atlas can perform pruning on multiple input images and extract patches from the pruned images. The extracted patches can be packed into a single image, which can refer to the image into which the patches are packed.

[0042] An atlas can consist of a texture atlas and a depth atlas. The texture atlas and depth atlas can be independently encoded and decoded. A texture atlas can also be referred to as an attribute atlas.

[0043] Hereinafter, with reference to the attached drawings, embodiments of the present invention will be described in more detail.

[0044] FIG. 1 is a block diagram showing the configuration of an image encoding device according to one embodiment of the present disclosure.

[0045] Referring to FIG. 1, the image encoder according to the present disclosure may include a picture segmentation unit (110), a prediction unit (120), a transformation unit (130), a quantization unit (135), a reordering unit (160), an entropy encoding unit (165), an inverse quantization unit (140), an inverse transformation unit (145), an in-loop filter unit (150), and a restored picture buffer (155).

[0046] The picture segmentation unit (110) can segment the input current picture into at least one processing unit. At this time, the processing unit may be a prediction unit (PU), a transform unit (TU), or a coding unit (CU). Hereinafter, in the embodiment of the present invention, the coding unit may be used to mean a unit that performs encoding or may be used to mean a unit that performs decoding.

[0047] The prediction unit (120) may further include an encoding mode determination unit in addition to an inter-prediction unit that performs inter-prediction or inter-screen prediction, and an intra-prediction unit that performs intra-prediction or intra-screen prediction. The encoding mode determination unit may determine whether the encoding mode of the prediction unit is inter-prediction or intra-prediction. The residual value (residual block) between the generated prediction block and the original block may be input to the transformation unit (130).

[0048] Additionally, prediction mode information, motion vector information, etc. used for prediction can be encoded in an entropy encoding unit (165) together with a residual value and transmitted to a decoding device.

[0049] The inter prediction unit may predict a prediction unit based on information from at least one picture among the previous picture or the next picture of the current picture, or in some cases, may predict a prediction unit based on information from a portion of a region within the current picture for which encoding has been completed. The inter prediction unit may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.

[0050] In the reference picture interpolation unit, the restored picture stored in the restored picture buffer (155) can be used as a reference picture. Reference picture information can be provided from the restored picture buffer and pixel information below an integer pixel can be generated from the reference picture. In the case of luminance pixels, a DCT-based 8-tap interpolation filter with different filter coefficients can be used to generate pixel information below an integer pixel in units of 1 / 4 pixels. In the case of a chrominance signal, a DCT-based 4-tap interpolation filter with different filter coefficients can be used to generate pixel information below an integer pixel in units of 1 / 8 pixels.

[0051] The motion prediction unit can perform motion prediction based on a reference picture interpolated by the reference picture interpolation unit. Various methods such as FBMA (Full Search-based Block Matching Algorithm), TSS (Three Step Search), and NTS (New Three-Step Search Algorithm) can be used to derive a motion vector. The motion vector can have a motion vector value in units of 1 / 2 or 1 / 4 pixels based on the interpolated pixel. The motion prediction unit can predict a current prediction unit by using a different motion prediction method. The motion prediction method can include at least one of a skip method, a merge method, a GPM mode (Geometric Partitioning Mode), an AMVP mode, an affine mode, and an affine merge mode.

[0052] The intra prediction unit can generate a prediction unit based on reference pixel information surrounding the current block, which is pixel information within the current picture. If a neighboring block of the current prediction unit is a block on which inter prediction has been performed and the reference pixel is a pixel on which inter prediction has been performed, the reference pixel included in the block on which inter prediction has been performed can be replaced and used with reference pixel information of a neighboring block on which intra prediction has been performed. In other words, if a reference pixel is not available, the unavailable reference pixel information can be replaced and used with at least one reference pixel among the available reference pixels.

[0053] Additionally, a residual block including residual value information, which is the difference between the prediction unit that performed the prediction based on the prediction unit generated in the prediction unit (120) and the original block of the prediction unit, can be generated. The generated residual block can be input to the transformation unit (130).

[0054] In the transformation unit (130), the residual block including the residual value information of the prediction unit generated through the original block and the prediction unit (120) can be transformed using a transformation method such as DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), or KLT. Whether to apply DCT, DST, or KLT to transform the residual block can be determined based on the intra prediction mode information of the prediction unit used to generate the residual block.

[0055] The quantization unit (135) can quantize values ​​converted to the frequency domain by the transformation unit (130). The quantization coefficients may vary depending on the block or the importance of the image. The values ​​produced by the quantization unit (135) can be provided to the dequantization unit (140) and the reordering unit (160).

[0056] The reordering unit (160) can reorder coefficient values ​​for quantized residual values. The reordering unit (160) can change a two-dimensional block-shaped coefficient into a one-dimensional vector form through a coefficient scanning method. For example, the reordering unit (160) can change the two-dimensional block-shaped coefficient into a one-dimensional vector form by scanning from the DC coefficient to the coefficient of the high-frequency region using a zig-zag scan method. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional block-shaped coefficient in the column direction and a horizontal scan that scans the two-dimensional block-shaped coefficient in the row direction may be used. That is, depending on the size of the transformation unit and the intra prediction mode, it is possible to determine which scanning method among the zig-zag scan, the vertical scan, and the horizontal scan is to be used.

[0057] The entropy encoding unit (165) can perform entropy encoding based on the values ​​produced by the rearrangement unit (160). Entropy encoding can use various encoding methods such as, for example, Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC). The entropy encoding unit (165) can encode various information such as residual value coefficient information of an encoding unit, block type information, prediction mode information, division unit information, prediction unit information, transmission unit information, motion vector information, reference frame information, block interpolation information, and filtering information from the rearrangement unit (160) and the prediction unit (120).

[0058] The inverse quantization unit (140) and the inverse transformation unit (145) inversely quantize the values ​​quantized in the quantization unit (135) and inversely transform the values ​​transformed in the transformation unit (130). The residual values ​​generated in the inverse quantization unit (140) and the inverse transformation unit (145) can be combined with the predicted prediction units through the motion estimation unit, motion compensation unit, and intra prediction block determination unit (630) included in the prediction unit (120) to generate a reconstructed block.

[0059] The in-loop filter unit (150) may include at least one of a deblocking filter, an offset correction unit, and an ALF (Adaptive Loop Filter). The deblocking filter may remove block distortion caused by boundaries between blocks in a restored picture. The offset correction unit may correct the offset from the original image on a pixel-by-pixel basis for the image on which deblocking has been performed. In order to perform offset correction for a specific picture, a method may be used in which the pixels included in the image are divided into a certain number of regions, the regions to be offset are determined, and the offset is applied to the regions, or the offset is applied by considering edge information of each pixel. The ALF (Adaptive Loop Filter) may be performed based on a value obtained by comparing the filtered restored image with the original image. After dividing the pixels included in the image into a predetermined group, one filter to be applied to the groups may be determined, and filtering may be performed differentially for each group.

[0060] The restoration picture buffer (155) can store restoration blocks or pictures produced through the in-loop filter unit (150), and the stored restoration blocks or pictures can be provided to the prediction unit (120) when performing inter prediction.

[0061] FIG. 2 is a block diagram showing the configuration of an image decoding device according to one embodiment of the present disclosure.

[0062] Referring to FIG. 2, the image decoding device according to the present disclosure may include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transformation unit (230), a prediction unit (240), an in-loop filter unit (250), and a restored picture buffer (260).

[0063] When a video bit stream is input to a video encoder, the input bit stream can be decoded in the opposite procedure to that of the video encoder.

[0064] The entropy decoding unit (210) can perform entropy decoding in a procedure opposite to that of the entropy encoding unit of the video encoder. For example, various methods such as Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC) can be used.

[0065] The entropy decoding unit (210) can decode information related to intra prediction and inter prediction performed in the encoder.

[0066] The quantized coefficients can be inversely quantized in the inverse quantization unit (220), and the inverse transform unit (230) can perform inverse transform, i.e., inverse DCT, inverse DST, and inverse KLT, on the transforms, i.e., DCT, DST, and KLT, performed in the transform unit for the quantization result performed in the image encoder. The inverse transform can be performed based on the transmission unit determined in the image encoder. In the inverse transform unit (230) of the image decoding device, a transform technique (e.g., DCT, DST, KLT) can be selectively performed according to a plurality of pieces of information, such as a prediction method, a size of a current block, and a prediction direction.

[0067] The prediction unit (240) can generate a prediction block based on prediction block generation related information provided by the entropy decoding unit (210) and previously decoded block or picture information provided by the restoration picture buffer (260).

[0068] The prediction unit (240) may include an encoding mode determination unit, an intra prediction unit, and an inter prediction unit. The encoding mode determination unit may receive various information such as prediction unit information input from the entropy decoding unit (210), prediction mode information of an intra prediction method, and motion prediction-related information of an inter prediction method, and may determine whether the current block performs inter prediction or intra prediction.

[0069] On the other hand, if the encoder does not transmit motion prediction-related information for the inter prediction, but instead transmits information indicating that motion information is to be derived and used on the side of the decoding device and information about the technique used to derive the motion information, the prediction unit determination unit determines the prediction performance of the inter prediction unit based on the information transmitted from the encoder.

[0070] The intra prediction unit can generate a prediction block based on pixel information within the current picture. If the prediction unit is a prediction unit that has performed intra prediction, intra prediction can be performed based on intra prediction mode information of the prediction unit provided by the image encoder.

[0071] The intra prediction unit may include an Adaptive Intra Smoothing (AIS) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter is a unit that performs filtering on the reference pixels of the current block, and can determine whether to apply the filter based on the prediction mode of the current prediction unit and apply it. AIS filtering can be performed on the reference pixels of the current block using the prediction mode and AIS filter information of the prediction unit provided by the image encoder. If the prediction mode of the current block is a mode that does not perform AIS filtering, the AIS filter may not be applied.

[0072] The reference pixel interpolation unit can interpolate the reference pixel to generate a reference pixel of a pixel unit less than an integer value when the prediction mode of the prediction unit is a prediction unit that performs intra prediction based on the pixel value interpolated from the reference pixel. When the prediction mode of the current block is a prediction mode that generates a prediction block without interpolating the reference pixel, the reference pixel may not be interpolated. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is the DC mode.

[0073] The inter prediction unit can perform inter prediction on the current prediction unit based on information included in at least one picture among the previous picture or the subsequent picture of the current picture including the prediction unit, using information required for inter prediction of the current prediction unit provided by the image encoder. In order to perform inter prediction, it can be determined based on the coding unit whether the motion prediction method of the prediction unit included in the coding unit is a skip method, a merge method, a GPM mode (Geometric Partitioning Mode), an AMVP mode (AMVP Mode), an affine mode, or an affine merge mode.

[0074] The restored block or picture may be provided to an in-loop filter unit (250). The in-loop filter unit (250) may include a deblocking filter, an offset correction unit, and an ALF.

[0075] Information about whether a deblocking filter has been applied to a corresponding block or picture can be received from a video encoder, and if a deblocking filter has been applied, information about whether a strong or weak filter has been applied. The deblocking filter of a video decoding device can receive information related to the deblocking filter provided by the video encoder and perform deblocking filtering on the corresponding block in the video decoding device.

[0076] The offset correction unit can perform offset correction on the restored image based on the type of offset correction applied to the image during encoding and offset value information. ALF can be applied to the encoding unit based on information on whether ALF is applied and ALF coefficient information provided from the encoder. This ALF information can be provided by being included in a specific parameter set.

[0077] The restored picture buffer (260) can store restored pictures or blocks so that they can be used as reference pictures or reference blocks, and can also provide restored pictures to the output unit.

[0078] The Combined Inter-Intra Prediction (CIIP) method may refer to a method of generating a corrected third prediction block by correcting a first prediction block that has been motion compensated according to motion information using a second prediction block generated using an intra prediction mode. In other words, it may refer to a method of generating a third prediction block, which is a final prediction block, by mixing a first prediction block predicted according to an inter prediction mode and a second prediction block predicted according to an intra prediction mode.

[0079] When generating a prediction block of a current block, encoding efficiency can be improved when a corrected prediction block is generated by correcting the first prediction block based on the second prediction block.

[0080] The generation of a third prediction block by correcting the first prediction block using the second prediction block can be performed by multiplying each sample of the first and second prediction blocks by a weight. For example, this can be performed using the following mathematical expression 1.

[0081]

[0082] Mathematical expression 1 is a formula for generating a prediction block of the current block through a weighted sum operation of prediction blocks.

[0083] w intra and w inter may refer to the weight applied to the intra prediction mode and the weight applied to the inter prediction mode, respectively.

[0084] For example, the weights can be determined based on whether intra predictions are made for the left and upper blocks that have been decoded / encoded relative to the current block.

[0085] Figure 3 is a diagram showing the positions of the left and upper blocks for determining weights when generating a prediction block for the current block through a weighted sum operation. The weights for generating a prediction block for the current block can be determined depending on whether the left and upper blocks illustrated in Figure 3 are intra-predicted.

[0086] For example, if the prediction modes of the left and top blocks are both determined as intra prediction modes, w intra and w inter can be determined as 3 and 1, respectively. Alternatively, if the prediction mode of either the left or top block is determined as the intra prediction mode, w intra and w inter can be determined as 2 respectively. Or, if the prediction modes of the left and upper blocks are both determined as inter prediction modes, w intra and w inter can be determined as 1 and 3, respectively. The determined weights can be explicitly signaled in the encoding device or implicitly derived in the decoding device.

[0087] However, the above-described figures are only examples and are not limited thereto.

[0088] In general, in CIIP mode, the intra prediction mode of the current block can be determined as Planar mode. However, if the intra prediction mode of the current block in CIIP mode is used by considering modes in various directions, the compression performance of the video encoding / decoding device can be improved. Accordingly, the present disclosure provides a method for determining the intra prediction mode of the current block based on motion information of an inter prediction block and generating a final prediction block of the current block. Hereinafter, the method according to the present disclosure will be described in detail.

[0089] FIG. 4 is a flowchart of an image encoding / decoding method based on a predetermined intra prediction mode derived based on motion information of an inter prediction mode, as an embodiment to which the present invention is applied.

[0090] According to one embodiment of the present disclosure, a video encoding / decoding method based on a predetermined intra prediction mode derived based on motion information of an inter prediction mode can be understood to be equally applicable to an encoding device and a decoding device.

[0091] Referring to FIG. 4, an inter prediction block of the current block can be generated (S410).

[0092] Generation of an inter prediction block can be performed based on an inter prediction mode determined through cost comparison between multiple inter prediction modes.

[0093] Inter prediction can be performed through motion prediction and motion compensation. Here, the motion prediction method can include at least one of the Skip method, the Merge method, the GPM mode (Geometric Partitioning Mode), the AMVP mode, the Affine mode, and the Affine Merge Mode, and any one of the above methods can be determined as the inter prediction mode of the current block.

[0094] In addition to the above method, the CIIP mode may be determined as the inter-prediction mode of the current block. This determination may be performed based on a predetermined parameter. The predetermined parameter may be encoded and signaled, or may be derived by the decoding device based on a predetermined parameter.

[0095] An inter-prediction mode candidate list can be constructed based on the determined inter-prediction mode. An inter-prediction block of a current block can be generated based on at least one inter-prediction mode candidate in the inter-prediction mode candidate list. An encoding device can encode index information specifying at least one of a plurality of candidates. A decoding device can specify at least one of the plurality of candidates based on index information signaled through a bitstream. Inter-prediction information of the current block can be derived based on the specified candidate. Inter-prediction can be performed based on the derived inter-prediction information.

[0096] If the encoding mode of the current block is determined to be CIIP mode, the current block can use the merge candidate list as an inter prediction mode candidate list.

[0097] In the case where the encoding mode of the current block is determined as the CIIP mode, an embodiment of the inter prediction mode candidate list is specifically examined in FIG. 8.

[0098] Referring to FIG. 4, an intra prediction block of a current block can be generated based on one or more intra prediction mode candidates (S420).

[0099] The intra prediction mode of the current block can be determined by any mode used by the video codec. For example, it can be determined as either directional or non-directional mode. Non-directional modes can include Planar mode and DC mode.

[0100] The directional modes may include modes 2 to 66 having angles from the lower left diagonal direction to the upper right diagonal direction. In particular, modes 2, 34, and 66 may be defined as diagonal modes, and the intra prediction mode between modes 2 and 34 may be defined as a mode having horizontal directionality, and the intra prediction mode between modes 34 and 66 may be defined as a mode having vertical directionality, respectively.

[0101] However, the intra prediction modes illustrated above are only examples, and a smaller number of intra prediction modes may be defined, or conversely, a larger number of intra prediction modes may be defined.

[0102] An intra prediction block can be generated by selecting at least one of the intra prediction modes disclosed above as a candidate. The process of deriving intra prediction mode candidates will now be examined in detail.

[0103] One or more intra prediction mode candidates can be derived based on the inter prediction block generated in S410. More specifically, one or more intra prediction mode candidates can be derived by applying a filter to the inter prediction block generated in S410.

[0104] Specifically, one or more intra prediction mode candidates can be derived by applying a filter to an inter prediction block to calculate the gradient of pixel values ​​between pixels. The filter can be a Sobel filter or a Gaussian filter, or a filter having a value that is changed to an arbitrary value. Here, the size of the filter can be NxM. Here, N and M can be integers greater than or equal to 0, and N and M can have the same value or different values.

[0105] According to one embodiment of the present disclosure, one or more intra prediction mode candidates can be derived by performing an inverse tangent operation.

[0106] According to one embodiment of the present disclosure, a filter may be applied to the entire region of an inter-prediction block. When the filter is applied to the entire region of the inter-prediction block, the filter may have a size smaller than the inter-prediction block.

[0107] According to one embodiment of the present disclosure, a filter may be applied to a portion of an inter prediction block.

[0108] According to one embodiment of the present disclosure, the intra prediction mode induced through the filter may be one or multiple.

[0109] For example, if there is only one intra prediction mode derived through the above filter, only one intra prediction mode can be derived and used as a candidate.

[0110] For example, if there are multiple intra prediction modes induced through the above filter, all of the multiple intra prediction modes can be induced as intra prediction mode candidates for the current block.

[0111] For example, when there are multiple intra prediction modes induced through the above filter, an intra prediction mode obtained through a weighted sum operation for the multiple intra prediction modes can be induced as an intra prediction mode candidate for the current block.

[0112] For example, when there are multiple intra prediction modes induced through the above filter, only some of the multiple intra prediction modes can be induced as intra prediction mode candidates for the current block.

[0113] According to one embodiment of the present disclosure, one filter may be applied to an inter prediction block, or at least two filters may be combined to derive an intra prediction mode.

[0114] According to one embodiment of the present disclosure, one or more intra prediction mode candidates can be derived by applying a filter to an inter prediction block to derive a directionality.

[0115] According to one embodiment of the present disclosure, one or more intra prediction mode candidates can be derived by applying a filter to an inter prediction block to produce a gradient and / or a directionality.

[0116] According to one embodiment of the present disclosure, an intra prediction mode can be derived by applying N filters capable of calculating gradients and / or directionality for an inter prediction block, and performing any operation capable of averaging, weighting, or combining the derived values.

[0117] As with the encoding device, the decoding device can also derive an intra prediction mode using the same method. Alternatively, if there are multiple derived intra prediction mode candidates, the encoding device can encode index information that specifies at least one of the multiple candidates. The decoding device can specify at least one of the multiple candidates based on the index information signaled through the bitstream. Based on the specified candidate, the intra prediction information of the current block can be derived. Intra prediction can be performed based on the derived intra prediction information.

[0118] If a candidate for intra prediction is not generated through filtering according to the present disclosure, the CIIP mode may be omitted from the encoding mode of the current block.

[0119] As another example, if filtering according to the present disclosure fails to generate a candidate for intra prediction, the intra prediction mode of the current block may be determined as a default mode among the intra prediction mode candidates. For example, the intra prediction mode of the current block may be determined as Planar mode. Accordingly, an intra prediction block may be generated based on the Planar mode.

[0120] Referring to FIG. 4, the final prediction block of the current block can be generated by weighting the inter-prediction block and intra-prediction block of the current block (S430).

[0121] The encoding / decoding device can generate a final prediction block based on the encoding mode of the current block. If the encoding mode of the current block is determined to be the CIIP mode, the final prediction block can be generated by correcting a first prediction block generated based on the inter-prediction mode with a second prediction block generated based on the intra-prediction mode.

[0122] FIG. 5 is a block diagram of an image encoding device that performs an image encoding method based on a predetermined intra prediction mode derived based on motion information of an inter prediction block, as an embodiment to which the present invention is applied.

[0123] An encoding device performing a method according to one embodiment of the present disclosure may include a prediction unit (120). The prediction unit (120) may include an inter-prediction block generation unit (510), an intra-prediction block generation unit (520), and a final prediction block generation unit (530).

[0124] The inter prediction block generation unit (510) can perform the operation of S410 as described with reference to FIG. 4, and a detailed description thereof is omitted here.

[0125] The intra prediction block generation unit (520) can perform the operation of S420 as described with reference to FIG. 4, and a detailed description thereof is omitted here.

[0126] The final prediction block generation unit (530) can perform the operation of S430 as described with reference to FIG. 4, and a detailed description thereof is omitted here.

[0127] FIG. 6 is a block diagram of an image decoding device that performs an image decoding method based on a predetermined intra prediction mode derived based on motion information of an inter prediction block, as an embodiment to which the present invention is applied.

[0128] An encoding device performing a method according to one embodiment of the present disclosure may include a prediction unit (240). The prediction unit (240) may include an inter-prediction block generation unit (610), an intra-prediction block generation unit (620), and a final prediction block generation unit (630).

[0129] The inter prediction block generation unit (610) can perform the operation of S410 as described with reference to FIG. 4, and a detailed description thereof is omitted here.

[0130] The intra prediction block generation unit (620) can perform the operation of S420 as described with reference to FIG. 4, and a detailed description thereof is omitted here.

[0131] The final prediction block generation unit (630) can perform the operation of S430 as described with reference to FIG. 4, and a detailed description thereof is omitted here.

[0132] FIG. 7 relates to one embodiment of generating an inter prediction block.

[0133] Inter prediction generates prediction blocks using motion vectors and the restored image stored in the Decoded Picture Buffer (DPB), and is achieved through motion estimation and motion compensation.

[0134] Motion estimation can refer to the process of finding the optimal prediction block from pictures stored in the restored picture buffer and used for inter prediction, and motion compensation can refer to generating a prediction block based on motion information (motion vector, reference picture index, etc.) for the optimal prediction block found in the motion estimation process.

[0135] Additionally, inter prediction can be divided into unidirectional prediction, which uses only one past or future picture as a reference picture on the temporal axis for a single block, and bidirectional prediction, which simultaneously references past and future pictures. This can be determined by the slice type.

[0136] In inter prediction, one slice can be divided into a P-slice (Predictive Slice) and a B-slice (Bi-predictive Slice). Inter prediction of a P-slice only performs unidirectional prediction, and inter prediction of a B-slice can selectively use either unidirectional prediction or bidirectional prediction.

[0137] Additionally, unidirectional prediction can be divided into forward direction prediction using one reference picture temporally output before the current picture, and backward direction prediction using one reference picture temporally output after the current picture.

[0138] If the encoding mode of the current block is determined to be the CIIP mode, the inter prediction mode of the current block can be derived to the merge mode.

[0139] Specifically, if the inter prediction mode of the current block is derived as a merge mode, the motion information of the surrounding blocks can be applied to the current block in the same manner to generate a prediction block of the current block. The encoding device can encode the motion information of the surrounding blocks and signal it to the decoding device.

[0140] Here, the surrounding block can be at least one of the left block, the top block, the bottom left block, the top right block, or the top left block.

[0141] For example, at least one of the motion information of the blocks located on the left and bottom left may be used. For example, at least one of the motion information of the blocks located on the top and top right may be used. For example, the motion information of the block located on the top left may be used.

[0142] Meanwhile, based on the determined inter prediction mode, an inter prediction mode candidate list can be constructed. If the encoding mode of the current block is determined to be CIIP mode, the merge candidate list can be used as the inter prediction mode candidate list.

[0143] The above merge candidate list may include N candidates, where N may be an integer greater than or equal to 1.

[0144] Candidates in the above merge candidate list may include spatial merge candidates, temporal merge candidates, history-based merge candidates, pair-wise average merge candidates, zero-vector merge candidates, etc.

[0145] The following figure 8 is an example of a merge candidate list among the inter prediction mode candidate lists.

[0146] Referring to FIG. 8, each merge_idx represents an index identifying each candidate. Each inter prediction mode candidate may include motion information (e.g., a motion vector and a reference picture index).

[0147] L0 and L1 illustrated in Fig. 8 simply represent reference picture lists for L0 prediction and L1 prediction, respectively. L0 prediction may refer to a prediction that refers to the reference picture list L0, and L1 prediction may refer to a prediction that refers to the reference picture list L1.

[0148] In the example merge candidate list, merge_idx 0 only has values ​​for L0, so it can only be used for L0 prediction. merge_idx 1 only has values ​​for L1, so it can only be used for L1 prediction. merge_idx 4 has values ​​for both L0 and L1, so it can be used for both L0 and L1 prediction.

[0149] If the encoding mode of the current block is determined to be the CIIP mode, the inter prediction block of the current block can be derived based on the motion information of the merge candidate in the merge candidate list.

[0150] For example, an inter prediction block can be generated based on the motion information of a candidate whose merge_idx is 0 as shown in FIG. 8. However, the above-described example is merely an example and is not limited thereto.

[0151] Meanwhile, when an inter prediction block of a current block is generated according to the present disclosure, a predetermined filter may be applied to derive an intra prediction mode of the current block.

[0152] The amount of change in pixel values ​​within an inter-prediction block can be derived based on the above-described filter. That is, the slope and / or directionality of an inter-prediction block can be derived based on the above-described filter.

[0153] Figure 9 is a drawing showing an example in which a Sobel filter is applied as a filter applied to an inter prediction block.

[0154] A single filter or multiple filters may be applied to an inter prediction block.

[0155] When a single filter is applied, the filter may be applied to the entire region or a portion of the inter-prediction block.

[0156] When a filter is applied to the entire area of ​​an inter-prediction block, the size of the filter may be smaller than or equal to the size of the inter-prediction block. When a filter having a size smaller than the size of the inter-prediction block is applied, the filter may be repeatedly applied to the inter-prediction block.

[0157] By repeatedly applying the above filter, an X-axis slope and / or a Y-axis slope can be obtained. The amplitude can be obtained through the absolute value of each slope. In addition, by performing an inverse tangent operation based on the slope, a corresponding intra prediction mode can be derived.

[0158] Referring to Fig. 9, for example, the X-axis direction slope can be calculated as 190, the Y-axis direction slope can be calculated as -90, and the amplitude can be obtained through the absolute value of the slope. In addition, by performing an inverse tangent operation based on the X-axis direction slope and the Y-axis direction slope, the corresponding mode 60 can be derived.

[0159] Meanwhile, a histogram can be obtained by accumulating the above amplitudes. From the histogram, the intra prediction mode with the largest amplitude can be selected as a candidate.

[0160] When multiple filters are applied, the filters may be applied to the entire region or a portion of the inter-prediction block. Multiple filters may refer to two or more different filters. Multiple different filters may be applied to the same location in the inter-prediction block.

[0161] There may be at least one gradient obtained by applying a filter to an inter prediction block.

[0162] If the slope is 1, the intra prediction mode corresponding to the slope can be selected as an intra prediction mode candidate.

[0163] When there are two or more slopes, multiple intra prediction modes corresponding to each of the multiple slopes can be selected as candidates.

[0164] Referring to Fig. 9, the predictor on the left side of the drawing is a block with a size of 8x8. A Sobel filter can be applied to the block. The Sobel filter can use at least two filter kernels including a Gx kernel and a Gy kernel. By applying the Sobel filter, an X-axis direction gradient value and a Y-axis direction gradient value can be calculated, and an intra prediction mode can be derived based on the gradient value.

[0165] Below, with respect to Table 1, we will examine in detail the process of deriving the corresponding angle based on the slope value and deriving the intra prediction mode.

[0166] predModeIntra-14-13-12-11-10-9-8-7-6-5-4-3-2-1234intraPredAngle5123412561711281028673645751453935322926predModeIntra567891011121314151617 18192021intraPredAngle232018161412108643210-1-2-3predModeIntra2223242526272829303132333435363738intraPredAngle-4-6-8-10-12-14-16-18-20-23 -26-29-32-29-26-23-20predModeIntra3940414243444546474849505152535455intraPredAngle-18-16-14-12-10-8-6-4-3-2-1012346predModeIntra565758596 0616263646566676869707172intraPredAngle810121416182023262932353945515764predModeIntra7374757677787980intraPredAngle7386102128171256341512

[0167] [Table 1] shows an example to which the present invention is applied, in which intraPredAngle represents an angle derived as a result of performing an inverse tangent operation on the X-axis direction slope and the Y-axis direction slope. predModeIntra in [Table 1] is an intra prediction mode corresponding to intraPredAngle. predModeIntra may be an intra prediction mode having an angle that is the same as or most similar to intraPredAngle.

[0168] Meanwhile, an intra prediction mode obtained by performing an average or weighted sum operation on multiple intra prediction mode candidates that have already been derived can be added as a candidate. For example, if the intra prediction mode candidate list includes modes 8, 10, and 12, mode 9 derived by averaging the indices of modes 8 and 10, and mode 11 derived by averaging the indices of modes 10 and 12 can be further included as candidates. The added candidates can be inserted into the list in a pre-defined order in the encoding device, and can be assigned indices within the list in a pre-defined order. The decoding device can reference the intra prediction mode candidate through the signaled index.

[0169] However, the above disclosed embodiments are merely examples and may have different values.

[0170] Meanwhile, the above-described embodiments can be more effectively used to determine the prediction mode of atlas images rather than natural images.

[0171] Fig. 10 illustrates an immersive video encoding / decoding device according to an embodiment of the present invention. The immersive video encoding / decoding device according to the present disclosure may also be interpreted as TMIV. Referring to Fig. 8, the immersive video encoding / decoding device according to the present disclosure can encoding / decoding a texture atlas, a depth atlas, and / or metadata by receiving texture information, depth information, and / or metadata sub-bitstreams.

[0172] The immersive video encoding / decoding device according to the present disclosure can perform preprocessing on a multi-view video to generate a small number of atlas images and perform compression using HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), etc.

[0173] However, most of the test sequences used as common test conditions (CTC) during the standardization process of traditional video codecs are natural images, and there is a problem in that the characteristics of the atlas image generated by the immersive image encoding / decoding device according to the present disclosure are not taken into account.

[0174] Figure 11 is a diagram showing the differences between atlas images and natural images.

[0175] Compared to natural images that contain brightness information and color information, atlas images can contain texture information and depth information.

[0176] As illustrated in Fig. 11, each patch included in the atlas has a distinct boundary between a valid area and an invalid area. That is, the boundary between the valid area and the invalid area has a specific directionality, and thus, when applying the embodiments described in the present disclosure, the prediction mode of the current block can be effectively determined.

[0177] The components described in the exemplary embodiments of the present disclosure may be implemented by hardware elements. For example, the hardware elements may include at least one of a digital signal processor (DSP), a processor, a controller, an application-specific integrated circuit (ASIC), a programmable logic element such as an FPGA, a graphics processing unit (GPU), other electronic devices, or a combination thereof. At least some of the functions or processes described in the exemplary embodiments of the present disclosure may be implemented in software, and the software may be recorded on a recording medium. The components, functions, and processes described in the exemplary embodiments may be implemented by a combination of hardware and software.

[0178] A method according to one embodiment of the present disclosure may be implemented as a program that can be executed by a computer, and the computer program may be recorded on various recording media such as a magnetic storage medium, an optical readable medium, a digital storage medium, etc.

[0179] The various technologies described in this disclosure may be implemented as digital electronic circuits or computer hardware, firmware, software, or a combination thereof. The technologies may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information medium (e.g., a machine-readable storage device (e.g., a computer-readable medium) or a data processing device), or a computer program embodied as a signal propagated to be processed by a data processing device or to cause the operation of a data processing device (e.g., a programmable processor, a computer, or multiple computers).

[0180] The computer program(s) may be written in any programming language, including compiled or interpreted languages, and may be distributed in any form, including standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. The computer program(s) may be executed by a single computer, or by multiple computers distributed across one or more sites and interconnected by a communications network.

[0181] Examples of processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and one or more processors of a digital computer. Typically, a processor receives instructions and data from read-only memory, random-access memory, or both. Components of a computer may include at least one processor for executing instructions, and one or more memory devices for storing instructions and data. Additionally, the computer may include one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or may be connected to such mass storage devices to receive and / or transmit data. Examples of information media suitable for implementing computer program instructions and data include semiconductor memory devices (e.g., magnetic media such as hard disks, floppy disks, and magnetic tape), optical media such as compact disc read-only memories (CD-ROMs), digital video discs (DVDs), magneto-optical media such as floptical disks, and read-only memory (ROM), random access memory (RAM), flash memory, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and other known computer-readable media. The processor and memory may be supplemented by, or integrated with, special purpose logic circuitry.

[0182] A processor can execute an operating system (OS) and one or more software applications running on the OS. The processor device can also access, store, manipulate, process, and generate data in response to the software execution. For simplicity, the processor device is described singularly; however, those skilled in the art will understand that the processor device may include multiple processing elements and / or different types of processing elements. For example, the processor device may include multiple processors or a processor and a controller. Additionally, the processor device may configure different processing structures, such as parallel processors. Furthermore, a computer-readable medium refers to any medium that a computer can access, and may include both computer storage media and transmission media.

[0183] While this disclosure contains detailed descriptions of various detailed implementation examples, it should be understood that such details do not limit the invention or the scope of the claims proposed by this disclosure, but rather illustrate features of specific exemplary embodiments.

[0184] Features described individually in the exemplary embodiments of this disclosure may be implemented by a single exemplary embodiment. Conversely, various features described in the present disclosure with respect to a single exemplary embodiment may also be implemented by combinations or appropriate subcombinations of multiple exemplary embodiments. Furthermore, the present disclosure may disclose that the features operate in a particular combination, and while the combination may initially be described as claimed, in some cases, one or more features may be excluded from the claimed combination, or the claimed combination may be modified into a subcombination or a modification of a subcombination.

[0185] Likewise, even if operations are depicted in a particular order in the drawings, this should not be construed as requiring the execution of the operations in a specific order or sequence, or the performance of all operations, to achieve the desired result. Multitasking and parallel processing may be useful in certain cases. Furthermore, the various device components in the exemplary embodiments of the present invention should not be construed as necessarily being separate, and the program components and devices described above may be packaged into a single software product or multiple software products.

[0186] The exemplary embodiments disclosed herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will recognize that various modifications to the exemplary embodiments can be made without departing from the spirit and scope of the claims and their equivalents.

[0187] Accordingly, the present disclosure is intended to include all other replacements, modifications and variations that fall within the scope of the following claims.

[0188] The present disclosure can be industrially utilized in the fields of methods, devices, and recording media for video encoding / decoding.

Claims

1. A step of generating an inter prediction block of a current block from a reference picture; generating an intra prediction block of a current block based on one or more intra prediction mode candidates; and A step of generating a final prediction block of the current block by weighting the inter prediction block and the intra prediction block, The one or more intra prediction mode candidates include a first intra prediction mode candidate derived based on the inter prediction block, and A video decoding method, wherein the inter prediction block is generated based on motion information of an inter prediction mode candidate of the current block.

2. In paragraph 1, A method for decoding an image, wherein the one or more intra prediction mode candidates are derived by applying a filter that produces at least one of a gradient or a directionality to the inter prediction block.

3. In paragraph 1, A method for decoding an image, wherein the one or more intra prediction mode candidates are derived by applying a filter to the entire area or a part of the inter prediction block.

4. In paragraph 1, A video decoding method, wherein the first intra prediction mode candidate is an intra prediction mode having an angle that is the same as or most similar to an angle formed by a slope of the inter prediction block.

5. In paragraph 1, The one or more intra prediction mode candidates further include a second intra prediction mode candidate and a third intra prediction mode candidate, and A method for decoding an image, wherein the third intra prediction mode candidate is derived by performing an average or weighted sum operation on the first intra prediction mode candidate and the second intra prediction mode candidate.

6. In paragraph 1, The above one or more intra prediction mode candidates further include a fourth intra prediction mode candidate which is a default mode, and A method for decoding an image, wherein the above default mode is planar mode.

7. In paragraph 3, If the above filter is applied to the entire region of the inter prediction block, The above filter has a size smaller than the inter prediction block, and A method for decoding an image, wherein the first intra prediction mode candidate has the highest amplitude on a histogram obtained by repeatedly applying the filter to the inter prediction block.

8. In paragraph 3, A method for decoding an image, wherein the above filters are at least two different ones.

9. A step of generating an inter prediction block of the current block from a reference picture; generating an intra prediction block of a current block based on one or more intra prediction mode candidates; and A step of generating a final prediction block of the current block by weighting the inter prediction block and the intra prediction block, The one or more intra prediction mode candidates include a first intra prediction mode candidate derived based on the inter prediction block, and A video encoding method, wherein the inter prediction block is generated based on motion information of an inter prediction mode candidate of the current block.

10. A step of generating an inter prediction block of the current block from a reference picture; generating an intra prediction block of a current block based on one or more intra prediction mode candidates; and A step of generating a final prediction block of the current block by weighting the inter prediction block and the intra prediction block, The one or more intra prediction mode candidates include a first intra prediction mode candidate derived based on the inter prediction block, and A computer-readable recording medium storing a bitstream generated by a video encoding method, wherein the inter prediction block is generated based on motion information of an inter prediction mode candidate of the current block.

Images