Image encoding / decoding method and apparatus and recording medium storing bitstream

Abstract

The present application provides an image decoding method, comprising the steps of: determining prediction samples of a current block by using a first reference block derived from a first reference picture and a second reference block derived from a second reference picture; determining that a correction according to a bi-directional optical flow is to be applied to the current block based on a time distance difference between the first reference picture and a current picture and a time distance difference between the second reference picture and the current picture; determining a bi-directional optical flow offset in response to the application of the correction according to the bi-directional optical flow; and correcting the prediction samples of the current block based on the bi-directional optical flow offset.

Description

Technical Field

[0001] This invention relates to image encoding / decoding methods and apparatus, and recording media for storing bitstreams. More specifically, this invention relates to improved image encoding / decoding methods and apparatus utilizing bidirectional optical flow, and recording media for storing bitstreams. Background Technology

[0002] Recently, the demand for high-resolution, high-quality images, such as ultra-high definition (UHD) images, is increasing across various application areas. As image data becomes higher in resolution and quality, the data volume increases relatively compared to existing image data. Therefore, transmission and storage costs increase when using media such as existing wired and wireless broadband lines to transmit image data, or when using existing storage media to store image data. To address these issues arising from the increasing resolution and quality of image data, efficient image encoding / decoding technologies are needed for images with higher resolution and quality.

[0003] Bidirectional optical flow (BDOF) is a technique introduced to improve the accuracy of inter-frame prediction. In bidirectional prediction, motion compensation is performed using previous and subsequent reference images at different times. However, there are limitations to accurately estimating motion on a block-by-block basis. Specifically, when the movement of objects or backgrounds is complex or their speeds change significantly, it is difficult to accurately capture these complex changes using only simple forward or backward references. Therefore, by utilizing optical flow techniques to correct the motion vectors obtained via bidirectional prediction, more accurate motion vectors can be obtained. Various methods are being discussed to improve the prediction accuracy of bidirectional prediction by refining bidirectional optical flow techniques. Summary of the Invention

[0004] Technical issues The object of this invention is to provide a method and apparatus for encoding / decoding images with improved encoding / decoding efficiency.

[0005] Another object of the present invention is to provide a recording medium for storing a bitstream generated by the method or apparatus for decoding images provided by the present invention.

[0006] Technical solution An image decoding method according to an embodiment of the present invention may include: determining a predicted sample of a current block using a first reference block derived from a first reference image and a second reference block derived from a second reference image; determining, based on the temporal distance difference between the first reference image and the current image and the temporal distance difference between the second reference image and the current image, a refinement based on bidirectional optical flow to be applied to the current block; in response to the application of the refinement based on bidirectional optical flow, determining a first horizontal gradient and a first vertical gradient of the reference sample of the first reference block and a second horizontal gradient and a second vertical gradient of the reference sample of the second reference block; determining a first residual small displacement of the first reference block and a second residual small displacement of the second reference block based on the first horizontal gradient, the first vertical gradient, the second horizontal gradient, and the second vertical gradient; determining a bidirectional optical flow offset based on the first residual small displacement and the second residual small displacement; and refining the predicted sample of the current block based on the bidirectional optical flow offset.

[0007] According to the implementation scheme, in response to the difference between the temporal distance difference between the first reference image and the current image and the difference between the temporal distance difference between the second reference image and the current image, it can be determined that the refinement based on bidirectional optical flow will be applied to the current block.

[0008] According to the implementation scheme, in response to the fact that the temporal direction of the first reference image from the current image is the same as the temporal direction of the first reference image from the current image, it can be determined that the refinement based on the bidirectional optical flow will be applied to the current block.

[0009] According to the implementation plan, the second residual small displacement of the second reference block can be determined to be equal in magnitude and opposite in direction to the first residual small displacement of the first reference block.

[0010] According to the implementation plan, the second residual small displacement of the second reference block can be determined by multiplying the first residual small displacement of the first reference block by a weight.

[0011] According to the implementation plan, the weights may include horizontal weights for the horizontal component of the second residual small displacement and vertical weights for the vertical component of the second residual small displacement.

[0012] According to the implementation plan, the weights can be determined based on the temporal distance difference between the first reference image and the current image and the temporal distance difference between the second reference image and the current image.

[0013] According to the implementation plan, the weight can be determined based on the ratio of the temporal distance difference between the first reference image and the current image to the temporal distance difference between the second reference image and the current image. The sign of the weight can be determined based on the sign of the temporal distance difference between the first reference image and the current image and the sign of the temporal distance difference between the second reference image and the current image.

[0014] According to the implementation plan, the weights can be determined based on the encoding information associated with the current block, and the encoding information associated with the current block can include at least one of the following: the number of frames per second of the video, the resolution of the image, the group of pictures (GOP) included in the image, the slice type of the slice, and quantization parameters.

[0015] According to the implementation scheme, the weights can be determined based on weight information that explicitly indicates the weights, and the weight information can be included in at least one of the slice header, image header, image parameter set, sequence parameter set, and video parameter set associated with the current block.

[0016] According to the implementation plan, the weight can be derived based on the weights of one or more neighboring blocks of the current block.

[0017] According to the implementation plan, when the decoder-side motion vector refinement is applied to the current block, the weights can be determined based on the weight information used for the decoder-side motion vector refinement.

[0018] According to the implementation plan, in determining the bidirectional optical flow offset, when the ratio of the temporal distance difference between the second reference image and the current image to the temporal distance difference between the first reference image and the current image is greater than a predetermined threshold, the bidirectional optical flow offset can be determined based solely on the first residual small displacement without relying on the second residual small displacement.

[0019] According to the implementation plan, in determining the bidirectional optical flow offset, when the temporal distance difference between the second reference image and the current image is greater than a predetermined threshold, the bidirectional optical flow offset can be determined based solely on the first residual small displacement without being based on the second residual small displacement.

[0020] According to the implementation plan, in determining the bidirectional optical flow offset, when the first motion vector of the first motion vector and the first motion vector of the current block can be refined only by the motion vector refinement on the decoder side, the bidirectional optical flow offset is determined only based on the first residual small displacement and not based on the second residual small displacement.

[0021] According to the implementation plan, in determining the bidirectional optical flow offset, when the position corresponding to the first motion vector and the first residual small displacement of the current block is within the image boundary or slice boundary, and the position corresponding to the second motion vector and the second residual small displacement is outside the image boundary or slice boundary, the bidirectional optical flow offset can be determined based solely on the first residual small displacement instead of the second residual small displacement.

[0022] An image encoding method according to an embodiment of the present invention may include: determining a prediction sample of a current block using a first reference block derived from a first reference image and a second reference block derived from a second reference image; determining, based on the temporal distance difference between the first reference image and the current image and the temporal distance difference between the second reference image and the current image, a refinement based on bidirectional optical flow to be applied to the current block; in response to the application of the refinement based on bidirectional optical flow, determining a first horizontal gradient and a first vertical gradient of the reference sample of the first reference block and a second horizontal gradient and a second vertical gradient of the reference sample of the second reference block; determining a first residual small displacement of the first reference block and a second residual small displacement of the second reference block based on the first horizontal gradient, the first vertical gradient, the second horizontal gradient, and the second vertical gradient; determining a bidirectional optical flow offset based on the first residual small displacement and the second residual small displacement; and refining the prediction sample of the current block based on the bidirectional optical flow offset.

[0023] According to embodiments of the present invention, a non-volatile computer-readable recording medium can store a bitstream generated by an image encoding method.

[0024] The transmission method according to an embodiment of the present invention transmits a bit stream generated by an image encoding method.

[0025] The features briefly outlined above are merely exemplary aspects of the invention as described in the following detailed description and do not limit the scope of the invention.

[0026] Beneficial effects This invention proposes various implementation schemes for a method of correcting motion vectors based on bidirectional optical flow technology.

[0027] In addition, this invention proposes various implementation schemes for adaptive bidirectional optical flow technology.

[0028] According to the above implementation schemes, the prediction accuracy of the bidirectional prediction method based on bidirectional optical flow technology is improved, which can improve the overall encoding and decoding efficiency. Attached Figure Description

[0029] Figure 1 This is a block diagram illustrating the configuration of an encoding device according to an embodiment of the present invention.

[0030] Figure 2 This is a block diagram illustrating the configuration of a decoding device according to an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram illustrating a video encoding / decoding system to which the present invention can be applied.

[0032] Figure 4An implementation scheme of the motion vector refinement method based on BDOF technology is shown when the distance between two reference images and the current image is the same.

[0033] Figure 5 This illustrates an implementation scheme for a motion vector refinement method based on BDOF technology when the distances between two reference images and the current image are different.

[0034] Figure 6 This illustrates an implementation scheme for a motion vector refinement method based on BDOF technology when two reference images referenced by the current image are in the same temporal direction as the current image.

[0035] Figure 7 An implementation scheme for a motion vector refinement method based on BDOF technology is shown.

[0036] Figure 8 An exemplary content streaming system applicable to embodiments of the present invention is shown. Detailed Implementation

[0037] Best mode An image decoding method according to an embodiment of the present invention may include: determining a predicted sample of a current block using a first reference block derived from a first reference image and a second reference block derived from a second reference image; determining, based on the temporal distance difference between the first reference image and the current image and the temporal distance difference between the second reference image and the current image, a refinement based on bidirectional optical flow to be applied to the current block; in response to the application of the refinement based on bidirectional optical flow, determining a first horizontal gradient and a first vertical gradient of the reference sample of the first reference block and a second horizontal gradient and a second vertical gradient of the reference sample of the second reference block; determining a first residual small displacement of the first reference block and a second residual small displacement of the second reference block based on the first horizontal gradient, the first vertical gradient, the second horizontal gradient, and the second vertical gradient; determining a bidirectional optical flow offset based on the first residual small displacement and the second residual small displacement; and refining the predicted sample of the current block based on the bidirectional optical flow offset.

[0038] The mode of the present invention This invention can have various modifications and embodiments, and specific embodiments are shown in the accompanying drawings and described in detail in the specification. However, this is not intended to limit the invention to the specific embodiments, but rather to include all modifications, equivalents, or alternatives contained within the spirit and scope of the invention. The same reference numerals in the drawings indicate the same or similar functions in various aspects. For clarity, the shapes and dimensions of the elements in the drawings are provided by way of example. The detailed description of the exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice them. It should be understood that the various embodiments differ from one another but are not necessarily mutually exclusive. For example, the specific shapes, structures, and features described herein may be implemented in other embodiments without departing from the spirit and scope of the invention with reference to one embodiment. It should also be understood that the position or arrangement of the various components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiments. Accordingly, the detailed description set forth below is not intended to be restrictive, and the scope of the exemplary embodiments is defined only by the appended claims and the full scope of their equivalents (if appropriately described).

[0039] In this invention, the terms first, second, etc., may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the invention, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component. The term "and / or" includes a combination of multiple related descriptive terms or any item from multiple related descriptive terms.

[0040] The components shown in the embodiments of the invention are depicted independently to indicate different functional characteristics, and do not represent each component as a separate hardware or software configuration unit. That is, for ease of interpretation, each component is listed and included as a separate component, and at least two components may be combined to form a single component, or a component may be divided into multiple components to perform functions, as long as it does not depart from the spirit of the invention. Embodiments in which components are integrated and embodiments in which each component is divided are also included within the scope of the invention.

[0041] The terminology used in this invention is for describing specific embodiments only and is not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, some components of this invention are not essential for performing the necessary functions and may be optional components used only to improve performance. This invention can be implemented by including only the essential components necessary for achieving the spirit of the invention, excluding components used only to improve performance, and structures that include only the essential components and excluding optional components used only to improve performance are also included within the scope of this invention.

[0042] In the implementation, the term "at least one" can mean one of a number greater than or equal to 1, such as 1, 2, 3, and 4. In the implementation, the term "a plurality of" can mean one of a number greater than or equal to 2, such as 2, 3, and 4.

[0043] In the following, embodiments of the invention will be described in detail with reference to the accompanying drawings. When describing embodiments of this specification, detailed descriptions will be omitted if determining that a detailed description of a related known configuration or function would obscure the subject matter of this specification; the same reference numerals will be used for the same components in the drawings, and repeated descriptions of the same components will be omitted.

[0044] Description of terms In the following text, "image" can refer to a picture that constitutes a video, or it can refer to the video itself. For example, "encoding and / or decoding of an image" can mean "encoding and / or decoding of a video," or it can mean "encoding and / or decoding of one of the images that constitute a video."

[0045] In the following text, "moving image" and "video" can be used with the same meaning and can be used interchangeably. Additionally, the target image can be an encoded target image that is the target of encoding and / or a decoded target image that is the target of decoding. Furthermore, the target image can be an input image to an encoding device and can also be an input image to a decoding device. Here, the target image can have the same meaning as the current image.

[0046] In the following text, “image,” “picture,” “frame,” and “picture” can be used with the same meaning and can be used interchangeably.

[0047] In the following text, "target block" can refer to an encoding target block that is the target of encoding and / or a decoding target block that is the target of decoding. Additionally, a target block can refer to the current block that is the target of the current encoding and / or decoding. For example, "target block" and "current block" can be used with the same meaning and can be used interchangeably.

[0048] In the following text, "block" and "unit" can be used with the same meaning and can be used interchangeably. Additionally, "unit" can refer to a block comprising a luma component block and its corresponding chroma component block, in order to distinguish it from a block. For example, a coding tree unit (CTU) can consist of a luma component (Y) coding tree block (CTB) and two associated chroma component (Cb, Cr) coding tree blocks.

[0049] In the following text, "sample," "image element," and "pixel" can be used with the same meaning and interchangeably. In this article, a sample can represent the basic unit that makes up a block.

[0050] In the following text, "between frames" and "between images" can be used with the same meaning and can be used interchangeably.

[0051] In the following text, "within the frame" and "within the screen" can be used with the same meaning and can be used interchangeably.

[0052] Figure 1 This is a block diagram illustrating the configuration of an encoding device according to an embodiment of the present invention.

[0053] The encoding device 100 may be an encoder, a video encoding device, or an image encoding device. The video may include one or more images. The encoding device 100 may encode one or more images sequentially.

[0054] refer to Figure 1 The encoding device 100 may include: an image partitioning unit 110, an intra-frame prediction unit 120, a motion prediction unit 121, a motion compensation unit 122, a switcher 115, a subtractor 113, a transform unit 130, a quantization unit 140, an entropy coding unit 150, an inverse quantization unit 160, an inverse transform unit 170, an adder 117, a filtering unit 180, and a reference image buffer 190.

[0055] Additionally, the encoding device 100 can generate a bitstream including information encoded by encoding the input image, and output the generated bitstream. The generated bitstream can be stored in a computer-readable recording medium or streamed via a wired / wireless transmission medium.

[0056] Image partitioning unit 110 can partition the input image into various forms to improve the efficiency of video encoding / decoding. That is, the input video consists of multiple images, and for compression efficiency, parallel processing, etc., an image can be partitioned and processed hierarchically. For example, an image can be partitioned into one or more tiles or slices, and then further partitioned into multiple codec tree units (CTUs). Alternatively, an image can first be partitioned into multiple sub-images defined as groups of rectangular slices, and each sub-image can be partitioned into tiles / slices. Here, sub-images can be used to support the function of partially independent encoding / decoding and transmission of images. Since multiple sub-images can be reconstructed individually, this has the advantage of ease of editing in applications where multi-channel input is configured as a single image. Additionally, tiles can be horizontally partitioned to generate bricks. Here, bricks can be used as the basic unit for parallel processing within an image. Furthermore, a CTU can be recursively partitioned into a quad tree (QT), and the terminal node of the partition can be defined as a codec unit (CU). A CTU can be partitioned into a Prediction Unit (PU) as a prediction unit and a Transform Unit (TU) as a transformation unit to perform prediction and partitioning. Alternatively, a CTU can be used as both a prediction unit and / or a transformation unit itself. For flexible partitioning, each CTU can be recursively partitioned into multi-type trees (MTTs) and quadtrees (QTs). Partitioning a CTU into a multi-type tree can begin from the terminal node of a QT, and an MTT can consist of binary trees (BTs) and triple trees (TTs). For example, an MTT structure can be categorized into vertical binary partitioning patterns (SPLIT_BT_VER), horizontal binary partitioning patterns (SPLIT_BT_HOR), vertical ternary partitioning patterns (SPLIT_TT_VER), and horizontal ternary partitioning patterns (SPLIT_TT_HOR). Additionally, during partitioning, the minimum block size (MinQTSize) of the quadtree for the luma block can be set to 16×16, the maximum block size (MaxBtSize) of the binary tree can be set to 128×128, and the maximum block size (MaxTtSize) of the ternary tree can be set to 64×64. Furthermore, the minimum block sizes (MinBtSize) of the binary tree and the minimum block size (MinTtSize) of the ternary tree can be specified as 4×4, and the maximum depth (MaxMttDepth) of the multi-type tree can be specified as 4. Moreover, to improve the coding efficiency of I-slices, a dual-tree CTU partitioning structure using different luma and chroma components can be applied.On the other hand, in P and B slices, the luma and chroma codec tree blocks (CTBs) within the CTU can be partitioned into single trees sharing a codec tree structure.

[0057] The encoding device 100 can encode the input image in intra-frame mode and / or inter-frame mode. Alternatively, the encoding device 100 can encode the input image in a third mode other than intra-frame mode and inter-frame mode (e.g., IBC mode, palette mode, etc.). However, if the third mode has similar functional characteristics to the intra-frame mode or inter-frame mode, it can be classified as an intra-frame mode or inter-frame mode for ease of explanation. In this invention, the third mode is classified and described separately only when a specific explanation of the third mode is required.

[0058] When using intra-frame mode as the prediction mode, switcher 115 can switch to intra-frame mode, and when using inter-frame mode as the prediction mode, switcher 115 can switch to inter-frame mode. Here, intra-frame mode can refer to intra-frame prediction mode, and inter-frame mode can refer to inter-frame prediction mode. Encoding device 100 can generate prediction blocks for input blocks of the input image. In addition, encoding device 100 can encode residual blocks using the residuals between the input block and the prediction block after generating the prediction blocks. The input image can be referred to as the current image as the current coding target. The input block can be referred to as the current block as the current coding target or the coding target block.

[0059] When the prediction mode is intra-frame mode, the intra-frame prediction unit 120 can use samples of already encoded / decoded blocks surrounding the current block as reference samples. The intra-frame prediction unit 120 can perform spatial prediction of the current block using the reference samples, or generate prediction samples for the input block through spatial prediction. In this paper, intra-frame prediction can refer to intra-frame prediction.

[0060] As an intra-frame prediction method, non-directional prediction modes such as DC mode and planar mode, as well as directional prediction modes (e.g., 65 directions) can be applied. Here, the intra-frame prediction method can be expressed as an intra-frame prediction mode or an intra-frame prediction mode.

[0061] When the prediction mode is inter-frame mode, the motion prediction unit 121 can retrieve the region that best matches the input block from the reference image during motion prediction processing, and derive the motion vector by utilizing the retrieved region. In this case, the search region can be used as the region. The reference image can be stored in the reference image buffer 190. Here, it can be stored in the reference image buffer 190 when encoding / decoding the reference image is performed.

[0062] The motion compensation unit 122 can generate a predicted block for the current block by performing motion compensation using motion vectors. In this paper, inter-frame prediction can refer to inter-picture prediction or motion compensation.

[0063] When the value of the motion vector is not an integer, the motion prediction unit 121 and the motion compensation unit 122 can generate prediction blocks by applying an interpolation filter to a portion of the reference image. To perform inter-frame prediction or motion compensation, the motion prediction and motion compensation modes of the prediction units included in the encoding / decoding unit can be determined based on the encoding / decoding unit. These modes include skip mode, merge mode, advanced motion vector prediction (AMVP) mode, and intra-block copy (IBC) mode, and inter-frame prediction or motion compensation can be performed according to each mode.

[0064] In addition, based on the above inter-frame prediction methods, the following modes can be applied: Affine mode based on sub-PU prediction, Subblock-based Temporal Motion Vector Prediction (SbTMVP) mode, Merge with MVD (MMVD) mode based on PU prediction, and Geometric Partitioning Mode (GPM) mode. In addition, to improve the performance of each mode, the following can be applied: History-based MVP (HMVP), Pairwise Average MVP (PAMVP), Combined Intra / Inter Prediction (CIIP), Adaptive Motion Vector Resolution (AMVR), Bi-Directional Optical-Flow (BDOF), Bi-predictive with CU Weights (BCW), Local Illumination Compensation (LIC), Template Matching (TM), and Overlapped Block Motion Compensation (OBMC).

[0065] Subtractor 113 can generate residual blocks by utilizing the difference between the input block and the prediction block. The residual block can be referred to as a residual signal. The residual signal can represent the difference between the original signal and the prediction signal. Alternatively, the residual signal can be a signal generated by transforming or quantizing, or transforming and quantizing the difference between the original signal and the prediction signal. The residual block can be a residual signal on a block-by-block basis.

[0066] Transform unit 130 can generate transform coefficients by performing a transform on the residual block and output the generated transform coefficients. In this document, the transform coefficients can be coefficient values ​​generated by performing a transform on the residual block. When a transform skip mode is applied, transform unit 130 can skip the transform of the residual block.

[0067] Quantization levels can be generated by applying quantization to the transform coefficients or residual signal. In the following text, quantization levels may also be referred to as transform coefficients in the implementation scheme.

[0068] For example, the 4×4 lumen residual block generated by intra-frame prediction can be transformed using basis vectors based on Discrete Sine Transform (DST), and the remaining residual block can be transformed using basis vectors based on Discrete Cosine Transform (DCT). Furthermore, the transformed block can be partitioned into a quadtree shape using Residual Quad Tree (RQT) technology, and after performing transformation and quantization on each transformed block partitioned by RQT, the coded block flag (CBF) can be transmitted when all coefficients become 0 to improve coding efficiency.

[0069] As an alternative, Multiple Transform Selection (MTS) can be applied, selectively using multiple transform bases to perform the transform. That is, instead of partitioning the CU into TUs via RQT, a sub-block transform (SBT) technique can perform a function similar to TU partitioning. Specifically, SBT is applied only to inter-frame prediction blocks, and unlike RQT, it can partition the current block into 1 / 2 or 1 / 4 sizes vertically or horizontally, and then perform the transform on only one block. For example, if the current block is vertically partitioned, the transform can be performed on the leftmost or rightmost block, and if the current block is horizontally partitioned, the transform can be performed on the topmost or bottommost block.

[0070] In addition, the Low Frequency Non-Separable Transform (LFNST) can be applied. This is a secondary transform technique that additionally transforms the residual signal into the frequency domain using either DCT or DST. LFNST performs an additional transform on the upper left 4×4 or 8×8 low-frequency region, allowing the residual coefficients to be concentrated on the upper left.

[0071] The quantization unit 140 can generate quantization levels by quantizing the transform coefficients or residual signals according to the quantization parameters (QP), and output the generated quantization levels. In this paper, the quantization unit 140 can quantize the transform coefficients using a quantization matrix.

[0072] For example, quantizers with QP values ​​from 0 to 51 can be used. Alternatively, if the image size is large and high coding efficiency is required, QP values ​​from 0 to 63 can be used. Furthermore, dependent quantization (DQ) methods that utilize two quantizers instead of one can be applied. DQ performs quantization using two quantizers (e.g., Q0 and Q1), but even without signaling information about the use of a particular quantizer, a state transition model can be used to select the quantizer for the next transform coefficient based on the current state.

[0073] Entropy coding unit 150 can generate and output a bitstream by performing entropy coding on values ​​calculated by quantization unit 140 or on encoding / decoding parameter values ​​calculated during encoding, according to a probability distribution. Entropy coding unit 150 can perform entropy coding on information about samples of the image and information used for decoding the image. For example, information used for decoding the image may include syntax elements.

[0074] When entropy coding is applied, symbols are represented such that fewer bits are allocated to symbols with high occurrence probabilities and more bits are allocated to symbols with low occurrence probabilities, thus reducing the size of the bitstream used for the symbols to be encoded. The entropy coding unit 150 can perform entropy coding using coding methods such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). For example, the entropy coding unit 150 can perform entropy coding by utilizing a variable-length coding / code (VLC) table. Furthermore, the entropy coding unit 150 can derive a binarization method for the target symbol and a probability model for the target symbol / binary representation, and perform arithmetic coding by utilizing the derived binarization method and context model.

[0075] Relatedly, when applying CABAC, to reduce the size of the probability table stored in the decoding device, the table probability update method can be modified into a table update method using a simple equation and then applied. Alternatively, two different probability models can be used to obtain more accurate symbol probability values.

[0076] In order to encode the transform coefficient level (quantization level), the entropy coding unit 150 can change the coefficients in two-dimensional block form into one-dimensional vector form by the transform coefficient scanning method.

[0077] The encoding and decoding parameters may include information (flags, indexes, etc.) encoded in the encoding device 100 and signaled to the decoding device 200, such as syntax elements, as well as information derived during the encoding or decoding process, and may represent the information required when encoding or decoding an image.

[0078] In this paper, emitting a flag or index with a signal can indicate that the corresponding flag or index is entropy encoded in the encoder and included in the bitstream, and can also indicate that the corresponding flag or index is entropy decoded from the bitstream in the decoder.

[0079] The encoded current image can be used as a reference image for another image to be processed later. Therefore, the encoding device 100 can reconstruct or decode the encoded current image again and store the reconstructed or decoded image as a reference image in the reference image buffer 190.

[0080] The quantization level can be dequantized in dequantization unit 160, or inverse transformed in inverse transform unit 170. The coefficients of dequantization and / or inverse transform can be added to the prediction block via adder 117. In this document, the coefficients of dequantization and / or inverse transform can represent coefficients from which at least one dequantization and inverse transform have been performed, and can represent the reconstructed residual block. Dequantization unit 160 and inverse transform unit 170 can be performed as the inverse of quantization unit 140 and transform unit 130.

[0081] The reconstructed block can be processed by filtering unit 180. Filtering unit 180 can utilize all or some filtering techniques to apply deblocking filters, sample adaptive offset (SAO) filters, adaptive loop filters (ALF), bilateral filters (BIF), luma mapping with chroma scaling (LMCS) filters, etc., to the reconstructed sample, reconstructed block, or reconstructed image. Filtering unit 180 can be referred to as an in-loop filter. In this case, in-loop filter can also be used as a name excluding LMCS.

[0082] Deblocking filters remove block distortion generated at the boundaries between blocks. To determine whether to apply a deblocking filter, the application of the filter to the current block can be determined based on samples included in several rows or columns contained within the block. When applying a deblocking filter to a block, different filters can be applied depending on the desired deblocking filtering intensity.

[0083] To compensate for coding errors using sample-adaptive offsets, appropriate offset values ​​can be added to the sample values. Sample-adaptive offsets can correct the offset between the deblocked image and the original image on a sample-by-sample basis. Methods include dividing the image into a predetermined number of regions, determining the regions to which the offset will be applied, and then applying the offset to those regions; or considering edge information about each sample when applying the offset.

[0084] Bilateral filters (BIF) can also correct the offset from the original image on a sample-by-sample basis for images that have already undergone deblocking.

[0085] Adaptive loop filters can perform filtering based on a comparison between the reconstructed image and the original image. Samples included in the image can be partitioned into predetermined groups, the filter to be applied to each group can be determined, and differential filtering can be performed on each group. Information regarding whether to apply the ALF can be emitted by the codec unit (CU) using a signal, and the form and coefficients of the adaptive loop filter to be applied to each block can vary.

[0086] In luma mapping with chroma scaling (LMCS), luma mapping (LM) represents remapping luma values ​​using a piecewise linear model, and chroma scaling (CS) represents scaling the residual values ​​of the chroma components based on the average luma values ​​of the predicted signal. Specifically, LMCS can be used as an HDR correction technique that reflects the characteristics of High Dynamic Range (HDR) images.

[0087] The reconstructed block or reconstructed image that has passed through filtering unit 180 can be stored in reference image buffer 190. The reconstructed block that has passed through filtering unit 180 can be a portion of the reference image. That is, the reference image is a reconstructed image composed of the reconstructed blocks that have passed through filtering unit 180. The stored reference image can be used later for inter-frame prediction or motion compensation.

[0088] Figure 2 This is a block diagram illustrating the configuration of a decoding device according to an embodiment of the present invention.

[0089] The decoding device 200 can be a decoder, a video decoding device, or an image decoding device.

[0090] refer to Figure 2 The decoding device 200 may include: an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra-frame prediction unit 240, a motion compensation unit 250, an adder 201, a switcher 203, a filtering unit 260, and a reference image buffer 270.

[0091] The decoding device 200 can receive a bitstream output from the encoding device 100. The decoding device 200 can receive a bitstream stored in a computer-readable recording medium, or it can receive a bitstream streamed via a wired / wireless transmission medium. The decoding device 200 can decode the bitstream in intra-frame mode or inter-frame mode. Furthermore, the decoding device 200 can generate a reconstructed image or a decoded image through decoding, and output the reconstructed image or the decoded image.

[0092] When the prediction mode used for decoding is intra-frame mode, switcher 203 can switch to intra-frame mode. Alternatively, when the prediction mode used for decoding is inter-frame mode, switcher 203 can switch to inter-frame mode.

[0093] The decoding device 200 can obtain a reconstructed residual block and generate a prediction block by decoding the input bitstream. When the reconstructed residual block and prediction block are obtained, the decoding device 200 can add them together to generate a reconstructed block that becomes the decoding target. The decoding target block can be referred to as the current block.

[0094] The entropy decoding unit 210 can generate symbols by entropy decoding the bitstream according to the probability distribution. The generated symbols can include symbols in quantization level form. In this paper, the entropy decoding method can be the inverse process of the entropy encoding method described above.

[0095] The entropy decoding unit 210 can change the coefficients of a one-dimensional vector shape into coefficients of a two-dimensional block shape through the transformation coefficient scanning method to decode the transformation coefficient level (quantization level).

[0096] The quantization level can be dequantized in the dequantization unit 220 or inverse transformed in the inverse transform unit 230. The quantization level can be the result of dequantization and / or inverse transform, and can be generated as a reconstructed residual block. In this document, the dequantization unit 220 can apply a quantization matrix to the quantization level. The dequantization unit 220 and inverse transform unit 230 applied to the decoding device can employ the same techniques as those applied to the dequantization unit 160 and inverse transform unit 170 of the aforementioned encoding device.

[0097] When using intra-frame mode, intra-frame prediction unit 240 can generate a prediction block by performing spatial prediction on the current block, which uses sample values ​​from blocks that have already been decoded around the target block. The intra-frame prediction unit 240 applied to the decoding apparatus can employ the same techniques as the intra-frame prediction unit 120 applied to the aforementioned encoding apparatus.

[0098] When using inter-frame mode, motion compensation unit 250 can generate a prediction block by performing motion compensation on the current block using motion vectors and a reference image stored in reference image buffer 270. When the value of the motion vector is not an integer value, motion compensation unit 250 can generate a prediction block by applying an interpolation filter to a portion of the reference image. To perform motion compensation, the motion compensation mode of the prediction unit included in the corresponding encoding / decoding unit can be determined based on the encoding / decoding unit, whether it is a skip mode, merge mode, AMVP mode, or current image reference mode, and motion compensation can be performed according to each mode. The motion compensation unit 250 applied to the decoding device can apply the same techniques as the motion compensation unit 122 applied to the encoding device described above.

[0099] Adder 201 generates a reconstructed block by adding the reconstructed residual block and the prediction block. Filtering unit 260 can apply at least one of inverse LMCS, deblocking filter, sample adaptive offset, and adaptive loop filter to the reconstructed block or reconstructed image. Filtering unit 260 applied to the decoding apparatus can apply the same filtering techniques as filtering unit 180 applied to the aforementioned encoding apparatus.

[0100] Filtering unit 260 can output a reconstructed image. The reconstructed block or reconstructed image can be stored in reference image buffer 270 and used for inter-frame prediction. The reconstructed block that has passed through filtering unit 260 can be a portion of the reference image. That is, the reference image can be a reconstructed image composed of reconstructed blocks that have passed through filtering unit 260. The stored reference image can be used later for inter-frame prediction or motion compensation.

[0101] Figure 3 This is a schematic diagram illustrating a video encoding / decoding system to which the present invention can be applied.

[0102] The video encoding / decoding system according to the implementation scheme may include an encoding device 10 and a decoding device 20. The encoding device 10 may send encoded video and / or image information or data to the decoding device 20 in the form of files or streaming via a digital storage medium or network.

[0103] The encoding apparatus 10 according to the embodiment may include a video source generation unit 11, an encoding unit 12, and a transmission unit 13. The decoding apparatus 20 according to the embodiment may include a receiving unit 21, a decoding unit 22, and a rendering unit 23. The encoding unit 12 may be referred to as a video / image encoding unit, and the decoding unit 22 may be referred to as a video / image decoding unit. The transmission unit 13 may be included in the encoding unit 12. The receiving unit 21 may be included in the decoding unit 22. The rendering unit 23 may include a display unit, and the display unit may be configured as a separate device or an external component.

[0104] The video source generation unit 11 can obtain video / images through processes of capturing, compositing, or generating video / images. The video source generation unit 11 may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive including previously captured video / images, etc. The video / image generation device may include, for example, a computer, tablet computer, and smartphone, and can generate video / images (electronically). For example, virtual video / images can be generated by a computer, etc., in which case the process of generating related data can replace the video / image capture process.

[0105] Encoding unit 12 can encode the input video / image. For compression and encoding efficiency, encoding unit 12 can perform a series of processes such as prediction, transformation, and quantization. Encoding unit 12 can output the encoded data (encoded video / image information) as a bitstream. Detailed configuration of encoding unit 12 can also be found in the above description. Figure 1 The encoding device 100 is configured in the same way.

[0106] The transmitting unit 13 can send encoded video / image information or data, output in bitstream form, to the receiving unit 21 of the decoding device 20 via a digital storage medium or network in the form of a file or stream. The digital storage medium can include various storage media, such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit 13 can include elements for generating media files according to a predetermined file format, and may include elements for transmission via a broadcast / communication network. The receiving unit 21 can extract / receive the bitstream from the storage medium or network and send it to the decoding unit 22.

[0107] Decoding unit 22 can decode video / images by performing a series of processes such as inverse quantization, inverse transform, and prediction, corresponding to the operations of encoding unit 12. Detailed configuration of decoding unit 22 can also be found in... Figure 2 The decoding device 200 described above is configured in the same manner.

[0108] Rendering unit 23 can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0109] This invention describes an improved method for bidirectional optical flow (BDOF) technology applied to bidirectional inter-frame prediction. According to BDOF, the final prediction block is determined by a weighted combination of two already encoded prediction blocks that exist in reference images in different directions relative to the current image. Two motion vectors (MVs) are used to obtain the two prediction blocks, each derived from a past reference image (list 0, L0) and a future reference image (list 1, L1) in playback order. Although BDOF offers high encoding / decoding performance, small displacements may be preserved between samples within the two prediction blocks due to limitations in block-based motion compensation (MC).

[0110] This small residual displacement can be compensated for using bidirectional optical flow (BDOF) technology. According to BDOF, the pixel variations within the bidirectional prediction block are estimated based on the optical flow differential equation. Furthermore, the refinement value of the prediction samples for the prediction block is determined based on the pixel variations within the bidirectional prediction block.

[0111] BDOF is designed based on the assumption that any object moves at a constant speed. According to one implementation, BDOF can be configured to apply only when the distances between two reference images and the current image (that is, the difference in the picture order count (POC) between the two reference images and the current image) are the same. Here, POC is the sequence number of the images according to the playback order. Furthermore, according to one implementation, BDOF can be applied in units of 4×4, 8×8, and 16×16 blocks. Additionally, BDOF can be applied in sample units.

[0112] Furthermore, BDOF can be executed two or more times from the top block to the bottom block. For example, BDOF can primarily execute on 16×16 sub-blocks, then on 8×8 sub-blocks, and finally on 4×4 sub-blocks. Alternatively, it can primarily execute on 8×8 sub-blocks, then on 4×4 sub-blocks. The BDOF implementation described below can be applied to all sequences of BDOF distinguished by the size of the sub-blocks.

[0113] Figure 4 An implementation scheme of the motion vector refinement method based on BDOF technology is shown when the distance between two reference images and the current image is the same. Figure 4 The current image 400, L0 reference image 420, and L1 reference image 440 are shown. Bidirectional inter-frame prediction is applied to the current block 410 of the current image 400, and the current block 410 is predicted by L0 motion vector 430 and L1 motion vector 450. Here, the POC difference between the current image 400 and the L0 reference image 420 is represented by DiffPOC. L0 The difference in POC between the current image 400 and the L1 reference image 440 (DiffPOC) L1 The absolute values ​​are the same. Here, according to the BDOF technique, the residual small displacement (v) can be calculated by minimizing the difference between the L0 prediction sample and the L1 prediction sample for each sub-block of arbitrary size. x v y ) and (-v x -v y Accordingly, the prediction block of the current block 410 can be refined based on the BDOF offset calculated from the residual small displacement according to the BDOF technique.

[0114] For example, first, the horizontal and vertical gradients of the L0 and L1 predicted samples are calculated. These gradients can be calculated based on the differences between any number of neighboring samples located around the predicted sample (i, j). Next, the autocorrelation and cross-correlation values ​​of the horizontal and vertical gradients are calculated. Furthermore, the residual small displacement (v) is derived using the correlation. x v yAssuming any object has a constant velocity, the residual small displacement of the L1 reference image is determined as (-v). x v y While moving a window of arbitrary size, residual small displacements can be determined on a sample-by-sample or block-by-block basis. Based on the gradient and residual small displacements, a BDOF offset for correcting predicted samples is calculated.

[0115] As shown in Equation 1, this can be achieved by shifting the BDOF by σ. BDOF Add the predicted values ​​(predicted samples or predicted blocks) to the existing bidirectional predictions to obtain refined predicted sample values ​​(final predicted samples or final predicted blocks). In Equation 1, pred L0 and pred L1 represents the predicted samples or blocks from L0 and L1, respectively, and o and s represent the offset and shift values ​​used to normalize the final predicted value in the existing bidirectional prediction, respectively.

[0116] [Equation 1] pred = (pred L0 + pred L1 + σ BDOF + o ) >> s According to one implementation, based on the assumption that the object moves at a constant speed and in a constant direction, BDOF can be set to only the distance between the current image from L0 and the reference image (=DiffPOC). L0 ) and the distance between the current image from L1 and the reference image (=DiffPOC) L1 They execute simultaneously. However, the velocity of a unit object in the real world can change every hour.

[0117] However, due to various factors, the two reference blocks used for bidirectional prediction of the current block may not always be at the same distance from each other. Therefore, BDOF can be configured to work even in DiffPOC. L0 and DiffPOC L1 It is not executed at the same time.

[0118] Specifically, decoder-side motion vector refinement (DMVR), which performs motion prediction by refining the motion information of neighboring blocks on the decoder side, can be configured not only in DiffPOC. L0 and DiffPOC L1 Execute simultaneously, and also in DiffPOC L0 and DiffPOC L1They cannot be executed simultaneously. Therefore, DMVR and BDOF can be executed not only in DiffPOC. L0 and DiffPOC L1 It can be applied simultaneously, and can also be used in DiffPOC. L0 and DiffPOC L1 They cannot be used simultaneously.

[0119] Figure 5 This illustrates an implementation scheme for motion vector refinement based on BDOF technology when the distances between two reference images and the current image are different. Figure 4 The difference lies in Figure 5 In the diagram, the difference in POC between the current image 500 and the L0 reference image 520 is called DiffPOC. L0 The difference in POC between the current image 500 and the L1 reference image 540 (DiffPOC) L1 The absolute values ​​are different. Therefore, the small displacement (v) applied to the L0 reference image 520 x v y ) and the small displacement (-ω) applied to the L1 reference image 540 x v x , -ω y v y ) can be calculated as having different sizes.

[0120] For example, calculate the autocorrelation and cross-correlation values ​​of the horizontal and vertical gradients of LX (X = 0 or 1). Furthermore, use the autocorrelation and cross-correlation values ​​to derive the residual small displacement (v). x v y Next, the residual small displacement of LY (Y=1-x) can be determined as (-ω). x v x , -ω y v y Here, L0 and L1 represent the first and second reference image lists. Furthermore, the BDOF offset is determined using the residual small displacements and the horizontal and vertical gradients of L0 and L1.

[0121] Figure 6 This illustrates an implementation scheme for a motion vector refinement method based on BDOF technology when two reference images referenced by the current image are in the same temporal direction as the current image. Figure 6 Reference images A 620 and B 640 are earlier than the current image 600. Furthermore, the difference in POC between the current image 600 and reference image A 620 is represented by the DiffPOC. A The difference in POC between the current image (600) and the reference image (B) (640) is called the POC difference. BThe absolute values ​​are different. At this point, for example, in... Figure 5 In the middle, it is applied to the small displacement (v) of reference image A 620. x v y ) and the small displacement (-ω) applied to reference image B 640 x v x , -ω y v y ) can be calculated as having different sizes.

[0122] For example, calculate the autocorrelation and cross-correlation values ​​of the horizontal and vertical gradients of reference image A. Furthermore, use the autocorrelation and cross-correlation values ​​to derive the residual small displacement (v). x v y Next, the residual small displacement of reference image B (which is in the same time domain direction as reference image A) can be determined as (-ω). x v x , -ω y v y Additionally, the BDOF offset is determined using the residual small displacement and the horizontal and vertical gradients of L0 and L1. Reference image A and reference image B can be reference images included in reference image lists L0 and L1, respectively. Alternatively, both reference image A and reference image B can be included in either reference image list L0 or reference image list L1.

[0123] The weight ω used in the above implementation scheme to determine the small residual displacement x and ω y It is any real number and can be determined in one of the following methods.

[0124] According to one implementation plan, the weight ω x and ω y Both can be determined to be 1. That is, the small residual displacement can be determined as (-v) x -v y Alternatively, according to one implementation, specific values ​​predetermined in the encoder / decoder can be used as weights.

[0125] like Figure 5 As shown, when the orientation of the reference image is different, the distance between the current image and the LX (X=0 or 1) reference image (=DiffPOC) can be used. LX ), the distance between the current image and the LY (Y=1-X) reference image (=DiffPOC) LY ) or their ratio to calculate the weight ω x and ω y If the weight ω x or / and ω yIf the value is greater than 1, then the motion vector of the LY reference image is refined more than that of the LX reference image. Conversely, if the weight ω is greater than 1, then the motion vector of the LY reference image is refined more. x or / and ω y A value less than 1 indicates that less refinement is applied to the motion vectors of the LY reference image compared to the motion vectors of the LX reference image.

[0126] like Figure 6 As shown, when the reference images are in the same orientation, the distance between the current image and reference image A from the reference image (=DiffPOC) can be used. A ), the distance between the current image and reference image B from the reference image (=DiffPOC) B ) or their ratio to calculate the weight ω x and ω y If the weight ω x or / and ω y A value greater than 1 indicates that the motion vector of reference image B is refined more than that of reference image A. Conversely, if the weight ω... x or / and ω y A value less than 1 indicates that less refinement was applied to the motion vector of reference image B compared to the motion vector of reference image A.

[0127] weight ω x and ω y It can be calculated or derived based on some or all of the information from the current / adjacent blocks. For example, ω x and ω y It can be derived from the inter-frame predicted syntax elements of the current block. Or, ω x and ω y It can be extracted from one of the adjacent blocks of the current block, and used to determine the ω of the current block. x value and ω y value.

[0128] The weights ω can be determined / calculated based on image / picture / patch related information (e.g., resolution, frames per second (fps), GOP, codec configuration, slice type, quantization parameters, etc.). x and ω y .

[0129] weight ω x and ω y The weight ω can be indicated by the explicit indication. x and ω y The syntax elements are determined by the syntax elements. Syntax elements can be signaled at one or more levels (such as sequences / pictures / slices / blocks).

[0130] weight ωx and ω y This can be determined to be equal to the weights used in the encoding / decoding techniques performed before BDOF (e.g., DMVR). Alternatively, the weight ω x and ω y This can be determined by adjusting the weights used in the encoding / decoding techniques performed before BDOF.

[0131] The BDOF offset for correcting the predicted samples can be calculated based on the residual small displacement and gradient calculated using the method described above. Furthermore, the calculated BDOF offset can be added to the predicted samples or blocks obtained through bidirectional prediction to refine the predicted samples or blocks.

[0132] As mentioned earlier, regardless of the distance between the reference image and the current image, applying BDOF can improve encoding and decoding efficiency in bidirectional prediction by correcting for the additional prediction blocks obtained through bidirectional prediction using BDOF. Furthermore, since DMVR is applied regardless of the distance between the reference image and the current image, BDOF and DMVR can be combined and applied to bidirectional prediction of blocks.

[0133] In this invention, an example of a method for adaptively calculating small residual displacements when applying BDOF is described. Hereinafter, an adaptive BDOF technique considering the distance between the current image and a reference image will be described. Additionally, an adaptive BDOF technique based on adaptive DMVR will be described. Furthermore, an adaptive BDOF considering image / tile boundaries will be described.

[0134] According to an implementation plan, such as Figure 5 As shown, when the distance between the current image 500 and the reference images 520 and 540 is different, adaptive BDOF can be performed by considering the distance between the current image 500 and the reference images 520 and 540. The residual small displacement of L1 can be based on the residual small displacement of L0 (v x v y ) and weight ω x and ω y To determine. However, if the distance between the current image 500 and the L1 reference image 540 is (=DiffPOC) L1 The distance between the current image 500 and the L0 reference image 520 (=DiffPOC) L0 The ratio of DiffPOC is greater than the predetermined value, or if DiffPOC L1 If the value exceeds any threshold, the prediction accuracy may be reduced when correcting the motion information corresponding to L1 based on the residual small displacement.

[0135] Therefore, if DiffPOC L1 c×DiffPOCL0 Or if DiffPOC L1 >Th, then the BDOF offset can be calculated based on the gradient and residual small displacement of the L0 reference image 520, without considering the gradient and residual small displacement of the L1 reference image 540. Conversely, if DiffPOC L0 c×DiffPOC L1 Or if DiffPOC L0 Th can then be used to calculate the BDOF offset based on the gradient and residual small displacement of the L1 reference image 540, without considering the gradient and residual small displacement of the L0 reference image 520. Here, c and Th are arbitrary positive integers.

[0136] Therefore, by calculating the BDOF offset based on the gradient and residual small displacement of a highly reliable reference image, the coding efficiency of BDOF can be improved.

[0137] According to one implementation, a method for applying adaptive BDOF is described based on whether the motion vectors of the L0 or L1 reference image are refined in the adaptive DMVR technique. Adaptive DMVR is a technique that corrects motion information only in one direction LX (X=0 or 1) of the bidirectional prediction of the merged candidates that satisfy the DMVR conditions. If adaptive DMVR is performed, it is also reasonable to perform refinement only in one direction of the bidirectional prediction in the subsequent BDOF.

[0138] Therefore, if only L0 motion information is refined in adaptive DMVR, the BDOF offset can be calculated based on the gradient and residual small displacement of the L0 reference image, without considering the gradient and residual small displacement of the L1 reference image. Conversely, if only L1 motion information is refined in adaptive DMVR, the BDOF offset can be calculated based on the gradient and residual small displacement of the L1 reference image, without considering the gradient and residual small displacement of the L0 reference image.

[0139] This implementation scheme, which combines adaptive DMVR and adaptive BDOF technologies, can be applied not only to situations where the distance between the current image and the reference image is different, but also to situations where the distance between the current image and the reference image is the same.

[0140] According to one implementation, an adaptive BDOF considering image boundaries and / or slice boundaries is described. When a residual small displacement calculated for LX (X=0 or 1) is applied in BDOF, the position indicated by the residual small displacement can be outside the image / slice boundary. In this case, for the accuracy of motion vector refinement, the residual small displacement can be disregarded for the LX reference image. Therefore, the BDOF offset can be calculated by considering only the residual small displacement of LY (Y=1-X) without considering the residual small displacement of the LX reference image. For example, if the position based on the residual small displacement of L0 is outside the image / slice boundary, then only the residual small displacement corresponding to the L1 reference image can be used in the BDOF offset calculation. Conversely, if the position based on the residual small displacement of L1 is outside the image / slice boundary, then only the residual small displacement corresponding to the L01 reference image can be used in the BDOF offset calculation.

[0141] This implementation scheme, which combines adaptive DMVR and adaptive BDOF technologies, can be applied not only to situations where the distance between the current image and the reference image is different, but also to situations where the distance between the current image and the reference image is the same.

[0142] Figure 7 An implementation scheme for a motion vector refinement method based on BDOF technology is shown.

[0143] In step 702, a predicted sample for the current block is determined using a first reference block derived from a first reference image and a second reference block derived from a second reference image. If bidirectional prediction is applied to the current block, a predicted sample for the current block can be determined from two reference images. The first and second reference images are reference images selected from a first list of reference images and a second list of reference images, respectively.

[0144] The first and second reference image lists are List 0 and List 1, respectively, or vice versa. List 0 preferably contains previous reference images that are earlier than the current image in chronological order. Additionally, if the slots in List 0 are not completely filled only by previous reference images, List 0 may contain subsequent reference images that are later than the current image in chronological order. List 1 preferably contains subsequent reference images that are later than the current image in chronological order. Additionally, if the slots in List 1 are not completely filled only by subsequent reference images, List 1 may contain previous reference images that are earlier than the current image in chronological order.

[0145] In step 704, based on the temporal distance difference between the first reference image and the current image and the temporal distance difference between the second reference image and the current image, it is determined that the refinement based on bidirectional optical flow will be applied to the current block. The temporal distance difference between images can be the Picture Order Count (POC) difference. Here, POC represents the temporal playback order of the current image.

[0146] According to one implementation, it can be determined that the refinement based on bidirectional optical flow will be applied to the current block only when the magnitude of the temporal distance difference between the first reference image and the current image is the same as the magnitude of the temporal distance difference between the second reference image and the current image.

[0147] However, even if the temporal distance difference between the first reference image and the current image is different from the temporal distance difference between the second reference image and the current image, it is still possible to determine that the refinement based on bidirectional optical flow will be applied to the current block.

[0148] According to one implementation, it can be determined that the refinement based on bidirectional optical flow will be applied to the current block only when the first reference image and the second reference image are in the opposite temporal direction to the current image.

[0149] However, even if the first and second reference images are in the same temporal direction as the current image, it can be determined that the refinement based on the bidirectional optical flow will be applied to the current block.

[0150] In step 706, in response to the refined application of bidirectional optical flow, a first horizontal gradient and a first vertical gradient of the reference sample of the first reference block, and a second horizontal gradient and a second vertical gradient of the reference sample of the second reference block are determined. The gradient represents the amount of change between neighboring samples. The horizontal gradient represents the amount of change between neighboring samples in the horizontal direction, and the vertical gradient represents the amount of change between neighboring samples in the vertical direction. Gradients can be calculated for samples adjacent to the reference block and for reference samples included in the reference block. Additionally, the gradient calculated for each sample is used to determine the residual small displacement.

[0151] In step 708, the first residual small displacement of the first reference block and the second residual small displacement of the second reference block are determined based on the first horizontal gradient, the first vertical gradient, the second horizontal gradient, and the second vertical gradient. Based on the first horizontal gradient, the first vertical gradient, the second horizontal gradient, and the second vertical gradient, the autocorrelation value and cross-correlation value between the gradients are calculated. Furthermore, based on the autocorrelation value and the cross-correlation value, the first residual small displacement and the second residual small displacement are determined.

[0152] According to one implementation, the second residual small displacement of the second reference block can be determined to be equal in magnitude and opposite in direction to the first residual small displacement of the first reference block.

[0153] According to one embodiment, the second residual small displacement of the second reference block can be determined by multiplying the first residual small displacement of the first reference block by a weight. The weight may include a horizontal weight for the horizontal component of the second residual small displacement and a vertical weight for the vertical component of the second residual small displacement. The horizontal and vertical weights can be determined independently. Alternatively, the horizontal and vertical weights can have the same value. Various embodiments for determining the weights will be described below.

[0154] According to one implementation scheme, weights can be determined based on the temporal distance difference between a first reference image and the current image, and the temporal distance difference between a second reference image and the current image. The magnitude of the weights can be determined based on the ratio of the magnitude of the temporal distance difference between the first reference image and the current image to the magnitude of the temporal distance difference between the second reference image and the current image. For example, the farther the second reference image is from the current image, the greater the weight can be assigned to the second reference image. The sign of the weights can be determined based on the sign of the temporal distance difference between the first reference image and the current image, and the sign of the temporal distance difference between the second reference image and the current image. For example, if the first reference image and the second reference image are located in the same temporal direction as the current image (where the signs of the temporal distance differences are the same), the sign of the weights can be determined such that the first residual small displacement and the second residual small displacement indicate the same direction. Conversely, if the first reference image and the second reference image are located in the opposite temporal direction to the current image (where the signs of the temporal distance differences are different), the sign of the weights can be determined such that the first residual small displacement and the second residual small displacement indicate opposite directions.

[0155] According to one implementation, weights can be determined based on coding information associated with the current block. This coding information may include the video's frames per second (fps), the image's resolution, the Group of Pictures (GOPs) contained within the image, the slice type, and quantization parameters.

[0156] According to one implementation, weights can be determined based on weight information that explicitly indicates weights. Weight information can be included in the slice header, image header, image parameter set, sequence parameter set, and video parameter set associated with the current block. Alternatively, weight information can be included in two or more listed data units. Weights can be determined by combining weight information included in two or more data units. Weight information can represent any value within a predetermined range. Alternatively, weight information can represent one of a predetermined number of weight candidates.

[0157] According to one implementation, the weight can be derived from the weights of one or more neighboring blocks of the current block.

[0158] According to one implementation, when decoder-side motion vector refinement is applied to the current block, weights can be determined based on the weight information used for decoder-side motion vector refinement. The motion vectors of the current block can be refined using decoder-side motion vector refinement techniques. The motion vectors refined using decoder-side motion vector refinement techniques can be further refined using bidirectional optical flow techniques. Considering the correlation between the weights used in the two refinement techniques, the weights used in bidirectional optical flow techniques can be derived from the weights used in decoder-side motion vector refinement techniques.

[0159] In step 710, the bidirectional optical flow offset is determined based on the first residual small displacement and the second residual small displacement. A first offset median is determined based on the first residual small displacement, the first horizontal gradient, and the first vertical gradient, and a second offset median is determined based on the second residual small displacement, the second horizontal gradient, and the second vertical gradient. Furthermore, the bidirectional optical flow offset is determined based on the first offset median and the second offset median. Here, the first offset median is a value used for the first reference image, and the second offset median is a value used for the second reference image.

[0160] According to one implementation, in the step of determining the bidirectional optical flow offset, the bidirectional optical flow offset can be adaptively determined by one of a first residual small displacement and a second residual small displacement. For example, if only the first residual small displacement is used, the bidirectional optical flow offset is determined only based on the median of the first offset.

[0161] According to one implementation scheme, when the ratio of the temporal distance difference between the second reference image and the current image to the temporal distance difference between the first reference image and the current image is greater than a predetermined threshold, the bidirectional optical flow offset can be determined based solely on the first residual small displacement, without using the second residual small displacement. For example, when the predetermined threshold is 2, if the temporal distance difference between the second reference image and the current image is greater than twice the temporal distance difference between the first reference image and the current image, the second residual small displacement corresponding to the second reference image is not used to determine the bidirectional optical flow offset.

[0162] According to one implementation scheme, when the temporal distance difference between the second reference image and the current image is greater than a predetermined threshold, the bidirectional optical flow offset can be determined based solely on the first residual small displacement, without using the second residual small displacement. For example, when the predetermined threshold is 8, if the temporal distance difference between the second reference image and the current image is greater than 8, the second residual small displacement corresponding to the second reference image is not used to determine the bidirectional optical flow offset.

[0163] According to one implementation, when only the first motion vector of the first motion vector and the second motion vector of the current block are refined through decoder-side motion vector refinement, the bidirectional optical flow offset can be determined based solely on the first residual small displacement and not on the second residual small displacement. This implementation leverages the similarity between decoder-side motion vector refinement and bidirectional optical flow techniques in motion vector refinement.

[0164] According to one implementation, if the position corresponding to the first motion vector and the first residual small displacement of the current block is within the image boundary or slice boundary, and the position corresponding to the second motion vector and the second residual small displacement is outside the image boundary or slice boundary, the bidirectional optical flow offset can be determined based solely on the first residual small displacement instead of the second residual small displacement. Conversely, the bidirectional optical flow offset can be determined based solely on the second residual small displacement without considering the first residual small displacement.

[0165] In step 712, the predicted samples of the current block are refined based on the bidirectional optical flow offset.

[0166] The current block can be encoded or decoded based on the prediction method performed in steps 702 to 712. In some embodiments, the first to third syntax elements are encoded in the encoder based on the prediction results. The information can be decoded in the decoder and used for block prediction.

[0167] Additionally, the bitstream generated by the encoder according to the prediction method performed in steps 702 to 712 can be stored in a recording medium or sent to the outside of the encoder.

[0168] Figure 8 An exemplary content streaming system applicable to embodiments of the present invention is shown.

[0169] like Figure 8 As shown, the content streaming system implementing the present invention can mainly include an encoding server, a streaming server, a network server, a media storage device, a user device, and a multimedia input device.

[0170] The encoding server compresses content received from multimedia input devices such as smartphones, cameras, and CCTV into digital data to generate a bitstream, which is then sent to the streaming server. As another example, if multimedia input devices such as smartphones, cameras, and CCTV directly generate the bitstream, the encoding server can be omitted.

[0171] The bitstream can be generated by the image encoding method and / or image encoding apparatus of the present invention, and the streaming server can temporarily store the bitstream during the sending or receiving of the bitstream.

[0172] A streaming server sends multimedia data to a user device based on a user request via a web server, and the web server can act as an intermediary to notify the user of any available services. When a user requests a desired service from the web server, the web server sends it to the streaming server, and the streaming server can send multimedia data to the user. In this case, the content streaming system may include a separate control server, which can control the commands / responses between devices within the content streaming system.

[0173] A streaming server can receive content from media storage devices and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a period of time.

[0174] Examples of user devices may include mobile phones, smartphones, laptops, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, board PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital televisions, desktop computers, digital signage, etc.

[0175] In the above-mentioned content streaming system, each server can operate as a distributed server, in which case the data received from each server can be distributed and processed.

[0176] The above embodiments can be performed in the same or corresponding manner as the encoding and decoding devices. Furthermore, at least one or a combination of the above embodiments can be used to encode / decode images.

[0177] The order in which the above-described embodiments are applied can differ in the encoding and decoding devices. Alternatively, the order in which the above-described embodiments are applied can be the same in both the encoding and decoding devices.

[0178] The above-described implementation scheme can be performed on each of the luminance and chrominance signals. Alternatively, the above-described implementation scheme can be performed on both the luminance and chrominance signals in the same way.

[0179] In the above embodiments, the method is described based on a flowchart having a series of steps or units. However, the present invention is not limited to the order of the steps; on the contrary, some steps may be performed simultaneously with other steps or in a different order. Furthermore, those skilled in the art should understand that the steps in the flowchart are not mutually exclusive, and other steps may be added to the flowchart or some steps may be deleted from the flowchart without affecting the scope of the present invention.

[0180] The implementation scheme can be implemented in the form of program instructions, which can be executed by various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include individual program instructions, data files, data structures, or combinations thereof. The program instructions recorded in the computer-readable recording medium may be specifically designed and constructed for this invention or may be well known to those skilled in the art of computer software.

[0181] The bitstream generated by the encoding method according to the above embodiment can be stored in a non-volatile computer-readable recording medium. Furthermore, the bitstream stored in the non-volatile computer-readable recording medium can be decoded by the decoding method according to the above embodiment.

[0182] Examples of computer-readable recording media include: magnetic recording media such as hard disks, floppy disks, and magnetic tapes; optical data storage media such as CD-ROMs or DVD-ROMs; magneto-optical media such as floppy disks; and hardware devices such as read-only memory (ROM), random access memory (RAM), flash memory, etc., specifically configured to store and execute program instructions. Examples of program instructions include not only machine language code formatted by a compiler but also high-level language code that can be implemented by a computer using an interpreter. The hardware device may be configured to operate by one or more software modules or vice versa to perform the processes according to the invention.

[0183] While the invention has been described with respect to specific items such as detailed elements, as well as limited embodiments and drawings, these are provided only to aid in a more complete understanding of the invention, and the invention is not limited to the embodiments described above. Those skilled in the art will understand that various modifications and changes can be made based on the above description.

[0184] Therefore, the spirit of the present invention should not be limited to the above embodiments, and the entire scope of the appended claims and their equivalents shall fall within the scope and spirit of the present invention.

[0185] Industrial applicability This invention can be used in devices for encoding / decoding images and in recording media for storing bit streams.

Claims

1. An image decoding method, comprising: The prediction sample for the current block is determined by using a first reference block derived from a first reference image and a second reference block derived from a second reference image. Based on the temporal distance difference between the first reference image and the current image and the temporal distance difference between the second reference image and the current image, it is determined that the refinement based on bidirectional optical flow will be applied to the current block. In response to the refined application based on bidirectional optical flow, a first horizontal gradient and a first vertical gradient of the reference sample of the first reference block and a second horizontal gradient and a second vertical gradient of the reference sample of the second reference block are determined. Based on the first horizontal gradient, the first vertical gradient, the second horizontal gradient, and the second vertical gradient, the first residual small displacement of the first reference block and the second residual small displacement of the second reference block are determined. The bidirectional optical flow offset is determined based on the first residual small displacement and the second residual small displacement; and The prediction samples for the current block are refined based on bidirectional optical flow offset.

2. The image decoding method according to claim 1, wherein, In response to the fact that the temporal distance difference between the first reference image and the current image is different from the temporal distance difference between the second reference image and the current image, it is determined that the refinement based on bidirectional optical flow will be applied to the current block.

3. The image decoding method according to claim 1, wherein, In response to the fact that the temporal direction of the first reference image from the current image is the same as that of the first reference image from the current image, it is determined that the refinement based on the bidirectional optical flow will be applied to the current block.

4. The image decoding method according to claim 1, wherein, The second residual small displacement of the second reference block is determined to be equal in magnitude and opposite in direction to the first residual small displacement of the first reference block.

5. The image decoding method according to claim 1, wherein, The second residual small displacement of the second reference block is determined by multiplying the first residual small displacement of the first reference block by a weight.

6. The image decoding method according to claim 5, wherein, The weights include the horizontal weights for the horizontal component of the second residual small displacement and the vertical weights for the vertical component of the second residual small displacement.

7. The image decoding method according to claim 5, wherein, The weights are determined based on the temporal distance difference between the first reference image and the current image, and the temporal distance difference between the second reference image and the current image.

8. The image decoding method according to claim 7, in, The weights are determined based on the ratio of the temporal distance difference between the first reference image and the current image to the temporal distance difference between the second reference image and the current image. The sign of the weight is determined based on the sign of the temporal distance difference between the first reference image and the current image, and the sign of the temporal distance difference between the second reference image and the current image.

9. The image decoding method according to claim 5, in, Weights are determined based on the encoding information associated with the current block, and The encoding information associated with the current block includes the video's frames per second, the image's resolution, the group of pictures (GOP) included in the image, the slice type of the slice, and at least one quantization parameter.

10. The image decoding method according to claim 5, in, The weights are determined based on the weight information explicitly indicated by the weights, and The weight information is included in at least one of the slice header, image header, image parameter set, sequence parameter set, and video parameter set associated with the current block.

11. The image decoding method according to claim 5, wherein, The weights are derived from the weights of one or more neighboring blocks of the current block.

12. The image decoding method according to claim 5, wherein, When the decoder-side motion vector refinement is applied to the current block, the weights are determined based on the weight information used for the decoder-side motion vector refinement.

13. The image decoding method according to claim 1, in, In determining the bidirectional optical flow offset When the ratio of the temporal distance difference between the second reference image and the current image to the temporal distance difference between the first reference image and the current image is greater than a predetermined threshold, the bidirectional optical flow offset is determined based solely on the first residual small displacement and not on the second residual small displacement.

14. The image decoding method according to claim 1, in, In determining the bidirectional optical flow offset When the temporal distance difference between the second reference image and the current image is greater than a predetermined threshold, the bidirectional optical flow offset is determined based solely on the first residual small displacement and not on the second residual small displacement.

15. The image decoding method according to claim 1, in, In determining the bidirectional optical flow offset When the first motion vector in the first motion vector and the second motion vector of the current block are refined only through the decoder-side motion vector refinement, the bidirectional optical flow offset is determined based only on the first residual small displacement and not on the second residual small displacement.

16. The image decoding method according to claim 1, in, In determining the bidirectional optical flow offset When the position corresponding to the first motion vector and the first residual small displacement of the current block is within the image boundary or slice boundary, and the position corresponding to the second motion vector and the second residual small displacement is outside the image boundary or slice boundary, the bidirectional optical flow offset is determined based solely on the first residual small displacement, rather than the second residual small displacement.

17. An image encoding method, comprising: The prediction sample for the current block is determined by using a first reference block derived from a first reference image and a second reference block derived from a second reference image. Based on the temporal distance difference between the first reference image and the current image and the temporal distance difference between the second reference image and the current image, it is determined that the refinement based on bidirectional optical flow will be applied to the current block. In response to the refined application based on bidirectional optical flow, a first horizontal gradient and a first vertical gradient of the reference sample of the first reference block and a second horizontal gradient and a second vertical gradient of the reference sample of the second reference block are determined. Based on the first horizontal gradient, the first vertical gradient, the second horizontal gradient, and the second vertical gradient, the first residual small displacement of the first reference block and the second residual small displacement of the second reference block are determined. The bidirectional optical flow offset is determined based on the first residual small displacement and the second residual small displacement; and The prediction samples for the current block are refined based on bidirectional optical flow offset.

18. A computer-readable recording medium storing a bitstream generated by an image encoding method, the image encoding method comprising: The prediction sample for the current block is determined by using a first reference block derived from a first reference image and a second reference block derived from a second reference image. Based on the temporal distance difference between the first reference image and the current image and the temporal distance difference between the second reference image and the current image, it is determined that the refinement based on bidirectional optical flow will be applied to the current block. In response to the refined application based on bidirectional optical flow, a first horizontal gradient and a first vertical gradient of the reference sample of the first reference block and a second horizontal gradient and a second vertical gradient of the reference sample of the second reference block are determined. Based on the first horizontal gradient, the first vertical gradient, the second horizontal gradient, and the second vertical gradient, the first residual small displacement of the first reference block and the second residual small displacement of the second reference block are determined. The bidirectional optical flow offset is determined based on the first residual small displacement and the second residual small displacement; and The prediction samples for the current block are refined based on bidirectional optical flow offset.

19. A method for transmitting a bitstream generated by an image coding method, the method comprising: Encoding images based on image coding methods; as well as Transmitting a bitstream of encoded images, The image encoding method includes: The prediction sample for the current block is determined by using a first reference block derived from a first reference image and a second reference block derived from a second reference image. Based on the temporal distance difference between the first reference image and the current image and the temporal distance difference between the second reference image and the current image, it is determined that the refinement based on bidirectional optical flow will be applied to the current block. In response to the refined application based on bidirectional optical flow, a first horizontal gradient and a first vertical gradient of the reference sample of the first reference block and a second horizontal gradient and a second vertical gradient of the reference sample of the second reference block are determined. Based on the first horizontal gradient, the first vertical gradient, the second horizontal gradient, and the second vertical gradient, the first residual small displacement of the first reference block and the second residual small displacement of the second reference block are determined. The bidirectional optical flow offset is determined based on the first residual small displacement and the second residual small displacement; and The prediction samples for the current block are refined based on bidirectional optical flow offset.

Images