Image decoding device, image decoding method and program

The image decoding device and method address filter mismatch issues by controlling motion vector reference precision and refining vectors to ensure consistent filter application, improving decoding accuracy and performance.

JP7583869B2Active Publication Date: 2024-11-14KDDI CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023117078
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-07-18
Publication Date
2024-11-14
Estimated Expiration
2039-09-20

AI Technical Summary

Technical Problem

Existing image decoding technologies face issues with the mixed use of different interpolation filters in motion compensation prediction and the propagation of mismatched filter effectiveness conditions due to pairwise average merging, MMVD, or DMVR, leading to inaccuracies in motion vector reference positions.

Method used

An image decoding device and method that controls the reference pixel precision position of motion vectors using a half-pel index, refining motion vectors through MMVD or DMVR, and determines the interpolation filter type based on the motion vector and half-pel index to ensure consistent filter application.

Benefits of technology

This approach eliminates deficiencies in filter effectiveness conditions for target blocks and prevents the propagation of such issues to future blocks, enhancing decoding accuracy and performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583869000001
    Figure 0007583869000001
  • Figure 0007583869000002
    Figure 0007583869000002
  • Figure 0007583869000003
    Figure 0007583869000003
Patent Text Reader

Abstract

To eliminate the deficiency of a smoothing filter valid condition of a target block and avoid the propagation of the deficiency of the smoothing filter valid condition to future blocks.SOLUTION: An image decoding device 200 according to the present invention includes a merge unit 241A2 that decodes a motion vector and a half-pel index from a merge index, a motion vector refinement unit 241B that refines the motion vector by MMVD or DMVR, a filter determination unit 241C1 that determines whether an interpolation filter is used and the type of interpolation filter on the basis of a sophisticated motion vector and the half-pel index, and a filter application unit 241C2 that generates a motion compensation prediction pixel signal by using the interpolation filter, and the merge unit 241A2 generates a merge list by a predetermined merge list construction method, and decodes the motion vector and the half-pel index from the merge list and the merge index.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image decoding device, an image decoding method, and a program. [Background technology]

[0002] In non-patent document 1, for a technique called motion compensation prediction (hereinafter, MC prediction), techniques for deriving motion vectors (hereinafter, mv), called merge coding (hereinafter, merge) and adaptive vector coding (hereinafter, AMVP), are provided, and two types of interpolation filters are provided for generating an MC predicted image signal using the derived mv.

[0003] The first one is the same interpolation filter as in Non-Patent Document 2 (hereinafter, HEVC filter), and the second one is an interpolation filter newly introduced in Non-Patent Document 1 (hereinafter, smoothing filter).

[0004] Both are applied when the reference destination of mv is a sub-pel precision position, but the smoothing filter described above is limited to when the reference destination of mv is a 1 / 2-pel precision position, and is applied only when the half-pel index (hpelIfIdx), which indicates whether the smoothing filter described above is valid, indicates that it is valid. In other cases, the HEVC filter is applied.

[0005] The above-mentioned hpelIfIdx is inherited from adjacent blocks for which processing has already been completed when the block to be coded (hereinafter, the target block) is a merge block, and when the target block is AMVP, the value of hpelIfIdx is determined from the derived mv. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Versatile Video Coding (Draft 6), JVET-N1001 [Non-Patent Document 2] ITU-T H.265 High Efficiency Video Coding Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the above-mentioned non-patent document 1, when the target block is a merge and pair-wise average merge, MMVD (Merge Motion Vector Difference), or DMVR (Decoder-side Motion Vector Refinement) is enabled, even if the hpelIfIdx inherited from the adjacent block indicates that the smoothing filter is enabled, the pair-wise average merge, MMVD, or DMVR may cause the reference destination of mv to be different from the 1 / 2 pixel accuracy position.

[0008] In this case, the following two issues arise in the target block and in future blocks that inherit hpelIfIdx and mv from the target block. 1. Mixed use of different filters in MC prediction of a target block. 2. Propagation of inconsistency in the validity conditions of the smoothing filter for future blocks (inconsistency between hpelIfIdx and mv).

[0009] Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide an image decoding device, an image decoding method and a program that can control the reference pixel precision position of a motion vector derived by pairwise average merging or MMVD or DMVR in accordance with a half-pel index, or control the half-pel index in accordance with a derived motion vector, thereby eliminating deficiencies in the effective conditions of a smoothing filter for a target block and further avoiding the propagation of deficiencies in the effective conditions of a smoothing filter to future blocks. [Means for solving the problem]

[0010] A first feature of the present invention is an image decoding device comprising: a merge unit configured to decode, from a merge index, a motion vector and a half-pel index indicating whether the motion vector references a half-pel accuracy position; a motion vector refinement unit configured to refine the motion vector by MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement); a filter determination unit configured to determine whether an interpolation filter is used and the type of the interpolation filter based on the refined motion vector and the half-pel index; and a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, wherein the merge unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index.

[0011] A second feature of the present invention is an image decoding method comprising: a step A of decoding a motion vector and a half-pel index indicating whether the motion vector references a half-pel accuracy position from a merge index; a step B of refining the motion vector by MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement); a step C of determining whether an interpolation filter is used and the type of the interpolation filter based on the refined motion vector and the half-pel index; and a step D of generating a motion compensation prediction pixel signal using the interpolation filter, wherein in the step A, a merge list is generated by a predetermined merge list construction method, and the motion vector and the half-pel index are decoded from the merge list and the merge index.

[0012] A third feature of the present invention is a program for causing a computer to function as an image decoding device, the image decoding device comprising: a merge unit configured to decode, from a merge index, a motion vector and a half-pel index indicating whether the motion vector references a half-pel accuracy position; a motion vector refinement unit configured to refine the motion vector by MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement); a filter determination unit configured to determine whether an interpolation filter is used and the type of the interpolation filter based on the refined motion vector and the half-pel index; and a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, and the merge unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index. Effect of the Invention

[0013] According to the present invention, it is possible to control the reference pixel precision position of a motion vector derived by pairwise average merging or MMVD or DMVR according to a half-pel index, or to control the half-pel index according to a derived motion vector, thereby providing an image decoding device, image decoding method and program that can eliminate deficiencies in the effective conditions of the smoothing filter of a target block and further avoid the propagation of deficiencies in the effective conditions of the smoothing filter to future blocks. [Brief description of the drawings]

[0014] [Figure 1] 1 is a diagram illustrating an example of a configuration of an image processing system 1 according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of functional blocks of an image encoding device 100 according to an embodiment. [Diagram 3]FIG. 2 is a diagram illustrating an example of functional blocks of an inter prediction unit 111 of an image encoding device 100 according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of functional blocks of an image decoding device 200 according to an embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of functional blocks of an inter prediction unit 241 of an image decoding device 200 according to an embodiment. [Figure 6] 11 is a flowchart showing an example of a process of determining whether an interpolation filter is applied and the type of the filter in a filter determination unit 111D1 / 241C1 according to an embodiment. [Figure 7] 11 is a flowchart illustrating an example of a process for determining a half-pel index (hpelIfIdx) according to an embodiment. [Figure 8] FIG. 1 is a diagram for explaining the determination process of Non-Patent Document 1. [Figure 9] FIG. 2 is a diagram showing an example of functional blocks of an mv refinement unit 241B of an image decoding device 200 according to an embodiment. [Figure 10] 11 is a flowchart illustrating an example of a merge list construction process according to an embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a merge list generated in a merge list construction process according to an embodiment. [Figure 12] FIG. 13 illustrates an example of spatial merging according to an embodiment. [Figure 13] FIG. 13 illustrates an example of a temporal merge according to one embodiment. [Figure 14] 11 is a diagram illustrating an example of a motion vector scaling process in a temporal merge according to an embodiment. FIG. [Figure 15] FIG. 13 is a diagram illustrating an example of history merging according to an embodiment. [Figure 16] FIG. 11 is a diagram illustrating an example of a set of new motion vectors and half-pel indexes calculated by pairwise average merging according to an embodiment. [Figure 17]11 is a flowchart illustrating an example of a process for setting a new half-pel index generated by pairwise average merging according to an embodiment. [Figure 18] 11A and 11B are diagrams for explaining an example of a method for resolving deficiencies in the setting conditions of half-pel indexes in pairwise average merging according to an embodiment. [Figure 19] 11A and 11B are diagrams for explaining an example of a method for resolving deficiencies in the setting conditions of half-pel indexes in pairwise average merging according to an embodiment. [Figure 20] 11A and 11B are diagrams for explaining an example of a method for resolving deficiencies in the setting conditions of half-pel indexes in pairwise average merging according to an embodiment. [Figure 21] 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in an MMVD according to one embodiment. FIG. [Figure 22] 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in an MMVD according to one embodiment. FIG. [Diagram 23] 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in an MMVD according to one embodiment. FIG. [Figure 24] 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in an MMVD according to one embodiment. FIG. [Diagram 25] 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in an MMVD according to one embodiment. FIG. [Figure 26] FIG. 13 is a diagram for explaining a modification example in which a motion vector derived by pairwise average merging according to an embodiment is modified by MMVD or DMVR. [Figure 27] FIG. 13 is a diagram for explaining an example of derivation of a motion vector by pairwise average merging according to an embodiment. [Figure 28]FIG. 13 is a diagram for explaining an example of derivation of a motion vector by pairwise average merging according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0016] First Embodiment An image processing system 10 according to a first embodiment of the present invention will be described below with reference to Figures 1 to 28. Figure 1 is a diagram showing an image processing system 10 according to this embodiment.

[0017] As shown in FIG. 1, an image processing system 10 according to this embodiment includes an image encoding device 100 and an image decoding device 200.

[0018] The image encoding device 100 is configured to generate encoded data by encoding an input image signal. The image decoding device 200 is configured to generate an output image signal by decoding the encoded data.

[0019] The encoded data may be transmitted from the image encoding device 100 to the image decoding device 200 via a transmission path. The encoded data may be provided from the image encoding device 100 to the image decoding device 200 after being stored in a storage medium.

[0020] (Image encoding device 100) The image encoding device 100 according to this embodiment will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of functional blocks of the image encoding device 100 according to this embodiment.

[0021] As shown in FIG. 2, the image encoding device 100 includes an inter prediction unit 111, an intra prediction unit 112, a subtractor 121, an adder 122, a transform / quantization unit 131, an inverse transform / inverse quantization unit 132, an encoding unit 140, an in-loop filter processing unit 150, and a frame buffer 160.

[0022] The inter prediction unit 111 is configured to generate a prediction signal by inter prediction (inter-frame prediction).

[0023] Specifically, the inter prediction unit 111 is configured to identify a reference block contained in the reference frame by comparing a frame to be encoded (hereinafter, the target frame) with a reference frame stored in the frame buffer 160, and to determine a motion vector (mv) for the identified reference block.

[0024] Moreover, the inter prediction unit 111 is configured to generate a prediction signal included in a block to be coded (hereinafter, a target block) for each target block based on the reference block and the motion vector. The inter prediction unit 111 is configured to output the prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

[0025] The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (intra-frame prediction).

[0026] Specifically, the intra prediction unit 112 is configured to identify a reference block included in a target frame, and generate a prediction signal for each target block based on the identified reference block. The intra prediction unit 112 is also configured to output the prediction signal to the subtractor 121 and the adder 122.

[0027] Here, the reference block is a block that is referenced for the current block, for example, a block adjacent to the current block.

[0028] The subtractor 121 is configured to subtract the prediction signal from the input image signal, and output the prediction residual signal to the transformation and quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal that is the difference between the prediction signal generated by intra prediction or inter prediction and the input image signal.

[0029] The adder 122 is configured to add the prediction signal to the prediction residual signal output from the inverse transform / inverse quantization unit 132 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 112 and the in-loop filter processing unit 150.

[0030] Here, the unfiltered decoded signal forms a reference block used by the intra prediction unit 112 .

[0031] The transform / quantization unit 131 is configured to perform a transform process on the prediction residual signal and to obtain a coefficient level value. Furthermore, the transform / quantization unit 131 may be configured to quantize the coefficient level value.

[0032] Here, the transform process is a process of transforming a prediction residual signal into a frequency component signal. In the transform process, a basis pattern (transformation matrix) corresponding to a discrete cosine transform (DCT) may be used, or a basis pattern (transformation matrix) corresponding to a discrete sine transform (DST) may be used.

[0033] The inverse transform / inverse quantization unit 132 is configured to perform an inverse transform process on the coefficient level values ​​output from the transform / quantization unit 131. Here, the inverse transform / inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level values ​​prior to the inverse transform process.

[0034] Here, the inverse transformation process and inverse quantization are performed in the reverse order to the transformation process and quantization performed by the transformation and quantization unit 131 .

[0035] The encoding unit 140 is configured to encode the coefficient level values ​​output from the transform / quantization unit 131, and output encoded data.

[0036] Here, for example, the coding is entropy coding, which assigns codes of different lengths based on the probability of occurrence of coefficient level values.

[0037] Furthermore, the encoding unit 140 is configured to encode control data for use in the decoding process in addition to the coefficient level values.

[0038] Here, the control data may include size data such as a coding block (CU: Coding Unit) size, a prediction block (PU: Prediction Unit) size, a transform block (TU: Transform Unit) size, and the like.

[0039] Furthermore, the control data may include header information such as a sequence parameter set (SPS), a picture parameter set (PPS), and a slice header, as will be described later.

[0040] The in-loop filtering unit 150 is configured to perform filtering on the unfiltered decoded signal output from the adder 122 , and to output the filtered decoded signal to the frame buffer 160 .

[0041] Here, for example, the filtering process is a deblocking filtering process that reduces distortion occurring at the boundary portions of blocks (encoded blocks, predicted blocks, or transformed blocks).

[0042] The frame buffer 160 is configured to store reference frames used by the inter prediction unit 111 .

[0043] Here, the filtered decoded signal forms a reference frame used in the inter prediction unit 111.

[0044] (Inter prediction unit 111) Hereinafter, the inter prediction unit 111 of the image encoding device 100 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding device 100 according to this embodiment.

[0045] As shown in FIG. 3, the inter prediction unit 111 has an mv derivation unit 111A, an AMVR unit 111B, an mv refinement unit 111B, and a predicted signal generation unit 111D.

[0046] The inter prediction unit 111 is an example of a prediction unit configured to generate a prediction signal to be included in a current block based on a motion vector.

[0047] As shown in FIG. 3, the mv derivation unit 111A has an AMVP (Adaptive Motion Vector Prediction) unit 111A1 and a merge unit 111A2, and is configured to receive a target frame and a reference frame from the frame buffer 160 as input and obtain a motion vector.

[0048] The AMVP unit 111A1 is configured to identify a reference block included in the reference frame by comparing the target frame with the reference frame, and to search for a motion vector for the identified reference block.

[0049] Moreover, the above-mentioned search process is performed on a plurality of reference frame candidates, a reference frame and a motion vector to be used for prediction of the current block are determined, and output to the subsequent predicted signal generating unit 111D.

[0050] A maximum of two reference frames and two motion vectors can be used for one block. When only one set of reference frames and motion vectors is used for one block, it is called "uni-prediction," and when two sets of reference frames and motion vectors are used, it is called "bi-prediction." Hereinafter, the first set will be called "L0" and the second set will be called "L1."

[0051] Furthermore, when the AMVP unit 111A finally transmits the above-mentioned determined motion vector to the decoding device, in order to reduce the amount of coding, it selects an mvp from among candidates of motion vector predictors (mvp) derived from adjacent encoded motion vectors that has a small difference from the motion vector of the target block, i.e., a small motion vector difference (mvd).

[0052] The index indicating the selected mvp and mvd and the index indicating the reference frame (hereinafter, Refidx) are coded by the coding unit 140 and transmitted to the image decoding device 200. This process is generally called adaptive motion vector prediction coding (AMVP: Adaptive: Motion Vector Prediction).

[0053] Note that the above-mentioned motion vector search method, reference frame and motion vector determination method, MVP selection method, and MVD calculation method can adopt known techniques, so details thereof will be omitted.

[0054] The AMVR unit 111B has an AMVR (Adaptive Motion Vector Resolution) function that changes the transmission accuracy of the mvd calculated by the AMVP unit 111A.

[0055] Since mvd is derived from the sum of the motion vector of the current block and mvp as described above, changing the transmission accuracy of mvd means changing the accuracy of the motion vector itself of the current block.

[0056] In Non-Patent Document 1, three variations of the transmission accuracy of the mvd by AMVR are prepared. The normal transmission accuracy of the mvd is 1 / 16 pixel accuracy, and in this case, the motion vector ultimately refers to a position with 1 / 16 pixel accuracy, but when AMVR is enabled, the transmission accuracy of the mvd is selected from 1 / 4 pixel accuracy, 1 / 2 pixel accuracy, and 1 pixel accuracy (i.e., integer pixel accuracy).

[0057] However, when the target block is affine, the transmission accuracy of mvd is selected from ¼ pixel accuracy, 1 / 16 pixel accuracy, and 1 pixel accuracy. Here, affine refers to Affine in Non-Patent Document 1.

[0058] Furthermore, in Non-Patent Document 1, only when 1 / 2 pixel precision is selected as the transmission precision of the above-mentioned mvd, an interpolation filter different from the interpolation filter applied when a precision other than the 1 / 2 pixel precision is selected is selected in the subsequent predicted signal generating unit 111D. Details will be described later.

[0059] Furthermore, when the AMVR unit 111B determines that the AMVR process is valid, it transmits to the image decoding device 200 a flag indicating that the AMVR is valid and an index indicating the accuracy to which the mvd is corrected by the AMVR.

[0060] The merge unit 111A2 does not search and derive motion information of the target block as in the AMVP unit 111A1, nor transmits it as mvd as a difference with an adjacent block, but instead takes the target frame and the reference frame as input, and inherits and uses the motion information of the reference block as it is, using an adjacent block in the same frame as the target block or a block at the same position in a frame different from the target frame as the reference block. This process is generally called merge coding (hereinafter, merge).

[0061] When the target block is a merge block, a merge list for the target block is first created. The merge list is a list of a plurality of combinations of reference frames and motion vectors. An index (hereinafter, a merge index) is assigned to each combination. Instead of encoding the information of Refidx and the motion vector separately, only the merge index is encoded and transmitted to the image decoding device 200.

[0062] Here, by standardizing the method of creating a merge list on the image encoding device 100 side and the image decoding device 200 side, it is possible for the image decoding device 200 side to decode information on Refidx and motion vectors from only the merge index information. The method of creating a merge list will be described later in detail.

[0063] The mv refinement unit 111C is configured to perform a refinement process for correcting the motion vectors output from the merge unit 111A2, as will be described in detail later.

[0064] The predicted signal generation unit 111D is configured to receive a motion vector as input and output an MC predicted image signal, and includes a filter determination unit 111D1 and a filter application unit 111D2.

[0065] The filter determination unit 111D1 determines whether or not an interpolation filter is applied and the type of filter based on the motion vector, as will be described in detail later.

[0066] The filter application unit 111D2 is configured to generate a prediction signal from the selected interpolation filter, the motion vector, and the reference frame when the filter determination unit 111D1 determines that the interpolation filter is valid.

[0067] When the filter determination unit 111D1 determines that the interpolation filter is invalid, the interpolation filter is not used, and a prediction signal is generated based on the motion vector and the reference frame.

[0068] (Image Decoding Device 200) Hereinafter, the image decoding device 200 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of functional blocks of the image decoding device 200 according to this embodiment.

[0069] As shown in FIG. 4, the image decoding device 200 includes a decoding unit 210, an inverse transform / inverse quantization unit 220, an adder 230, an inter prediction unit 241, an intra prediction unit 242, an in-loop filtering unit 250, and a frame buffer 260.

[0070] The decoding unit 210 is configured to decode the coded data generated by the image coding device 100, and to decode the coefficient level values.

[0071] Here, the decoding is, for example, entropy decoding that is the reverse procedure of the entropy encoding performed by the encoding unit 140.

[0072] The decoding unit 210 may also be configured to obtain the control data by decoding the encoded data.

[0073] As mentioned above, the control data may include size data such as the coding block size, the prediction block size, the transform block size, and the like.

[0074] The inverse transform / inverse quantization unit 220 is configured to perform an inverse transform process on the coefficient level values ​​output from the decoding unit 210. Here, the inverse transform / inverse quantization unit 220 may be configured to perform an inverse quantization process on the coefficient level values ​​prior to the inverse transform process.

[0075] Here, the inverse transformation process and inverse quantization are performed in the reverse order to the transformation process and quantization performed by the transformation and quantization unit 131 .

[0076] The adder 230 is configured to add the prediction signal to the prediction residual signal output from the inverse transform / inverse quantization unit 220 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 242 and the in-loop filter processing unit 250.

[0077] Here, the unfiltered decoded signal forms a reference block used by the intra prediction unit 242 .

[0078] Similar to the inter prediction unit 111, the inter prediction unit 241 is configured to generate a prediction signal by inter prediction (inter-frame prediction).

[0079] Specifically, the inter prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from encoded data and a reference signal included in a reference frame. The inter prediction unit 241 is configured to output the prediction signal to the adder 230.

[0080] Similar to the intra prediction unit 112, the intra prediction unit 242 is configured to generate a prediction signal by intra prediction (intra-frame prediction).

[0081] Specifically, the intra prediction unit 242 is configured to identify a reference block included in the target frame, and generate a prediction signal for each prediction block based on the identified reference block. The intra prediction unit 242 is configured to output the prediction signal to the adder 230.

[0082] Similar to in-loop filter processing unit 150, in-loop filter processing unit 250 is configured to perform filtering on the unfiltered decoded signal output from adder 230, and to output the filtered decoded signal to frame buffer 260.

[0083] Here, for example, the filtering process is a deblocking filtering process that reduces distortion occurring at the boundary between blocks (encoded blocks, predicted blocks, transformed blocks, or subblocks obtained by dividing these blocks).

[0084] Similar to the frame buffer 160, the frame buffer 260 is configured to store reference frames used by the inter prediction unit 241.

[0085] Here, the filtered decoded signal forms a reference frame used by the inter prediction unit 241.

[0086] (Inter prediction unit 241) Hereinafter, the inter prediction unit 241 according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of functional blocks of the inter prediction unit 241 according to this embodiment.

[0087] As shown in FIG. 5, the inter prediction unit 241 has an mv decoding unit 241A, an mv refinement unit 241B, and a prediction signal generation unit 111C.

[0088] The inter prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

[0089] The mv decoding unit 241A has an AMVP unit 241A1 and a merge unit 241A2, and is configured to obtain motion vectors by decoding the target frame and reference frame input from the frame buffer 260, and the control data received from the image encoding device 100.

[0090] The AMVP unit 241A1 is configured to receive the target frame, the reference frame, the index indicating mvp and mvd from the image encoding device 100, Refidx, and the index indicating the transmission accuracy of mvd from the AMVR unit 111B, and decode the motion vector. Since a known method can be adopted for decoding the motion vector, details thereof will be omitted.

[0091] The merging unit 241A2 is configured to receive a merge index from the image encoding device 100 and decode a motion vector.

[0092] Specifically, the merging unit 241A2 is configured to construct a merge list and obtain a motion vector corresponding to the received merge index from the constructed merge list in the same manner as in the image encoding device 100. The method of constructing the merge list will be described later in detail.

[0093] The mv refinement unit 241B is configured to execute a refinement process to modify a motion vector, similar to the refinement unit 111C.

[0094] The predicted signal generation unit 241C has a filter decision unit 241C1 and a filter application unit 241C2, and is configured to generate a predicted signal based on a motion vector, similar to the predicted signal generation unit 111C.

[0095] Note that the filter determination unit 241C1 of the image decoding device 200 has exactly the same configuration as the filter determination unit 111D1 of the image encoding device 100, and therefore in the present embodiment, the operation of the filter determination unit 111D1 will be representatively described below.

[0096] (Interpolation filter decision process) Hereinafter, with reference to FIG. 6, a process of determining whether or not an interpolation filter is applied and the filter type in the filter determination unit 111D1 / 241C1 according to this embodiment will be described.

[0097] FIG. 6 is a flowchart showing an example of a process order for determining whether or not an interpolation filter is applied and the filter type in the filter determination unit 111D1 according to this embodiment.

[0098] 6, in step S6-1, the filter determination unit 111D1 determines whether the reference position of mv is a sub-pel position. If the reference position of mv is a sub-pel position, the process proceeds to step S6-2, and if the reference position of mv is not a sub-pel position, that is, if the reference position of mv is an integer pel value position, the process proceeds to step S6-3.

[0099] In step S6-2, filter determination section 111D1 determines to apply an interpolation filter, and the process proceeds to step S6-4.

[0100] In step S6-3, filter determination section 111D1 determines not to apply an interpolation filter, and ends this process.

[0101] In step S6-4, filter determination unit 111D1 determines whether a half-pel index (hpelIfIdx), which will be described later, is invalid, i.e., whether it is 0. If hpelIfIdx is 0, the process proceeds to step S6-5, and if hpelIfIdx is valid, i.e., not 0, the process proceeds to step S6-6.

[0102] In step S6-5, the filter determination unit 111D1 determines that the HEVC filter is to be used as the interpolation filter, and ends this process.

[0103] Here, the HEVC filter is an 8-tap linear interpolation filter similar to that described in Non-Patent Document 2.

[0104] In step S6-6, filter determination section 111D1 determines to use a smoothing filter as the interpolation filter.

[0105] Here, the smoothing filter may be, for example, a 6-tap Gaussian filter adopted in Non-Patent Document 1. In addition, the use of a plurality of filters as the interpolation filter may be set according to the value indicated by hpelIfIdx.

[0106] In this way, when the reference position of mv is a decimal pixel position, it is possible to adaptively switch between two or more types of interpolation filters based on hpelIfIdx, and it is possible to select the interpolation filter to be used in accordance with the image characteristics, which is expected to result in improved coding performance.

[0107] (Half-pel index (hpelIfIdx) setting process) Hereinafter, the process of determining the half-pel index (hpelIfIdx) according to this embodiment will be described with reference to FIG.

[0108] FIG. 7 is a flowchart showing an example of a determination process order of hpelIfIdx in the filter determination unit 111D1 according to this embodiment.

[0109] As described above, hpelIfIdx is related to the determination of the type of the interpolation filter, and when hpelIfIdx is "0", that is, when it is invalid, an interpolation filter similar to that described in Non-Patent Document 2 is used. On the other hand, when hpelIfIdx is not "0", that is, when it is valid, a smoothing filter is used.

[0110] When the current block is a merge block, hpelIfIdx is obtained from the reference block together with the motion vector and RefIdx, and is associated with the merge index.

[0111] On the other hand, when the current block is not in merge mode and the motion vector is decoded by AMVP, the value of hpelIfIdx is determined depending on whether the above-mentioned AMVR is applied and on the predetermined conditions.

[0112] 7, in step S7-1, the filter determination unit 111D1 determines whether or not AMVR is enabled. For example, as in Non-Patent Document 1, the determination may be made by decoding a flag (amvr_flag) indicating whether or not AMVR is applied.

[0113] If it is determined in step S7-1 that the AMVR is valid, the process proceeds to step S7-2, and if it is determined that the AMVR is invalid, the process proceeds to step S7-3.

[0114] In step S7-2, the filter determination unit 111D1 determines, based on the predetermined condition 1, whether the value of hpelIfIdx is "0".

[0115] If it is determined in step S7-2 that the predetermined condition 1 is satisfied, the process proceeds to step S7-4, and if it is determined that the predetermined condition 1 is not satisfied, the process proceeds to step S7-5.

[0116] Here, in the specified condition 1 of step S7-2, the index (amvr#precision#idx) indicating the transmission accuracy of the AMVR may be decoded and judged, and if the transmission accuracy of the AMVR is 1 / 2 pixel accuracy, it is judged that the specified condition 1 is satisfied, and otherwise it is judged that the specified condition 1 is not satisfied.

[0117] In addition, the specified condition 1 in step S7-2 may be determined based on whether the target block is affine. If the target block is affine, it is determined that the specified condition 1 is not satisfied, and if the target block is not affine, it is determined that the specified condition 1 is satisfied if amvr#precision#idx indicates that the transmission accuracy of mvd is 1 / 2 pixel accuracy.

[0118] Here, "Affine" refers to the Affine employed in Non-Patent Document 1, and since the present invention can employ a known method, a description thereof will be omitted.

[0119] In addition, the specified condition 1 in step S7-2 may be determined based on whether the target block is IBC. If the target block is IBC, it is determined that the specified condition 1 is not satisfied, and if the target block is not IBC, it is determined that the specified condition 1 is satisfied if amvr#precision#idx indicates that the transmission accuracy of mvd is 1 / 2 pixel accuracy.

[0120] Here, IBC refers to the Intra Block Copy (IBC) adopted in Non-Patent Document 1, and since the present invention can adopt a known method, a description thereof will be omitted.

[0121] In step S7-3, the filter determination unit 111D1 determines that hpelIfIdx is "0", and ends this process.

[0122] In step S7-4, the filter determination unit 111D1 determines that hpelIfIdx is "1" and ends this process.

[0123] In step S7-5, the filter determination unit 111D1 determines that hpelIfIdx is "0", and ends this process.

[0124] Up to this point, the determination process of hpelIfIdx has been explained using a flowchart, but as in the determination process of non-patent document 1 shown in Figure 8, the filter determination unit 111D1 may determine the Avmrshift value based on the decoding results of amvr#flag, amvr#precision#idx, and inter#affine#flag and whether the target block is IBC, and determine the value of hpelIfidx based on this Amvr#shift value.

[0125] (MV refinement department) The mv refinement unit 241B according to this embodiment will be described below with reference to Fig. 9. Note that since the mv refinement unit 111C of the encoding device has exactly the same configuration as the mv refinement unit 241B of the decoding device, this embodiment will representatively describe the mv refinement unit 241B.

[0126] FIG. 9 is a diagram showing an example of functional blocks of the mv refinement unit 241B according to this embodiment.

[0127] As shown in FIG. 9, the mv refinement unit 241B has an MMVD unit 241B1 and a DMVR unit 241B2.

[0128] Specifically, in the MMVD section 241B1 and the DMVR section 241B2, the motion vectors are refined by MMVD (Merge mode with MVD) and DMVR (Decoder side Motion Vector Refinement) adopted in Non-Patent Document 1.

[0129] Specifically, the MMVD unit 241B1 is configured to set the correction range of the reference position by distance and direction (vertical and horizontal) as shown in FIG. 9, based on the reference position of the motion vector output from the merge unit 111A2.

[0130] The MMVD unit 241B1 is configured to perform a refinement process for identifying a correction position with the smallest predetermined cost within a correction range for the correction distance, and correcting the motion vector based on the correction reference position.

[0131] In addition, the DMVR unit 241B2 is configured to set a search range based on a reference position identified by the motion vector output from the merge unit 111A2, identify a modified reference position having the smallest specified cost from within the search range, and perform a refinement process to modify the motion vector based on the modified reference position.

[0132] Here, the mv refinement unit 241B also determines whether or not to apply MMVD and whether or not to apply DMVR to the target block, but since the present invention can use the known method described in non-patent document 1 for this determination process, a description thereof will be omitted.

[0133] It should be noted that when both the MMVD and DMVR are invalid, the motion vector output from the merging unit 111A2 is not corrected and is output as is to the subsequent predicted signal generating unit 241C.

[0134] (How to build a merge list) The merge list construction process according to this embodiment will be described below with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the merge list construction process according to this embodiment.

[0135] As shown in FIG. 10, the merge list construction process according to the present invention is made up of a total of five merge list construction processes, similar to Non-Patent Document 1.

[0136] Specifically, the process includes a spatial merge in step S10-1, a time merge in step S10-2, a history merge in step S10-3, a pairwise average merge in step S10-4, and a merge list construction process using zero merge in step S10-5. Each of the merge list construction processes will be described later.

[0137] FIG. 11 is a diagram showing an example of a merge list generated in the merge list construction process according to this embodiment.

[0138] As described above, the merge list is a list in which the motion vector, Refidx, and hepelIfIdx corresponding to the merge index are registered.

[0139] Here, the maximum number of merge indexes is set to "5" in Non-Patent Document 1, but may be freely set according to the designer's intention.

[0140] Also, mvL0, mvL1, RefIdxL0, and RefIdxL1 in FIG. 11 indicate the motion vectors and reference image indexes of the reference image lists L0 and L1, respectively.

[0141] Here, the reference image lists L0 and L1 refer to lists in which reference frames are registered, and the reference frames are identified by RefIdx.

[0142] In addition, the merge list shown in FIG. 11 shows both L0 and L1 motion vectors and reference image indexes, but some reference blocks may be unidirectionally predicted. In that case, the motion vectors and reference image indexes for one direction are registered in the merge list. In addition, if there is no motion vector in the reference block, the construction process of the merge list for that reference block is skipped.

[0143] (Spatial Merge) 12 is a diagram showing spatial merging. Spatial merging is a technique for inheriting mv, RefIdx, and hpelIfIdx from an adjacent block that exists in the same frame as the target block.

[0144] Specifically, the above-mentioned mv, Refidx, and hpelfIdx are inherited from adjacent blocks having the positional relationship shown in FIG. 12, and the processing order may be the order shown in FIG.

[0145] In the merge list process, parameters are registered in the merge list in the above-mentioned processing order, but there is a check mechanism to prevent the same motion vector and reference image index from being registered in the merge list twice. This is called pruning.

[0146] The reason for the existence of the pruning process is to increase the variety of motion vectors and reference image indices registered in the merge list. From the perspective of the image encoding device 100, this means that a motion vector that has the smallest specified cost in accordance with the image characteristics can be selected. The image decoding device 200 can generate a prediction signal with high prediction accuracy based on the selected motion vector and reference image index, which is expected to result in improved encoding performance.

[0147] In spatial merge, for example, when adjacent block A1 is registered, the identity of the motion vector and reference image index registered in adjacent block A1 is confirmed in adjacent block B1, which is the next block to be processed, and if identity is confirmed, the motion vector and reference image index of adjacent block B1 are not registered in the merge list.

[0148] In addition, in Non-Patent Document 1, for adjacent blocks B0, A0, and B2, in addition to checking the identity of the motion vector and reference image index already registered in the merge list, there is also a confirmation as to whether the target block is a triangle merge.

[0149] In this embodiment, a determination based on this triangle merge may be added to the pruning process, and a known method can be used for the triangle merge, so a description thereof will be omitted.

[0150] In addition, in Non-Patent Document 1, the maximum number of merge indexes that can be registered by spatial merging is set to "4", and for adjacent block B2, if four merge indexes have already been registered by the previous spatial merging process, the processing of B2 is skipped.

[0151] In this embodiment, similarly to Non-Patent Document 1, the process for the adjacent block B2 may be determined based on the number of existing merge indexes registered.

[0152] (Time Merge) Fig. 13 is a diagram showing temporal merging. Temporal merging is a technique for identifying a block that exists in a different frame from the target block but is located at the same position as the target block (C1 in Fig. 13) or a block located at the same position as the target block (C0 in Fig. 13) as a reference block, and inheriting a motion vector and a reference image index.

[0153] The maximum number of merge indexes that can be registered in a merge list based on such a time merge list is set to "1" in Non-Patent Document 1, and the maximum number of registrations that can be registered may be set in the same manner in this embodiment.

[0154] Furthermore, the motion vectors inherited in the temporal merge are characterized by being scaled, and FIG. 14 is a diagram showing such a scaling process.

[0155] Specifically, as shown in FIG. 14, based on the distance tb between the reference frame of the target block and the frame in which the target frame is located, and the distance td between the reference frame of the reference block and the reference frame of the reference block, mv of the reference block is scaled as follows:

[0156] mv' = (td / tb) × mv In the temporal merge, this scaled mv' is registered in the merge index as the motion vector corresponding to the merge index.

[0157] (History Merge) 15 is a diagram showing history merge. History merge is a technique in which a motion vector, a reference image index, and a half-pel index of an inter-prediction block that has been coded before a target block are separately stored in a storage area called a history merge table, and when the merge list is not filled by the previous spatial merge process or temporal merge process, the merge index registered in the history merge table is registered in the merge list.

[0158] Figure 15 is a diagram showing an example of a history merge table construction process, in which a motion vector, a reference image index, and a half-pel index corresponding to a previously encoded block are registered in the history merge table using a history merge index.

[0159] The maximum number of history merge indexes to be registered in this history merge table is set to a maximum of "6" in Non-Patent Document 1, but may be freely set according to the designer's intention.

[0160] In addition, the process of registering history merge indexes in this history merge table is a FIFO process, and when the history merge table becomes full, the last registered history merge index is deleted each time a new history merge table is added.

[0161] Furthermore, similarly to Non-Patent Document 1, the history merge index registered in the history merge table may be initialized when the current block crosses a coding tree block (CTU).

[0162] (pairwise average merging) FIG. 16 is a diagram showing a pair of a new motion vector and a half-pel index calculated by pair-wise average merging.

[0163] Pairwise average merging is a technique for generating new motion vectors, reference image indexes, and half-pel indexes using motion vectors, reference image indexes, and half-pel indexes corresponding to two pairs of merge indexes already registered in a merge list.

[0164] The two sets of merge indexes used for pairwise average merging may be fixed to the 0th and 1st merge indexes registered in the merge list, as in Non-Patent Document 1, or the designer may freely set a different combination of the two sets according to his or her intention.

[0165] A method of generating a new motion vector in pair-wise average merge is to generate a new motion vector by averaging motion vectors corresponding to two pairs of merge indexes in a merge list, as shown in FIG.

[0166] Specifically, for example, when there are two motion vectors corresponding to two sets of merge indexes (i.e., in the case of bi-prediction), the pair-wise average merge motion vectors mvL0Avg and mvL1Avg are calculated independently by the motion vectors mvL0P0 / mvL1P0 and mvL0P1 / mvL1P1 in the L0 and L1 directions, respectively, as follows:

[0167] mvL0Avg=(mvL0P0+mvL0P1) / 2 mvL1Avg=(mvL1P0+mvL1P1) / 2 Here, if either mvL0P0 / mvL1P0 or mvL0P1 / mvL1P1 does not exist, the above calculation is performed excluding the non-existent motion vector.

[0168] At this time, Non-Patent Document 1 specifies that the reference image indexes RefIdxL0P0 and RefIdxL1P0 associated with the merge index P0 should always be used as the reference image indexes associated with the pairwise average merge index.

[0169] In addition, the new half-pel index hpelIfIdxAvg generated by the pair-wise average merge is set based on two pairs of half-pel indexes hpelIfIdxP0 and hpelIfIdxP1 corresponding to two pairs of merge indexes already registered in the merge list, as shown in Figures 17(a) and 17(b).

[0170] Specifically, in step S17-1, it is determined whether hpelIfIdxP0 and hpelIfIdxP1 are the same. If they are the same, the process proceeds to step S17-2, and if they are not the same, the process proceeds to step S17-3.

[0171] In step S17-2, hpelIfIdxAvg is set to hpelIfIdxP0.

[0172] In step S17-3, hpelIfIdxAvg is set to invalid, that is, to "0".

[0173] Here, when the possible values ​​of hpelIfIdxP0 and hpelIfIdxP1 are "0" or "1", the processes of steps S17-1 and S17-3 shown in FIG. 17(a) may be replaced with processes such as steps S17-4 to S17-6 in FIG. 17(b).

[0174] (Fixed the half-pel index setting conditions for pairwise average merging) Hereinafter, an example of a method for resolving deficiencies in the setting conditions of the half-pel index in pairwise average merging according to this embodiment will be described with reference to FIGS.

[0175] FIG. 18 is a diagram showing an example in which a flaw occurs in the setting conditions of the half-pel index due to the pairwise average merging according to this embodiment.

[0176] Specifically, as described above, the new motion vector and half-pel index generated by pairwise average merging are derived by averaging the motion vectors corresponding to two different sets of merge indexes already registered in the merge list and determining the half-pel index value according to specified conditions.

[0177] In the example shown in FIG. 16, the hpelIfIdxAvg generated by pairwise average merging is “1”, and both mvL0Avg and mvL1Avg are in the 1 / 2 pixel accuracy position, so the target block satisfies the validity condition of the smoothing filter described above.

[0178] On the other hand, even if hpelIfIdxAvg generated by pairwise average merging is valid, i.e., "1", and mvL0P0, mvL1P0, mvL0P1, and mvL1P1 are all at 1 / 2 pixel accuracy positions, if one of mvL0P0 and mvL0P1 to be averaged is at a 1 / 2 pixel position and the other is at an integer pixel position, mvL0Avg will refer to a 1 / 4 pixel accuracy position. This is the same for mvL1Avg generated by averaging mvL1P0 and mvL1P1 as shown in FIG. 18.

[0179] Considering that the target block generates a prediction signal using a motion vector by pairwise average merging, in this case, even though the half-pel index inherited from the surrounding reference blocks is "1", the motion vector refers to a 1 / 4 pixel accuracy position, so the effective condition of the smoothing filter is not satisfied. This causes the use of different interpolation filters between blocks, since the HEVC filter is used as the interpolation filter for the target block, while the smoothing filter is used as the interpolation filter for the surrounding reference blocks.

[0180] Furthermore, future blocks that reference the target block inherit half-pel index = 1 and mv whose reference destination is at a 1 / 4 pixel accuracy position from the target block, which leads to the propagation of a mismatch with the effective conditions of this smoothing filter to future blocks.

[0181] This problem has not been solved by known methods, and the following two methods are proposed in this embodiment. Figures 19 and 20 are diagrams showing these two solving methods.

[0182] First, the first method is shown in Fig. 19. In Fig. 19, specifically, in the case as described above, that is, in the case as shown in step S19-4, even if the hpelIfIdxAvg generated by inheritance is "1", when mvL0Avg or mvL1Avg refers to a position other than the half-pixel accuracy position, the process proceeds to step S19-5, where the inherited hpelIfIdxAvg is corrected from "1" to "0". This makes it possible to avoid improper setting of the validity conditions of the smoothing filter for the target block and future blocks as described above and the associated propagation.

[0183] The second method is shown in Fig. 20. The process shown in Fig. 20 is the same as the process shown in Fig. 19 except for step S20-5 in Fig. 20 and step S19-5 in Fig. 19, and in this embodiment, the solution method shown in step S20-5 is proposed as another solution method equivalent to step 19-5.

[0184] Specifically, in step S20-5, if the reference destinations of mvL0Avg and mvL1Avg are other than 1 / 2 pixel accuracy positions even though hpelIfIdxAvg is "1" in the previous step, the reference destinations of mvL0Avg and mvL1Avg are rounded to 1 / 2 pixel accuracy positions. This makes it possible to avoid inappropriate setting of the effective conditions of the smoothing filter and the associated propagation for the current block and future blocks.

[0185] The former solution is a technique for modifying hpelIfIdx according to mv, and the latter solution is a technique for modifying mv according to hpelIfIdx. Designers can freely decide which solution to adopt.

[0186] (Resolving deficiencies in MMVD) The above-mentioned improper setting conditions of the smoothing filter based on the half-pel index and the motion vector in the pair-wise average merging may occur even in a target block for which the MMVD is valid, because even if the reference destination of the motion vector output from the merging unit 241A2 is a 1 / 2 pixel accuracy position and the half-pel index is "1", the motion vector may be corrected by the MMVD in the MMVD unit 241B1, and the reference destination of the motion vector may be shifted from the 1 / 2 pixel accuracy position.

[0187] Therefore, we propose a solution in which hpelIfIdx is modified according to mv, similar to pairwise average merging, and a solution in which mv is modified according to hpelIfIdx.

[0188] Hereinafter, with reference to Figs. 21 to 23, an example of a method for resolving deficiencies in the setting conditions of the half-pel index in the MMVD according to this embodiment will be described.

[0189] Fig. 21 is a diagram showing the MMVD. As described above, the MMVD is configured to correct the reference position of a new motion vector based on the values ​​of mmvd#distance#idx and mmvd#direction#idx shown in Fig. 21, with the motion vector output by the merge unit 241A2 as a reference.

[0190] In Non-Patent Document 1, mvd is calculated based on the above two indexes and added to the original motion vector. When such mvd has 1 / 4 pixel accuracy, the motion vector after applying MMVD will be shifted from the 1 / 2 pixel accuracy position.

[0191] Note that even if the mvd in the MMVD has integer precision, the pixel precision of the original motion vector does not change, unlike the AMVR case described above. For example, if the original motion vector is at a 1 / 2 pixel precision position and mvd is "1", the new motion vector does not change and refers to the 1 / 2 precision position.

[0192] In order to solve the above-mentioned case, two solutions will be described below with reference to FIGS.

[0193] FIG. 22 shows the solution of modifying hpelIfIdx depending on mv.

[0194] Specifically, in step S22-1, if hpelIfIdx is "1", the process proceeds to step S22-2, and if hpelIfIdx is not "1", the process ends.

[0195] In step S22-2, if the predetermined condition 2 is satisfied, this process ends, and if the predetermined condition 2 is not satisfied, this process proceeds to step S22-3.

[0196] Here, a determination as to whether mv after application of MMVD refers to a 1 / 2 pixel accuracy position may be added to the predetermined condition 2. Also, a determination as to whether mv input to the MMVD unit 241B1 is at a 1 / 2 pixel position and mvd added by the MMVD has 1 / 2 pixel accuracy may be added.

[0197] In step S22-3, since mv is shifted from the 1 / 2 pixel accuracy position due to MMVD in step S22-2, the half-pel index is corrected from "1" to "0", and this process ends.

[0198] FIG. 23 shows the solution of modifying mv depending on hpelIfIdx.

[0199] Specifically, another solution equivalent to the solution shown in step S22-3 in FIG. 22, that is, a process of rounding mv after application of MMVD to a 1 / 2 pixel accuracy position, is implemented in step S23-3.

[0200] [Example of change: Variation of rounding process for MMVD] Below, a description will be given of variations in the rounding process of mv when mv is modified by MMVD shown in step S23-3 above.

[0201] The method described above involves directly rounding mv after applying MMVD to a half-pixel accuracy position, but as an alternative, mvd when applying MMVD may be rounded indirectly to a half-pixel accuracy position.

[0202] For example, a method of changing the table of Fig. 24 showing the offset value of mvd corresponding to mmvd#distance#idx can be considered. Specifically, the following two methods can be considered.

[0203] The first method is the idea of ​​adding a check for hpelIfIdx to the table in Figure 24. Figure 25 shows the table with the check for hpelIfIdx added.

[0204] In the conventional table shown in Figure 24, the offset value of the mvd by the MMVD was determined only by slice#fpel#mmvd#enabled#flag, which is a slice-level determination flag that indicates whether the correction accuracy of the MMVD is to be integer pixel precision.However, in Figure 25, the table for the case where slice#fpel#mmvd#enabled#flag is "0", i.e., when the correction accuracy of the motion vector by the MMVD is allowed up to decimal pixel precision, is divided into two based on the value of hpelIfIdx.

[0205] When hpelIfIdx is "0", mmvd#distance#idx starts from "1", and when hpelIfIdx is "1", mmvd#distance#idx starts from "2".

[0206] The offset value of the actual MVD of the MMVD in Non-Patent Document 1 is multiplied by the value shown in the table by 4, and then multiplied by 1 / 16 pixel precision, which is the calculation precision of the motion vector in Non-Patent Document 1, so that when MmvdDistance shown in Fig. 25 is "1", the motion vector after application of the MMVD has 1 / 4 pixel precision, and when MmvdDistance is "2", the motion vector after application of the MMVD has 1 / 2 pixel precision. As a result, it is possible to eliminate the defects in mv after application of the half-pel index and MMVD described above.

[0207] (Resolving DMVR issues) The above-mentioned pairwise average merging, half-pel index in MMVD, and improper setting conditions of the smoothing filter based on the motion vector may occur in a target block for which DMVR is valid, because even if the reference destination of the motion vector output from the merging unit 241A2 is a 1 / 2 pixel accuracy position and the half-pel index is "1", the motion vector may be corrected by the DMVR in the DMVR unit 241B2, and the reference destination of the motion vector may be shifted from the 1 / 2 pixel accuracy position.

[0208] Therefore, in this modified example, we propose a solution of modifying hpelIfIdx according to mv, as in pairwise average merging, and a solution of modifying mv according to hpelIfIdx. However, since the configuration example is exactly the same as the solution in the MMVD described above, i.e., the configuration example shown in Figures 22 and 23, a description will be omitted.

[0209] [Example of change: Variation of rounding process for DMVR] The following describes the rounding process for mv after application of DMVR, similar to the variation of the rounding process for mv after application of MMVD described above.

[0210] As an example of rounding variations in MMVD, we introduced a technique to directly round mv after applying MMVD, or to indirectly round mv after applying MMVD by rounding mvd when applying MMVD, but the variations regarding DMVR are basically the same, so we will omit the explanation of the overlapping parts. On the other hand, we will explain the parts where there are differences in the rounding processes of DMVR and MMVD.

[0211] Specifically, in DMVR, mvd is calculated independently for the integer part and the decimal part. Therefore, when mvd is selected as the target for rounding, the sum of mvd may be directly rounded as in the above-mentioned MMVD, but in DMVR, only the decimal part of mvd may be rounded. This is because the cases where mv deviates from the 1 / 2 pixel accuracy position are limited to the cases where the decimal part of mvd by DMVR becomes other than 1 / 2 pixel accuracy.

[0212] In addition, the derivation of mvd in DMVR is configured so that the position of the motion vector to be corrected is identified by searching as described above, and then calculated using a parabolic fitting process.By performing a rounding process in this parabolic fitting process, mv after application of DMVR can be rounded to a 1 / 2 pixel accuracy position.

[0213] [Changes: Other changes related to DMVR] Other examples of changes to the DMVR are described below.

[0214] Since DMVR involves searching for a motion vector to be modified, it is known that DMVR is a relatively heavy process for the image decoding device 200. Therefore, in order to facilitate the realization of parallelization of block processing, the motion vector after application of DMVR is not used, for example, as a determination condition for a deblocking filter, and the motion vector before application of DMVR is used.

[0215] The same is true for half-pel indexes. In non-patent document 1, the motion vector used to generate a prediction signal for the target block is the one after DMVR is applied, but for example, the motion vector inherited by surrounding future blocks in the merge process is the one before DMVR is applied.

[0216] Here, the solutions shown in FIG. 22 and FIG. 23 are all to check the reference position of mv after application of DMVR, and then to modify hpelIfIdx or mv.

[0217] However, with this solution, correction processing must be performed after DMVR is applied, which may reduce the feasibility of parallelizing block processing.

[0218] Therefore, the following three measures can be taken as alternatives:

[0219] The first method is that when DMVR is enabled for the target block, as described above, the mv is modified only for the target block, and the mv inherited for future blocks that reference the target block inherits the mv before DMVR was applied.

[0220] The second method is to always set hpelIfIdx to "0" when it is determined that DMVR may be enabled in the target block. The timing at which it can be determined that DMVR may be enabled in the target block is when control data is decoded by CABAC, and since a known method can be used as the timing in this modification example, a description thereof will be omitted.

[0221] The third method is to disable DMVR for the target blocks where hpelIfIdx is "1".

[0222] By taking any of the above three alternative methods, it is expected that the decrease in feasibility of parallelizing the processing of the above-mentioned blocks can be avoided.

[0223] [Example of change: Resolved the problem when pairwise average and MMVD / DMVR are in series] Hereinafter, with reference to FIG. 26, a modification example in which the motion vector derived by the pairwise average merging according to this embodiment is corrected by the MMVD or DMVR will be described.

[0224] FIG. 26 illustrates a modified example where motion vectors derived by pairwise average merging are modified by MMVD or DMVR.

[0225] In the above example, a solution was described in which if the half-pel index is "1" and the motion vector deviates from the 1 / 2 pixel accuracy position due to pairwise average merging, the half-pel index is corrected to "0".

[0226] However, after the half-pel index is corrected to "0", the motion vector that has shifted from the 1 / 2 pixel accuracy position by the MMVD or DMVR may possibly return to the 1 / 2 pixel accuracy position again.

[0227] In such a case, it is ideal to correct the half-pel index that was once corrected from "1" to "0" back to "1". An example of this method will be described with reference to FIG.

[0228] In addition, the target of MMVD in Non-Patent Document 1 is limited to the merge index "0" and the first motion vector. At this time, motion vectors derived by pairwise average merging are not subject to NNVD. However, assuming that the valid targets of MMVD will be expanded in the future to include motion vectors with merge index "2" or more, an example of such a method will be described below.

[0229] First, up to step S26-4, the flowchart is the same as that in the pairwise average merging described above, that is, the flowchart shown in FIG. 19, but the processing from step S26-4 onwards has been changed to take into account the above-mentioned case.

[0230] Specifically, in step S26-4, if the motion vector deviates from the 1 / 2 pixel accuracy position, hpelIfIdxAvg is corrected to "0", and hpelIfIdxAvg' is set to "1".

[0231] Here, hpelIfIdxAvg' is an index to hold the unmodified value of the half-pel index for pairwise average merging.

[0232] It should be noted that in step S26-4, if the motion vector does not deviate from the half-pixel accuracy position, hpelIfIdx remains at "1" without being corrected, and hpelIfIdxAvg' is set to "1".

[0233] Next, in step S26-7, a judgment is made regarding specified condition 3, and if it is judged that specified condition 3 is satisfied, the process proceeds to step S26-8, and if it is judged that specified condition 3 is not satisfied, the process terminates.

[0234] Here, a determination as to whether the motion vector after application of MMVD or DMVR refers to a half-pixel precision position may be added to the predetermined condition 3. Also, a determination as to whether the mvd added to the motion vector by MMVD or DMVR is half-pixel precision may be added.

[0235] Next, in step S26-8, predetermined condition 4 is judged, and if it is determined that predetermined condition 4 is satisfied, the process proceeds to step S26-9, and if it is determined that predetermined condition 4 is not satisfied, the process ends.

[0236] Here, the predetermined condition 4 may include a determination as to whether hpelIfIdxAvg and hpelIfIdxAvg' are both "1".

[0237] In step S26-9, hpelIfIdx, which was once corrected from "1" to "0" in the previous step, is re-corrected to "1" because it has been determined by the MMVD or DMVR that mv refers to a 1 / 2 pixel accuracy position.

[0238] (Scaling of motion vectors in pairwise average merging) Hereinafter, the scaling process of the motion vector in the pairwise average merging according to the present embodiment will be described with reference to Fig. 27 and Fig. 28. Fig. 27 is a diagram showing an example of derivation of the motion vector by the pairwise average merging according to the present embodiment.

[0239] As shown in FIG. 27, the pairwise average merge is derived using the motion vectors and reference image indexes associated with two different merge indexes already registered in the merge list.

[0240] In Non-Patent Document 1, the merge indexes used in the pairwise average merge are specified as 0th and 1st. If only one of the two merge indexes used in the pairwise average merge can be used, the one available merge index is registered in the merge list as a merge index associated with the pairwise average merge index.

[0241] That is, a pair of a motion vector and a reference image index that is available is registered as a motion vector and a reference image index that are directly associated with the pairwise average merge index.

[0242] Also, if neither of the two different merge indexes is available, the pairwise average merge is not registered in the merge list.

[0243] For example, when there are motion vectors mvL0P0 / mvL1P0 and mvL0P1 / mvL1P1 associated with two different pairs of merge indexes used for the derivation, mvL0Avg and mvL1Avg, which are newly derived vectors by pairwise average merging, are derived as follows:

[0244] mvL0Avg=(mvL0P0+mvL0P1) / 2 mvL1Avg=(mvL1P0+mvL1P1) / 2 Here, if either mvL0P0 / mvL1P0 or mvL0P1 / mvL1P1 does not exist, the above calculation is performed excluding the non-existent motion vector.

[0245] At this time, Non-Patent Document 1 specifies that the reference image indexes RefIdxL0P0 and RefIdxL1P0 associated with the merge index P0 should always be used as the reference image indexes associated with the pairwise average merge index.

[0246] However, the reference frames of mvL0P1 and mvL1P1 averaged by the above formula should correspond to RefIdxL0P1 and RefIdxL1P1, which are reference image indexes associated with merge index "1".

[0247] Therefore, in this embodiment, mvL0P1 and mvL1P1 are replaced with mvL0P1' and mvL0P1' scaled by RefIdxL0P0, RefIdxL1P0, RefIdxL0P1 and RefIdxL1P1, and new motion vectors are derived by pair-wise average merging.

[0248] Here, the scaling process performed on the above-mentioned motion vector will be described with reference to FIG.

[0249] For example, as shown in FIG. 28, there are motion vectors and reference image indices corresponding to two different merge indices used to derive the pairwise average merge.

[0250] At this time, this means that the scaled mvL0P1' and mvL0P1' are corrected based on the distances between the target frame and the reference frames indicated by RefIdxL0P0, RefIdxL1P0, RefIdxL0P1, and RefIdxL1P1 which are the reference image indices of the target frame and each merge index.

[0251] Specifically, as shown in FIG. 28, when the distances between the target frame and each reference frame are expressed as tL0P0, tL0P1, tL1P0, and tL1P1, mvL0P1' and mvL0P1' are calculated as follows.

[0252] mvL0P1' = (tL0P1 / tL0P0) × mvL0P1 mvL1P1' = (tL1P1 / tL1P0) × mvL1P1 If the merge index 1 used for pairwise average merging corresponds to the motion vector and reference image index associated with the temporal merge (i.e., when the reference frame is not the same as the target frame, as in spatial merge or history merge), the distance is calculated and scaled starting from the reference frame that contains the reference block that is in the same position as the target block in a different frame during temporal merging.

[0253] As an effect of introducing the above-mentioned scaling process, when the pairwise average merge is calculated from motion vectors associated with two different merge indexes, by scaling one of the motion vectors to the other reference image index, two pairs of motion vectors corresponding to the reference frames indicated by the common reference image index can be averaged, thereby improving the accuracy of the motion vectors and, as a result, improving the coding performance.

[0254] [Example of change: Change of scaling target] In the above example, the motion vectors associated with the merge index P1 are scaled, but the motion vectors that refer to reference frames farther from the target frame may be scaled. This allows the reference frame for pairwise average merging to be closer to the target frame, which is expected to improve prediction accuracy.

[0255] (Half-pel index control by block size) Hereinafter, the control of the half-pel index based on the target block size according to this embodiment will be described.

[0256] In the above example, it has been described that the smoothing filter is applied when the half-pel index is valid and the motion vector is in a 1 / 2 pixel position.

[0257] On the other hand, if the block size of the target block is small, it is better not to apply a smoothing filter to the target block.

[0258] This is because it is generally known that there is a correlation between block size and image characteristics; small block sizes tend to have edges and complex patterns, while large block sizes tend to be flat and simple.

[0259] In addition, since the reference block of the target block is also assumed to have the same characteristics, it is possible to take these characteristics into account and prohibit the application of a smoothing filter when the block size of the target block is small, i.e., to always set the value of the half-pel index to "0".

[0260] For example, the block size threshold for always setting the half-pel index to 0 may be set to 32 pixels or less.

[0261] The expected effects are the following two.

[0262] First, in a certain block size where the half-pel index is always set to "0", the application of a smoothing filter is prohibited, eliminating the need for RDO processing on the image encoding device 100 side. This is expected to result in an increase in the speed of the image encoding device 100.

[0263] Second, when the target block shown in FIG. 8 is both AMVP and AMVR, the bit length of the flag used to determine the half-pel index, specifically, amvr_precision_idx, can be shortened.

[0264] In Figure 8, since there are variations in which the target block indicates a 1 / 2 pixel accuracy position, the bit length of amvr_precision_idx requires 3 bits, but this can be shortened to 2 bits for the specified block size mentioned above. As a result, the amount of code that needs to be transmitted from image encoding device 100 to image decoding device 200 is reduced, and improved encoding performance can be expected.

[0265] According to the present invention, it is possible to control the reference pixel precision position of a motion vector derived by pairwise average merging or MMVD or DMVR according to a half-pel index, or to control the half-pel index according to a derived motion vector, thereby providing an image decoding device, image decoding method and program that can eliminate deficiencies in the effective conditions of the smoothing filter of a target block and further avoid the propagation of deficiencies in the effective conditions of the smoothing filter to future blocks.

[0266] The above-described image encoding device 100 and image decoding device 200 may be realized as a program that causes a computer to execute each function (each process).

[0267] In each of the above-described embodiments, the present invention has been described by taking as an example its application to the image encoding device 100 and the image decoding device 200. However, the present invention is not limited to this, and can be similarly applied to an image encoding system and an image decoding system having the functions of the image encoding device 100 and the image decoding device 200. [Explanation of symbols]

[0268] 10. Image processing system 100...Image encoding device 111, 241…Inter prediction section 111A…mv derivation part 111A1, 241A1…AMVP section 111A2, 241A2…Merge section 111B…AMVR section 111C, 241B...mv refinement department 111D, 241C...Prediction signal generation unit 111D1, 241C1...Filter judgment section 111D2, 241C2...Filter application section 112, 242...Intra prediction section 121…Subtractor 122, 230…Adder 131...Transformation and quantization section 132, 220...Inverse transform and inverse quantization unit 140...encoding section 150, 250...In-loop filter processing section 160, 260... frame buffer 200...Image decoding device 210…Decoding section 241A…mv decoding section 241B1…MMVD section 241B2…DMVR section

Claims

1. An image decoding device, a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal; a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the motion vector and the half-pel index; a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index; The target block is a merge. The image decoding device, wherein the merging unit is configured to control a value of the half-pel index to be derived depending on a reference position of the motion vector derived by pair-wise average merging.

2. An image decoding device, a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal; a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the motion vector and the half-pel index; a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index; The target block is a merge. The image decoding device, wherein the merging unit is configured to control a reference position of the motion vector to be derived in accordance with the half-pel index derived by pair-wise average merging.

3. An image decoding device, a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal; a motion vector refinement unit configured to refine the motion vector by MMVD (Merge Motion Vector Difference); a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the refined motion vector and the half-pel index; a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index; The target block is a merge. The image decoding device, characterized in that, when MMVD is applied, the motion vector refinement unit is configured to control a reference position of a motion vector after application of MMVD in accordance with the half-pel index.

4. An image decoding device, a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal; a motion vector refinement unit configured to refine the motion vector by a decoder-side motion vector refinement (DMVR); a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the refined motion vector and the half-pel index; a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index; The target block is a merge. An image decoding device characterized in that, when DMVR is applied, the motion vector refinement unit is configured to control the half-pel index depending on the reference position of the motion vector after application of DMVR.

5. An image decoding device, a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal; a motion vector refinement unit configured to refine the motion vector by a decoder-side motion vector refinement (DMVR); a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the refined motion vector and the half-pel index; a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index; The target block is a merge. An image decoding device characterized in that, when DMVR is applied, the motion vector refinement unit is configured to control the reference position of the motion vector after application of DMVR in accordance with the half-pel index.

6. A step A of decoding a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal from the merge index; a step B of determining whether an interpolation filter is used and a type of the interpolation filter based on the motion vector and the half-pel index; and C. generating a motion compensation prediction pixel signal using the interpolation filter, In the step A, a merge list is generated by a predetermined merge list construction method, and the motion vector and the half-pel index are decoded from the merge list and the merge index; The target block is a merge. An image decoding method, comprising the steps of: controlling a value of a half-pel index to be derived in the step A according to a reference position of the motion vector derived by pair-wise average merging.

7. A program for causing a computer to function as an image decoding device, The image decoding device comprises: a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal; a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the motion vector and the half-pel index; a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index; The target block is a merge. The program, wherein the merging unit is configured to control a value of a half-pel index to be derived depending on a reference position of the motion vector derived by pair-wise average merging.

Citation Information

Patent Citations

  • Using Interpolation Filters for History-Based Motion Vector Prediction.

    JP2022527896A

  • Reference picture resampling using switchable filters.

    JP2022539455A