Image decoding device, image decoding method, and program product

Abstract

An image decoding device (200) according to the present invention is provided with: a merging unit (241A2) that decodes a motion vector and a half-pixel index on the basis of a merging index; a motion vector refinement unit (241B) that refines the motion vector by MMVD or DMVR; a filter determination unit (241C1) that determines whether or not to use an interpolation filter and an interpolation filter type on the basis of the refined motion vector and the half-pixel index; and a filter application unit (241C2) that generates a motion-compensated prediction pixel signal using an interpolation filter. Furthermore, the merging unit (241A2) generates a merge list by a specific merge list construction method, and decodes the motion vector and the half-pixel index on the basis of the merge list and the merge index. According to the invention, the defect of the effective condition of the smoothing filter of the object block can be eliminated, and the defect of the effective condition of the smoothing filter can be further prevented from being transmitted to the subsequent block.

Description

[0001] This application is a divisional application of an application with a filing date of August 21, 2020, application number 202080057645.2, and invention name “Image decoding device, image decoding method and program”. Technical Field

[0002] The present invention relates to an image decoding device, an image decoding method and a program product. Background Art

[0003] In non-patent document 1, for the technology of motion compensated prediction (hereinafter referred to as MC prediction), there are technologies such as merge coding (hereinafter referred to as merge) and adaptive vector coding (hereinafter referred to as AMVP) for deriving motion vectors (hereinafter referred to as mv), and two interpolation filters are prepared for generating MC prediction image signals using the derived mv.

[0004] The first one is the same interpolation filter as that in Non-Patent Document 2 (hereinafter referred to as HEVC filter), and the second one is the interpolation filter newly introduced in Non-Patent Document 1 (hereinafter referred to as smoothing filter).

[0005] Although both are applicable to the case where the reference target of mv is at a fractional pixel precision position, the above smoothing filter is limited to the case where the reference target of mv is at a 1 / 2 pixel precision position, and is only applicable to the case where the display half-pixel index (hereinafter referred to as hpelIfIdx) is valid, the half-pixel index indicates whether the above smoothing filter is valid. In other cases, the HEVC filter is applied.

[0006] When a block of a coding object (hereinafter referred to as an object block) is merged, the hpelIfIdx is used from the adjacent block that has been processed. When the object block is AMVP, the value of hpelIfIdx is determined based on the derived mv.

[0007] [Prior art literature]

[0008] Non-patent literature

[0009] Non-patent document 1: Versatile Video Coding (Draft 6), JVET-N1001 Non-patent document 2: ITU-T H.265 High Efficiency Video Coding Summary of the invention

[0010] Problems to be solved by the invention

[0011] However, in the above-mentioned non-patent document 1, when the object blocks are merged and pair-wise average merge or MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement) is valid, even if hpelIfIdx inherited from the adjacent block indicates that the smoothing filter is valid, the reference target of mv is sometimes not at the 1 / 2 pixel accuracy position due to pair-wise average merge or MMVD or DMVR.

[0012] At this time, the following two problems will occur in the object block and the subsequent blocks that inherit hpelIfIdx and mv from the object block:

[0013] 1. Mix different filters in MC prediction of the object block;

[0014] 2. The inconsistency in the effective condition of the smoothing filter (inconsistency between hpelIfIdx and mv) will be passed on to the subsequent blocks.

[0015] Therefore, the present invention is completed in view of the above-mentioned problems, and its purpose is to provide an image decoding device, an image decoding method and a program product, which can control the reference pixel accuracy position of the motion vector derived by paired average merging or MMVD or DMVR according to the half-pixel index, or can control the half-pixel index according to the derived motion vector, so as to eliminate the defects of the effective conditions of the smoothing filter of the object block and further avoid the defects of the effective conditions of the smoothing filter from being transmitted to subsequent blocks.

[0016] Means for solving problems

[0017] The main purpose of the first technical solution of the present invention is an image decoding device, including: a merging unit, configured to decode a motion vector and a half-pixel index according to a merge index, wherein the half-pixel index indicates whether the motion vector refers to a 1 / 2 pixel precision position; a motion vector refinement unit, configured to refine the motion vector through MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement); a filter judgment unit, configured to judge whether to use an interpolation filter and the type of interpolation filter based on the refined motion vector and the half-pixel index; and a filter application unit, configured to use the interpolation filter to generate a motion compensated predicted pixel signal; and the merging unit is configured to generate a merge list through a specific merge list construction method, and decode the motion vector and the half-pixel index according to the merge list and the merge index.

[0018] The second technical solution of the present invention is a method for decoding an image, comprising: step A, decoding a motion vector and a half-pixel index according to a merge index, wherein the half-pixel index indicates whether the motion vector refers to a 1 / 2 pixel precision position; step B, refining the motion vector by MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement); step C, judging whether to use an interpolation filter and the type of interpolation filter based on the refined motion vector and the half-pixel index; and step D, using the interpolation filter to generate a motion compensated predicted pixel signal; and, in step A, generating a merge list by a specific merge list construction method, and decoding the motion vector and the half-pixel index according to the merge list and the merge index.

[0019] The third technical solution of the present invention is a program product, including a program for enabling a computer to function as an image decoding device, the image decoding device including: a merging unit, configured to decode a motion vector and a half-pixel index according to a merge index, the half-pixel index indicating whether the motion vector refers to a 1 / 2 pixel precision position; a motion vector refinement unit, configured to refine the motion vector through MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement); a filter determination unit, configured to determine whether to use an interpolation filter and an interpolation filter type based on the refined motion vector and the half-pixel index; and a filter application unit, configured to generate a motion compensated predicted pixel signal using the interpolation filter; and the merging unit is configured to generate a merge list through a specific merge list construction method, and decode the motion vector and the half-pixel index according to the merge list and the merge index.

[0020] Effects of the Invention

[0021] According to the present invention, an image decoding device, an image decoding method and a program product can be provided, which can control the reference pixel precision position of the motion vector derived by paired average merging or MMVD or DMVR according to the half-pixel index, or can control the half-pixel index according to the derived motion vector, so as to eliminate the defects of the effective conditions of the smoothing filter of the object block and further avoid the defects of the effective conditions of the smoothing filter from being transmitted to subsequent blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a diagram showing an example of the configuration of an image processing system 10 according to one embodiment.

[0023] Figure 2 This is a diagram showing an example of a functional block of the image encoding device 100 according to one embodiment.

[0024] Figure 3 FIG. 1 is a diagram showing an example of a functional block of the inter-frame prediction unit 111 of the image encoding device 100 according to one embodiment.

[0025] Figure 4 This is a diagram showing an example of a functional block of an image decoding device 200 according to an embodiment.

[0026] Figure 5 This is a diagram showing an example of a functional block of the inter prediction unit 241 of the image decoding device 200 according to one embodiment.

[0027] Figure 6 This is a flowchart showing an example of a process for determining whether to apply an interpolation filter and a filter type in the filter determination unit 111D1 / 241C1 according to one embodiment.

[0028] Figure 7 This is a flowchart showing an example of a half-pixel index (hpelIfIdx) determination process according to one embodiment.

[0029] Figure 8 This is a diagram for explaining the determination process of Non-Patent Document 1.

[0030] Fig. 9 This is a diagram showing an example of a functional block of the mv refinement unit 241B of the image decoding device 200 according to an embodiment.

[0031] Fig.10 This is a flowchart showing an example of a merge list construction process according to one embodiment.

[0032] Fig.11 This is a diagram showing an example of a merged list generated by a merged list building process according to an embodiment.

[0033] Fig.12 This is a diagram showing an example of space merging according to one embodiment.

[0034] Fig.13 This is a diagram showing an example of time integration according to one embodiment.

[0035] Fig.14 This is a diagram for explaining an example of a motion vector scaling process in temporal merging according to one embodiment.

[0036] Fig.15 This is a diagram showing an example of history merging according to one embodiment.

[0037] Fig.16 This is a diagram showing an example of a combination of a new motion vector and a half-pixel index calculated by pairwise average merging according to one embodiment.

[0038] Fig.17 This is a flowchart showing an example of a process for setting a new half-pixel index generated by pairwise average merging according to one embodiment.

[0039] Fig.18 This is a diagram for explaining an example of a defect elimination method for setting conditions of half-pixel indices in pairwise average merging according to one embodiment.

[0040] Fig.19This is a diagram for explaining an example of a defect elimination method for setting conditions of half-pixel indices in pairwise average merging according to one embodiment.

[0041] Fig. 20 This is a diagram for explaining an example of a defect elimination method for setting conditions of half-pixel indices in pairwise average merging according to one embodiment.

[0042] Fig.21 This is a diagram for explaining an example of a defect elimination method for setting conditions of a half-pixel index in an MMVD according to an embodiment.

[0043] Fig. 22 This is a diagram for explaining an example of a defect elimination method for setting conditions of a half-pixel index in an MMVD according to an embodiment.

[0044] Fig.23 This is a diagram for explaining an example of a defect elimination method for setting conditions of a half-pixel index in an MMVD according to an embodiment.

[0045] Fig.24 This is a diagram for explaining an example of a defect elimination method for setting conditions of a half-pixel index in an MMVD according to an embodiment.

[0046] Fig.25 This is a diagram for explaining an example of a defect elimination method for setting conditions of a half-pixel index in an MMVD according to an embodiment.

[0047] Fig.26 This is a diagram for explaining an example of changes when the motion vector derived by pairwise average merging is corrected by MMVD or DMVR according to one embodiment.

[0048] Fig. 27 This is a diagram for explaining an example of deriving motion vectors by pairwise average merging according to one embodiment.

[0049] Fig.28 This is a diagram for explaining an example of deriving motion vectors by pairwise average merging according to one embodiment. DETAILED DESCRIPTION

[0050] The following is an explanation of the embodiments of the present invention with reference to the accompanying drawings. In addition, the constituent elements in the following embodiments may be appropriately replaced with existing constituent elements, etc., and various modifications including combinations with other existing constituent elements may be performed. Therefore, the content of the invention recorded in the claims should not be limited by the description of the following embodiments.

[0051] <First embodiment>

[0052] Below, reference Figures 1 to 28An image processing system 10 according to a first embodiment of the present invention will be described. Figure 1 1 is a diagram showing an image processing system 10 according to the present embodiment.

[0053] like Figure 1 As shown, the image processing system 10 of this embodiment includes an image encoding device 100 and an image decoding device 200 .

[0054] The image encoding device 100 is configured to generate encoded data by encoding an input image signal, and the image decoding device 200 is configured to generate an output image signal by decoding the encoded data.

[0055] The encoded data may be transmitted from the image encoding device 100 to the image decoding device 200 via a transmission path. The encoded data may also be provided from the image encoding device 100 to the image decoding device 200 after being stored in a storage medium.

[0056] (Image Coding Device 100)

[0057] Below, reference Figure 2 The image encoding device 100 according to this embodiment will be described. Figure 2 This is a diagram showing an example of functional blocks of the image encoding device 100 according to the present embodiment.

[0058] like Figure 2 As shown, the image encoding device 100 includes an inter-frame prediction unit 111, an intra-frame prediction unit 112, a subtractor 121, an adder 122, a transform and quantization unit 131, an inverse transform and inverse quantization unit 132, an encoding unit 140, a loop filter processing unit 150 and a frame buffer 160.

[0059] The inter-frame prediction unit 111 is configured to generate a prediction signal by inter-frame prediction.

[0060] Specifically, the inter-frame prediction unit 111 is configured to specify a reference block included in the reference frame by comparing a frame to be encoded (hereinafter referred to as a target frame) with a reference frame stored in the frame buffer 160, and determine a motion vector (mv) of the specified reference block.

[0061] The inter-frame prediction unit 111 is configured to generate a prediction signal included in a block to be coded (hereinafter referred to as a target block) for each target block based on the reference block and the motion vector. The inter-frame prediction unit 111 is configured to output the prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

[0062] The intra prediction unit 112 is configured to generate a prediction signal by intraframe prediction.

[0063] Specifically, the intra prediction unit 112 is configured to specify a reference block included in the target frame, and to generate a prediction signal for each target block based on the specified reference block. Furthermore, the intra prediction unit 112 is configured to output the prediction signal to the subtractor 121 and the adder 122 .

[0064] Here, the reference block is a block that the object block refers to. For example, the reference block is a block adjacent to the object block.

[0065] The subtractor 121 is configured to subtract the prediction signal from the input image signal and output a prediction residual signal to the transform and quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal which is a difference between a prediction signal generated by intra prediction or inter prediction and the input image signal.

[0066] The adder 122 is configured to add the prediction signal and the prediction residual signal output from the inverse transform and inverse quantization unit 132 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 112 and the loop filter processing unit 150 .

[0067] Here, the decoded signal before filtering constitutes a reference block used in the intra prediction unit 112 .

[0068] The transform and quantization unit 131 is configured to perform a transform process on the prediction residual signal and obtain a coefficient level value. In addition, the transform and quantization unit 131 may be configured to perform quantization on the coefficient level value.

[0069] Here, the transformation process is a process of transforming the prediction residual signal into a frequency component signal. In this transformation process, a base model (transformation matrix) corresponding to a discrete cosine transform (DCT) or a base model (transformation matrix) corresponding to a discrete sine transform (DST) can be used.

[0070] The inverse transform and inverse quantization unit 132 is configured to perform an inverse transform process on the coefficient level value output from the transform and quantization unit 131. Here, the inverse transform and inverse quantization unit 132 may be configured to perform an inverse quantization on the coefficient level value before the inverse transform process.

[0071] Here, the inverse transform process and the inverse quantization are performed in the reverse order of the transform process and the quantization performed by the transform and quantization unit 131 .

[0072] The encoding unit 140 is configured to encode the coefficient level value output from the transform and quantization unit 131 and output the encoded data.

[0073] Here, for example, the encoding is entropy encoding that assigns codes of different lengths based on the probability of occurrence of coefficient level values.

[0074] Furthermore, the encoding unit 140 is configured to encode not only the coefficient level value but also the control data used in the decoding process.

[0075] Here, the control data may include size data such as the coding block (CU, Coding Unit) size, the prediction block (PU, Prediction Unit) size, and the transform block (TU, Transform Unit) size.

[0076] Furthermore, as described below, the control data may also include header information such as a sequence parameter set (SPS, Sequence Parameter Set), a picture parameter set (PPS, Picture Parameter Set), and a slice header.

[0077] The loop filter processing unit 150 is configured to perform filtering on the pre-filtering decoded signal output from the adder 122 , and output the post-filtering decoded signal to the frame buffer 160 .

[0078] Here, the filtering process is, for example, a deblocking filtering process for reducing distortion generated at a boundary portion of a block (coding block, prediction block, or transform block).

[0079] The frame buffer 160 is configured to store reference frames used by the inter-frame prediction unit 111 .

[0080] Here, the decoded signal after the filtering process constitutes a reference frame used in the inter-frame prediction unit 111.

[0081] (Inter-frame prediction unit 111)

[0082] Below, reference Figure 3 The inter-frame prediction unit 111 of the image encoding device 100 according to this embodiment will be described. Figure 3 This is a diagram showing an example of a functional block of the inter-frame prediction unit 111 of the image encoding device 100 according to the present embodiment.

[0083] like Figure 3 As shown, the inter prediction unit 111 includes an mv derivation unit 111A, an AMVR (Adaptive Motion Vector Resolution) unit 111B, an mv refinement unit 111C, and a prediction signal generation unit 111D.

[0084] The inter prediction unit 111 is an example of a prediction unit configured to generate a prediction signal included in a target block based on a motion vector.

[0085] like Figure 3 As shown, the mv derivation unit 111A is configured to include an AMVP (Adaptive Motion Vector Prediction) unit 111A1 and a merging unit 111A2 , and acquires a motion vector by taking a target frame and a reference frame from the frame buffer 160 as inputs.

[0086] The AMVP unit 111A1 is configured to specify a reference block included in the reference frame by comparing the target frame with the reference frame, and search for a motion vector of the specified reference block.

[0087] Then, the above-mentioned search process is performed on a plurality of candidate reference frames to determine a reference frame and a motion vector used for prediction in the target block, and output them to the prediction signal generator 111D at the next stage.

[0088] A maximum of two reference frames and motion vectors can be used for a block. The case where only one set of reference frames and motion vectors is used for a block is called "unidirectional prediction", and the case where two sets of reference frames and motion vectors are used is called "bidirectional prediction". In the following, the first set is called "L0" and the second set is called "L1".

[0089] In addition, when the above-mentioned determined motion vector is finally transmitted to the decoding device, in order to reduce the amount of coding, the AMVP unit 111A selects an MVP with a smaller difference with the motion vector of the object block, i.e., a motion vector difference (mvd), from the candidates of the motion vector predictor (mvp) derived based on the adjacent encoded motion vectors.

[0090] The index indicating the mvp and mvd selected in this way and the index indicating the reference frame (hereinafter referred to as Refidx) are encoded by the encoding unit 140 and transmitted to the image decoding device 200. This process is generally called adaptive motion vector prediction encoding (AMVP, Adaptive Motion Vector Prediction).

[0091] In addition, regarding the above-mentioned motion vector search method, reference frame and motion vector determination method, MVP selection method, and MVD calculation method, known methods can be used, and therefore, their detailed descriptions are omitted.

[0092] The AMVR unit 111B has an AMVR (Adaptive Motion Vector Resolution) function of changing the transmission accuracy of the mvd calculated by the AMVP unit 111A.

[0093] Since mvd is derived from the sum of the motion vector of the object block and mvp as described above, changing the transmission accuracy of mvd means changing the accuracy of the motion vector of the object block itself.

[0094] In non-patent document 1, there are three variations of the transmission accuracy of MVD based on AMVR. The transmission accuracy of MVD is usually 1 / 16 pixel accuracy, and the motion vector will eventually refer to the position of 1 / 16 pixel accuracy, but when AMVR is effective, the transmission accuracy of MVD is selected from 1 / 4 pixel accuracy, 1 / 2 pixel accuracy, and 1 pixel accuracy (i.e., integer pixel accuracy).

[0095] However, when the target block is affine, the transmission accuracy of mvd is selected from 1 / 4 pixel accuracy, 1 / 16 pixel accuracy, and 1 pixel accuracy. Here, affine refers to Affine in Non-Patent Document 1.

[0096] In addition, in non-patent document 1, only when 1 / 2 pixel accuracy is selected as the transmission accuracy of the mvd, the prediction signal generator 111D at the subsequent stage selects an interpolation filter different from the interpolation filter applied when pixel accuracy other than 1 / 2 pixel accuracy is selected. The details will be described below.

[0097] Furthermore, when it is determined in the AMVR unit 111B that the AMVR process is valid, a flag indicating that the AMVR is valid and an index indicating to which precision the mvd is corrected by the AMVR are transmitted to the image decoding device 200 .

[0098] In the merge unit 111A2, unlike the AMVP unit 111A1, the motion information of the target block is not searched and derived, nor is the differential mvd with the adjacent block transmitted. Instead, the target frame and the reference frame are taken as input, and the adjacent block in the same frame as the target block or the block at the same position in a frame different from the target frame is taken as the reference block, and the motion information of the reference block is directly used. This process is generally called merge coding (hereinafter referred to as merge).

[0099] When the object block is merged, a merge list for the object block is first created. The merge list is a list of combinations of multiple reference frames and motion vectors. An index (hereinafter referred to as a merge index) is assigned to each combination, and only the merge index is encoded and transmitted to the image decoding device 200 instead of encoding the Refidx and motion vector information separately.

[0100] Here, by sharing the method of creating the merge list in advance between the image encoding device 100 and the image decoding device 200, the image decoding device 200 can decode the Refidx and motion vector information based on only the merge index information. The details of the method of creating the merge list will be described below.

[0101] The mv refinement unit 111C is configured to perform refinement processing for correcting the motion vector output from the merging unit 111A2. The details will be described later.

[0102] The prediction signal generation unit 111D is configured to receive a motion vector as an input and output an MC prediction image signal, and includes a filter determination unit 111D1 and a filter application unit 111D2.

[0103] The filter determination unit 111D1 determines whether to apply an interpolation filter and the filter type based on the motion vector. The details will be described later.

[0104] The filter application unit 111D2 is configured to generate a prediction signal based on the selected interpolation filter, motion vector, and reference frame when the filter determination unit 111D1 determines that the interpolation filter is effective.

[0105] When the filter determination unit 111D1 determines that the interpolation filter is invalid, the interpolation filter is not used and a prediction signal is generated based on the motion vector and the reference frame.

[0106] (Image Decoding Device 200)

[0107] Below, reference Figure 4 The image decoding device 200 according to this embodiment is described. Figure 4 This is a diagram showing an example of functional blocks of the image decoding device 200 according to this embodiment.

[0108] like Figure 4 As shown, the image decoding apparatus 200 includes a decoding unit 210 , an inverse transform and inverse quantization unit 220 , an adder 230 , an inter prediction unit 241 , an intra prediction unit 242 , a loop filter processing unit 250 , and a frame buffer 260 .

[0109] The decoding unit 210 is configured to decode the encoded data generated by the image encoding device 100 and decode the coefficient level values.

[0110] Here, for example, decoding is entropy decoding in the reverse order to the entropy encoding performed by the encoding unit 140 .

[0111] Furthermore, the decoding unit 210 may be configured to acquire control data by decoding the encoded data.

[0112] In addition, as described above, the control data may include size data such as coding block size, prediction block size, and transform block size.

[0113] The inverse transform and inverse quantization unit 220 is configured to perform an inverse transform process on the coefficient level values ​​output from the decoding unit 210. Here, the inverse transform and inverse quantization unit 220 may be configured to perform an inverse quantization on the coefficient level values ​​before the inverse transform process.

[0114] Here, the inverse transform process and the inverse quantization are performed in the reverse order of the transform process and the quantization performed by the transform and quantization unit 131 .

[0115] The adder 230 is configured to add the prediction signal and the prediction residual signal output from the inverse transform and inverse quantization unit 220 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 242 and the loop filter processing unit 250 .

[0116] Here, the decoded signal before filtering constitutes a reference block used in the intra prediction unit 242 .

[0117] Similar to the inter-frame prediction unit 111 , the inter-frame prediction unit 241 is configured to generate a prediction signal by inter-frame prediction.

[0118] Specifically, the inter-frame prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from the coded data and a reference signal included in a reference frame. The inter-frame prediction unit 241 is configured to output the prediction signal to the adder 230 .

[0119] Similar to the intra prediction unit 112 , the intra prediction unit 242 is configured to generate a prediction signal by intra prediction (intraframe prediction).

[0120] Specifically, the intra prediction unit 242 is configured to designate a reference block included in the target frame, and to generate a prediction signal for each prediction block based on the designated reference block. The intra prediction unit 242 is configured to output the prediction signal to the adder 230 .

[0121] Similar to the loop filter processing unit 150 , the loop filter processing unit 250 is configured to perform filtering on the pre-filtering decoded signal output from the adder 230 and output the post-filtering decoded signal to the frame buffer 260 .

[0122] Here, the filtering process is, for example, a deblocking filtering process for reducing distortion generated at a boundary portion of a block (a coding block, a prediction block, a transform block, or a sub-block obtained by dividing these blocks).

[0123] Similar to the frame buffer 160 , the frame buffer 260 is configured to store reference frames used by the inter-frame prediction unit 241 .

[0124] Here, the filtered decoded signal constitutes a reference frame used in the inter-frame prediction unit 241 .

[0125] (Inter-frame prediction unit 241)

[0126] Below, reference Figure 5 The inter-frame prediction unit 241 according to this embodiment will be described. Figure 5 This is a diagram showing an example of functional blocks of the inter-frame prediction unit 241 according to the present embodiment.

[0127] like Figure 5 As shown, the inter prediction unit 241 includes an mv decoding unit 241A, an mv refinement unit 241B, and a prediction signal generation unit 241C.

[0128] The inter prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

[0129] The mv decoding unit 241A is configured to include an AMVP unit 241A1 and a merging unit 241A2 , and acquires a motion vector by decoding a target frame and a reference frame input from the frame buffer 260 , and control data received from the image encoding device 100 .

[0130] The AMVP unit 241A1 is configured to receive the target frame and the reference frame, as well as the index Refidx indicating mvp and mvd from the image encoding device 100, and the index indicating the transmission accuracy of mvd from the AMVR unit 111B, to decode the motion vector. A known method can be used for decoding the motion vector, so its detailed description is omitted.

[0131] The merging unit 241A2 is configured to receive a merge index from the image encoding device 100 to decode the motion vector.

[0132] Specifically, the merging unit 241A2 is configured to construct a merge list by the same method as the image encoding device 100, and obtain a motion vector corresponding to the received merge index from the constructed merge list. The details of the method for constructing the merge list will be described below.

[0133] Similar to the thinning unit 111C, the mv thinning unit 241B is configured to execute a thinning process for correcting a motion vector.

[0134] The prediction signal generating unit 241C is configured to include a filter determining unit 241C1 and a filter applying unit 241C2 , and generates a prediction signal based on a motion vector similarly to the prediction signal generating unit 111D.

[0135] Note that the filter determination unit 241C1 of the image decoding device 200 has exactly the same configuration as the filter determination unit 111D1 of the image encoding device 100 , and therefore, in this embodiment, the operation of the filter determination unit 111D1 will be described as a representative.

[0136] (Interpolation filter determination process)

[0137] Below, reference Figure 6 , the determination process of whether to apply an interpolation filter and the filter type in the filter determination unit 111D1 / 241C1 of this embodiment is described.

[0138] Figure 6 1 is a flowchart showing an example of a process procedure for determining whether to apply an interpolation filter and a filter type in the filter determination unit 111D1 of the present embodiment.

[0139] like Figure 6 As shown, in step S6-1, the filter determination unit 111D1 determines whether the reference position of mv is a decimal pixel position. When the reference position of mv is a decimal pixel position, the process proceeds to step S6-2, and when the reference position of mv is not a decimal pixel position, that is, when the reference position of mv is an integer pixel value position, the process proceeds to step S6-3.

[0140] In step S6-2, the filter determination unit 111D1 determines that an interpolation filter is to be applied, and the process proceeds to step S6-4.

[0141] In step S6-3, the filter determination unit 111D1 determines that the interpolation filter is not to be applied, and ends the present process.

[0142] In step S6-4, the filter determination unit 111D1 determines whether the following half-pixel index (hpelIfIdx) is invalid, that is, whether it is "0". When hpelIfIdx is "0", the process proceeds to step S6-5, and when hpelIfIdx is valid, that is, not "0", the process proceeds to step S6-6.

[0143] In step S6 - 5 , the filter determination unit 111D1 determines that the HEVC filter is used as the interpolation filter, and ends the present process.

[0144] Here, the HEVC filter refers to an 8-tap linear interpolation filter similar to that in Non-Patent Document 2.

[0145] In step S6 - 6 , the filter determination unit 111D1 determines to use a smoothing filter as the interpolation filter.

[0146] Here, for example, the smoothing filter may be a 6-tap Gaussian filter adopted in Non-Patent Document 1. Furthermore, a plurality of filters may be set and used as interpolation filters according to the value indicated by hpelIfIdx.

[0147] In this way, when the reference position of mv is a decimal pixel position, it is possible to adaptively switch between two or more interpolation filters based on hpelIfIdx. Therefore, the interpolation filter to be used can be selected according to the image characteristics, and as a result, the effect of improved encoding performance can be expected.

[0148] (Half-pixel index (hpelIfIdx) setting process)

[0149] Below, reference Figure 7 The determination process of the half-pixel index (hpelIfIdx) according to this embodiment is described.

[0150] Figure 7 This is a flowchart showing an example of a determination processing procedure of hpelIfIdx in the filter determination unit 111D1 according to the present embodiment.

[0151] As described above, hpelIfIdx is related to the determination of the interpolation filter type, and when hpelIfIdx is "0", that is, invalid, the same interpolation filter as that in Non-Patent Document 2 is used. On the other hand, when hpelIfIdx is not "0", that is, valid, a smoothing filter is used.

[0152] When the object block is merged, hpelIfIdx obtains the motion vector and RefIdx from the reference block at the same time, and corresponds to the merge index.

[0153] On the other hand, when the object block is not in merge mode and the motion vector is decoded by AMVP, the value of hpelIfIdx is determined based on whether AMVR is applied and the specific conditions described above.

[0154] like Figure 7 As shown, in step S7-1, the filter determination unit 111D1 determines whether AMVR is effective. For example, in the same manner as in Non-Patent Document 1, the determination may be made by decoding a flag (amvr_flag) indicating whether AMVR is applied.

[0155] When it is determined in step S7-1 that AMVR is valid, the process proceeds to step S7-2, and when it is determined that AMVR is invalid, the process proceeds to step S7-3.

[0156] In step S7 - 2 , the filter determination unit 111D1 determines whether the value of hpelIfIdx is “0” based on the specific condition 1 .

[0157] When it is determined in step S7-2 that specific condition 1 is satisfied, the process proceeds to step S7-4, and when it is determined that specific condition 1 is not satisfied, the process proceeds to step S7-5.

[0158] Here, in the specific condition 1 of step S7-2, the index (amvr_precision_idx) representing the transmission accuracy of AMVR can be decoded to judge that when the transmission accuracy of AMVR is 1 / 2 pixel accuracy, it is judged that the specific condition 1 is met, otherwise, it is judged that the specific condition 1 is not met.

[0159] Moreover, in the specific condition 1 of step S7-2, it can also be judged by whether the object block is affine. When the object block is affine, it is judged that the specific condition 1 is not satisfied. When the object block is not affine, and when amvr_precision_idx indicates that the transmission accuracy of mvd is 1 / 2 pixel accuracy, it is judged that the specific condition 1 is satisfied.

[0160] Here, affine refers to Affine used in Non-Patent Document 1. In the present invention, a known method can be used, so its description is omitted.

[0161] Moreover, in the specific condition 1 of step S7-2, it can also be judged by whether the object block is IBC. When the object block is IBC, it is judged that the specific condition 1 is not satisfied. When the object block is not IBC, and when amvr_precision_idx indicates that the transmission accuracy of mvd is 1 / 2 pixel accuracy, it is judged that the specific condition 1 is satisfied.

[0162] Here, IBC refers to IBC (Intra_Block_Copy) adopted in Non-Patent Document 1. In the present invention, a known method can be adopted, so its description is omitted.

[0163] In step S7-3, the filter determination unit 111D1 determines that hpelIfIdx is "0" and ends the present process.

[0164] In step S7 - 4 , the filter determination unit 111D1 determines that hpelIfIdx is “1” and ends the present process.

[0165] In step S7-5, the filter determination unit 111D1 determines that hpelIfIdx is "0" and ends the present process.

[0166] In addition, the determination process of hpelIfIdx has been described so far using a flowchart, but it can also be like Figure 8 As shown in the determination process of non-patent document 1, the filter determination unit 111D1 determines the Avmrshift value based on the decoding results of amvr_flag, amvr_precision_idx, and inter_Affine_flag and whether the target block is IBC, and determines the value of hpelIfIdx based on the AMVR_shift value.

[0167] (mv refinement department)

[0168] Below, reference Fig. 9 The mv refinement unit 241B of the present embodiment will be described. The mv refinement unit 111C of the encoding device has the same configuration as the mv refinement unit 241B of the decoding device, and therefore, in the present embodiment, the mv refinement unit 241B will be described as a representative.

[0169] Fig. 9 2 is a diagram showing an example of functional blocks of the mv refinement unit 241B according to the present embodiment.

[0170] like Fig. 9 As shown, the mv refinement section 241B includes an MMVD section 241B1 and a DMVR section 241B2.

[0171] Specifically, in the MMVD unit 241B1 and the DMVR unit 241B2, the motion vector is refined using MMVD (Merge mode with MVD) and DMVR (Decoder side Motion Vector Refinement) adopted in Non-Patent Document 1.

[0172] Specifically, the MMVD unit 241B1 is configured to use the reference position of the motion vector output from the merging unit 111A2 as a reference. Fig. 9 As shown, the correction range of the reference position is set by distance and direction (vertical and horizontal).

[0173] The MMVD unit 241B1 is configured to perform the following refinement processing: specify a correction position with the smallest specific cost from the correction range as a correction distance, and correct the motion vector based on the correction reference position.

[0174] Furthermore, the DMVR unit 241B2 is configured to perform the following refinement processing: a search range is set based on a reference position specified by a motion vector output from the merging unit 111A2, a correction reference position with the smallest specific cost is specified from the search range, and the motion vector is corrected based on the correction reference position.

[0175] Here, the mv refinement unit 241B also determines whether to apply MMVD or DMVR to the target block. In the present invention, this determination process can use the known method described in Non-Patent Document 1, so the description thereof is omitted.

[0176] In addition, when both MMVD and DMVR are invalid, the motion vector output from the merging unit 111A2 is directly output to the prediction signal generating unit 241C at the next stage without being modified.

[0177] (Merge list construction method)

[0178] Below, reference Fig.10 The merge list construction process of this embodiment is described. Fig.10 This is a flowchart showing an example of the merge list construction process according to the present embodiment.

[0179] like Fig.10 As shown, similarly to Non-Patent Document 1, the merged list construction process of the present invention is composed of a total of five merged list construction processes.

[0180] Specifically, it is composed of five merge list construction processes, namely, spatial merging in step S10-1, temporal merging in step S10-2, historical merging in step S10-3, pairwise average merging in step S10-4, and zero merging in step S10-5. Each merge list construction process will be described below.

[0181] Fig.11 This is a diagram showing an example of a merged list generated by the merged list building process according to the present embodiment.

[0182] As described above, the merge list refers to a list in which motion vectors, Refidx, and hepelIfIdx corresponding to the merge index are registered.

[0183] Here, the maximum number of merge indexes is set to "5" in Non-Patent Document 1, but can be freely set according to the designer's intention.

[0184] and, Fig.11 mvL0, mvL1, RefIdxL0, and RefIdxL1 represent the motion vectors and reference image indexes of reference image lists L0 and L1, respectively.

[0185] Here, the reference image lists L0 and L1 represent lists in which reference frames are registered, and the reference frames are specified based on RefIdx.

[0186] In addition, Fig.11The merge list shown shows motion vectors and reference image indexes for both L0 and L1, but depending on the reference block, it may be a unidirectional prediction. In this case, the unidirectional motion vector and reference image index are registered in the merge list. In addition, when there is no motion vector in the reference block, the merge list construction process for the reference block is skipped.

[0187] (Space Merge)

[0188] Fig.12 1 is a diagram showing spatial merging. Spatial merging is a technique in which mv, RefIdx, and hpelIfIdx are inherited from adjacent blocks existing in the same frame as the target block.

[0189] Specifically, from the Fig.12 In the adjacent blocks with the positional relationship shown in FIG. 1 , the above mv, Refidx, and hpelfIdx are used, and the processing order can also be the same as that of non-patent document 1, as follows: Fig.12 in the order shown.

[0190] In the merge list process, the parameters are registered in the merge list according to the above process order, but there is a check mechanism that the same motion vector and reference image index will not be registered in the merge list. This is called pruning process.

[0191] The reason for the existence of pruning processing is to increase the changes in the motion vectors and reference image indices registered in the merge list. From the perspective of the image encoding device 100, a motion vector with the smallest specific cost can be selected according to the image characteristics. In the image decoding device 200, a prediction signal with higher prediction accuracy can be generated based on the selected motion vector and reference image index. As a result, the effect of improved encoding performance can be expected.

[0192] In the spatial merge, for example, the structure is as follows: if the adjacent block A1 is registered, then for the adjacent block B1 in the next processing order, the identity of the motion vector and reference image index registered for the adjacent block A1 is confirmed. If the identity is confirmed, the motion vector and reference image index of the adjacent block B1 will not be registered in the merge list.

[0193] Furthermore, in Non-Patent Literature 1, for neighboring blocks B0, A0, and B2, in addition to checking the identity of the motion vector and reference image index registered in the merge list, it is also confirmed whether the target block is triangular merged.

[0194] In the present embodiment, the determination of triangle merging may be added to the pruning process, and a known method may be adopted as triangle merging, so the description thereof is omitted.

[0195] Furthermore, in Non-Patent Document 1, the maximum number of merge indices that can be registered for spatial merging is set to "4", and when four merge indices have been registered for the adjacent block B2 through the spatial merging process so far, the process of B2 is skipped.

[0196] In this embodiment, as in Non-Patent Literature 1, the processing of the adjacent block B2 may be determined based on the existing number of merge index registrations.

[0197] (Time Merge)

[0198] Fig.13 is a diagram showing time merging. Time merging is a technique in which the neighboring block ( Fig.13 C1) or a block located at the same location ( Fig.13 C0) is designated as the reference block, and the motion vector and reference image index are used.

[0199] The maximum number of registrants of the merge index of the time merge list in the merge list is "1" in Non-Patent Document 1, and the maximum number of registrants can also be set in the same manner in the present embodiment.

[0200] Furthermore, the motion vector used in the temporal merging has the characteristic of being scaled. Fig.14 is a diagram showing the scaling process.

[0201] Specifically, if Fig.14 As shown, based on the distance tb between the reference frame of the target block and the frame where the target frame is located, and the distance td between the reference frame of the reference block and the reference frame of the reference block, the mv of the reference block is scaled as follows.

[0202] mv'=(td / tb)×mv

[0203] In the temporal merging, the scaled mv' is registered in the merging list as a motion vector corresponding to the merging index.

[0204] (Historical Merge)

[0205] Fig.15 : is a diagram showing historical merging. Historical merging is a technique in which the motion vector, reference image index, and half-pixel index of the inter-frame prediction block that has been encoded before the target block are separately stored in a storage area called a historical merging table. When the merge list is not filled in the spatial merging process and the temporal merging process of the previous stage, the merge index registered in the historical merging table is registered in the merge list.

[0206] Fig.151 is a diagram showing an example of a history merge table construction process, which is configured as follows: based on a history merge index, a motion vector, a reference image index, and a half-pixel index corresponding to a previously encoded block are registered in the history merge table.

[0207] The maximum number of registered history merge indexes in the history merge table is set to "6" in Non-Patent Document 1, but can be freely set according to the designer's intention.

[0208] Furthermore, the registration process of the history merge index in the history merge table is FIFO processing, and is configured as follows: every time the history merge table is filled and a new history merge table is added, the last registered history merge index is deleted.

[0209] Furthermore, similar to Non-Patent Literature 1, if the target block spans across coding tree blocks (CTUs), the history merging index registered in the history merging table may be initialized.

[0210] (pairwise average merge)

[0211] Fig.16 It is a diagram showing the combination of new motion vectors and half-pixel indices calculated by pairwise average merging.

[0212] The pairwise average merge is a technique for generating a new motion vector, reference image index, and half-pixel index using the motion vectors, reference image indexes, and half-pixel indexes corresponding to two sets of merge indexes registered in the merge list.

[0213] As for the two groups of merge indexes used for pairwise average merge, the 0th and 1st merge indexes registered in the merge list may be fixedly used as in Non-Patent Document 1, or they may be freely set to another combination of two groups according to the designer's intention.

[0214] like Fig.16 As shown, the method for generating a new motion vector in the pairwise average merging is to generate the new motion vector by taking the average value of the motion vectors corresponding to the two groups of merge indexes in the merge list.

[0215] Specifically, for example, when there are two motion vectors corresponding to two groups of merge indexes (i.e., in the case of bidirectional prediction), based on the motion vectors mvL0P0 / mvL1P0 and mvL0P1 / mvL1P1 in the L0 and L1 directions, the pairwise average merged motion vectors mvL0Avg and mvL1Avg are independently calculated as follows.

[0216] mvL0Avg=(mvL0P0+mvL0P1) / 2

[0217] mvL1Avg=(mvL1P0+mvL1P1) / 2

[0218] Here, when one of mvL0P0 / mvL1P0 or mvL0P1 / mvL1P1 does not exist, the above calculation is performed excluding the non-existent motion vector.

[0219] At this time, Non-Patent Document 1 stipulates that the reference image indices RefIdxL0P0 and RefIdxL1P0 associated with the merge index P0 are always used as reference image indices associated with the pairwise average merge index.

[0220] And, if Fig.17 (a) and Fig.17 As shown in (b), the new half-pixel index hpelIfIdxAvg generated by pairwise average merging is set based on two sets of half-pixel indexes hpelIfIdxP0 and hpelIfIdxP1 corresponding to the two sets of merged indexes registered in the merge list.

[0221] Specifically, in step S17-1, it is determined whether hpelIfIdxP0 is identical to hpelIfIdxP1. If so, the process proceeds to step S17-2, and if not, the process proceeds to step S17-3.

[0222] In step S17 - 2 , hpelIfIdxAvg is set to hpelIfIdxP0 .

[0223] In step S17-3, hpelIfIdxAvg is set to invalid, that is, "0".

[0224] Here, when the values ​​of hpelIfIdxP0 and hpelIfIdxP1 are "0" or "1", Fig.17 The processing of step S17-1 and step S17-3 shown in (a) is replaced by Fig.17 (b) Processing of steps S17-4 to S17-6.

[0225] (Defect elimination of setting conditions for half-pixel index in pairwise average merging)

[0226] Below, reference Figures 18 to 20 , an example of a defect elimination method for setting conditions of half-pixel indices in paired average merging of the present embodiment is described.

[0227] Fig.18 This is a diagram showing an example in which a defect occurs in the setting conditions of the half-pixel index due to the pairwise average merging of the present embodiment.

[0228] Specifically, as described above, the following is explained: new motion vectors and half-pixel indices generated by pairwise average merging are derived by averaging the motion vectors corresponding to two sets of different merge indices registered in the merge list and judging the half-pixel index values ​​using specific conditions.

[0229] exist Fig.16 In the example shown, hpelIfIdxAvg generated by pairwise average merging is "1", and mvL0Avg and mvL1Avg are both at 1 / 2 pixel accuracy positions, so the target block satisfies the effectiveness conditions of the above-mentioned smoothing filter.

[0230] On the other hand, Fig.18 As shown, even when hpelIfIdxAvg generated by pairwise average merging is valid, i.e., "1", and mvL0P0, mvL1P0, mvL0P1, and mvL1P1 are all at 1 / 2 pixel precision positions, if one of the averaged mvL0P0 and mvL0P1 is at a 1 / 2 pixel position and the other is at an integer pixel position, mvL0Avg also refers to a 1 / 4 pixel precision position. Fig.18 As shown in FIG. 1 , mvL1Avg generated by averaging mvL1P0 and mvL1P1 has the same configuration.

[0231] Considering that the target block generates a prediction signal using the motion vector after pairwise average merging, at this time, although the half-pixel index inherited from the surrounding reference block is "1", since the motion vector references the 1 / 4 pixel precision position, the effective conditions of the smoothing filter are not met. Therefore, the HEVC filter is used as the interpolation filter for the target block, and the smoothing filter is used as the interpolation filter for the blocks around the reference target, which results in the mixed use of different interpolation filters between blocks.

[0232] In addition, since the subsequent block that refers to the object block inherits the mv with half-pixel index=1 and the reference target at the 1 / 4 pixel precision position from the object block, the inconsistency of the effective condition of the smoothing filter will be transmitted to the subsequent block.

[0233] This problem has not been solved by any known method, and the following two methods are proposed in this embodiment. Fig.19 and Fig. 20 is a diagram showing these two solutions.

[0234] First, in Fig.19 The first method is shown in Fig.19Specifically, when the hpelIfIdxAvg generated by the use shown in step S19-4 is "1" in the above case, but mvL0Avg or mvL1Avg does not refer to the 1 / 2 pixel accuracy position, the present process proceeds to step S19-5, and the used hpelIfIdxAvg is corrected from "1" to "0". This can avoid the setting defect of the effective condition of the smoothing filter in the above-mentioned target block and subsequent blocks and its propagation.

[0235] And, in Fig. 20 The second method is shown in FIG. Fig. 20 Step S20-5 and Fig.19 In addition to step S19-5, Fig. 20 The treatment shown is the same as Fig.19 The same processing as that shown in the figure is performed. In this embodiment, the elimination method shown in step S20-5 is proposed as another elimination method equivalent to step 19-5.

[0236] Specifically, in step S20-5, when hpelIfIdxAvg is "1" in the previous step, but the reference targets of mvL0Avg and mvL1Avg are not at the 1 / 2 pixel accuracy position, the reference targets of mvL0Avg and mvL1Avg are rounded to the 1 / 2 pixel accuracy position. This can avoid the setting defects of the effective conditions of the smoothing filters in the target block and the subsequent blocks and the propagation thereof.

[0237] The former solution is a technique of correcting hpelIfIdx according to mv, and the latter solution is a technique of correcting mv according to hpelIfIdx. It is also possible to freely set which solution to use according to the designer's intention.

[0238] (Defect Elimination in MMVD)

[0239] The above-mentioned defect in the setting conditions of the smoothing filter based on the half-pixel index and the motion vector in the pairwise average merging may also occur in the target block where MMVD is effective. This is because even if the reference target of the motion vector output from the merging unit 241A2 is the 1 / 2 pixel precision position and the half-pixel index is "1", in the MMVD unit 241B1, the motion vector is corrected by MMVD, and the reference target of the motion vector may deviate from the 1 / 2 pixel precision position.

[0240] Therefore, in the present invention, similar to the pairwise average merging, a solution of correcting hpelIfIdx based on mv and a solution of correcting mv based on hpelIfIdx are proposed.

[0241] Below, reference Figure 21 to Figure 23 , an example of a defect elimination method for setting conditions of a half-pixel index in the MMVD of this embodiment is described.

[0242] Fig.21 is a diagram showing MMVD. As described above, MMVD is configured as follows: based on the motion vector output by the merging unit 241A2, Fig.21 The values ​​of mmvd_distance_idx and mmvd_direction_idx shown are used to correct the reference position of the new motion vector.

[0243] In non-patent document 1, the following configuration is used: based on the two indices, mvd is calculated and added to the original motion vector. When the mvd takes 1 / 4 pixel accuracy, the motion vector after applying MMVD will deviate from the 1 / 2 pixel accuracy position.

[0244] In addition, even if the mvd of MMVD is integer precision, the pixel precision of the original motion vector will not change, which is different from the above-mentioned AMVR. For example, when the original motion vector is at the 1 / 2 pixel precision position and mvd is "1", the new motion vector will not change and still refer to the 1 / 2 precision position.

[0245] Therefore, to eliminate the above situation, use Fig. 22 and Fig.23 , two solutions are shown below.

[0246] Fig. 22 A solution is shown that modifies hpelIfIdx according to mv.

[0247] Specifically, in step S22-1, when hpelIfIdx is "1", the process proceeds to step S22-2, and when hpelIfIdx is not "1", the process ends.

[0248] In step S22-2, when specific condition 2 is satisfied, the present process ends, and when specific condition 2 is not satisfied, the present process proceeds to step S22-3.

[0249] Here, the following judgment may be added to the specific condition 2: whether the mv after applying MMVD refers to the 1 / 2 pixel precision position. The following judgment may be added: whether the mv input to the MMVD unit 241B1 is at the 1 / 2 pixel position and the mvd added by MMVD is at the 1 / 2 pixel precision.

[0250] In step S22-3, since mv deviates from the 1 / 2 pixel precision position due to MMVD in step S22-2, the half-pixel index is corrected from "1" to "0", and this processing ends.

[0251] Fig.23 A solution is shown that modifies mv according to hpelIfIdx.

[0252] Specifically, the following structure is used: In step S23-3, the Fig. 22 Another solution equivalent to the solution shown in step S22-3 is the process of rounding the mv after applying MMVD to a 1 / 2 pixel precision position.

[0253] [Change example: Modification of rounding process related to MMVD]

[0254] Next, a modification of the rounding process of mv when mv is corrected by MMVD shown in the above-mentioned step S23-3 will be described.

[0255] The above method is to directly round off the mv after applying MMVD to a 1 / 2 pixel precision position, but as a first alternative method, the mv after applying MMVD can also be rounded off indirectly by rounding off the mvd when applying MMVD to a 1 / 2 pixel precision position.

[0256] For example, consider Fig.24 The method for changing the table is as follows: Fig.24 The table shows the offset value of mvd corresponding to mmvd_distance_idx. Specifically, the following two methods are considered.

[0257] The first method is to append the hpelIfIdx judgment to Fig.24 The ideas in the table. Fig.25 , a table after the determination of hpelIfIdx is added is shown.

[0258] exist Fig.24 In the previous table shown in FIG. 1 , the offset value of mvd after MMVD is determined only by slice_fpel_mmvd_enabled_flag, which is a determination flag indicating whether the correction accuracy of MMVD is set to integer pixel accuracy at the slice level. Fig.25 In the table, when slice_fpel_mmvd_enabled_flag is "0", that is, when the correction accuracy of the motion vector by MMVD can be allowed to be down to fractional pixel accuracy, the table is divided into two according to the value of hpelIfIdx.

[0259] When hpelIfIdx is "0", mmvd_distance_idx starts from "1", and when hpelIfIdx is "1", mmvd_distance_idx starts from "2".

[0260] The actual MVD offset value of the MMVD in the non-patent document 1 is 4 times the value shown in the table, and is multiplied by the calculation accuracy of the motion vector in the non-patent document 1, that is, 1 / 16 pixel accuracy. Therefore, when Fig.25 When MmvdDistance is "1", the motion vector after MMVD is 1 / 4 pixel precision, and when MmvdDistance is "2", the motion vector after MMVD is 1 / 2 pixel precision. As a result, the defects of the above-mentioned half-pixel index and mv after MMVD can be eliminated.

[0261] (Defect Elimination in DMVR)

[0262] The above-mentioned defects in the setting conditions of the smoothing filter based on the half-pixel index and the motion vector in the pairwise average merging and MMVD may also occur in the target block where DMVR is effective. This is because even if the reference target of the motion vector output from the merging unit 241A2 is the 1 / 2 pixel precision position and the half-pixel index is "1", the motion vector is corrected by DMVR in the DMVR unit 241B2, and the reference target of the motion vector may deviate from the 1 / 2 pixel precision position.

[0263] Therefore, in this modification example, similar to the paired average merge, a solution for correcting hpelIfIdx based on mv and a solution for correcting mv based on hpelIfIdx are proposed, but since its configuration example is different from the solution in the above-mentioned MMVD, that is, Fig. 22 and Fig.23 The configuration examples shown in are exactly the same, so their description is omitted.

[0264] [Change example: Modification of rounding processing related to DMVR]

[0265] Next, the rounding process performed on mv after applying DMVR will be described in the same manner as the above-mentioned variation of the rounding process performed on mv after applying MMVD.

[0266] In MMVD, as a variation of rounding processing, the following techniques are introduced: directly rounding the mv after applying MMVD, or rounding the mvd when applying MMVD, and indirectly rounding the mv after applying MMVD. Variations related to DMVR are also basically the same, so the description of the repeated parts is omitted. On the other hand, the differences between the rounding processing of DMVR and MMVD are described.

[0267] Specifically, in DMVR, the calculation of mvd is to calculate the integer part and the decimal part independently. Therefore, when selecting mvd as the object of rounding, the sum of mvd can be directly used as the object of rounding as in the above-mentioned MMVD, but in DMVR, only the decimal part of mvd can be used as the object of rounding. This is because the case where mv deviates from the 1 / 2 pixel precision position is limited to the case where the decimal part of the mvd of DMVR is not 1 / 2 pixel precision.

[0268] In addition, the derivation of mvd in DMVR is constructed as follows: after specifying the position of the motion vector to be corrected in the above manner through search, it is calculated through parabola fitting processing, and rounding processing can also be performed through the parabola fitting processing to round off the mv after applying DMVR to a 1 / 2 pixel precision position.

[0269] [Example of change: Other examples of changes related to DMVR]

[0270] Next, other examples of changes related to DMVR are described.

[0271] It is known that DMVR is a process involving searching for a motion vector to be corrected, and therefore, is a relatively heavy process for the image decoding apparatus 200. Therefore, in order to improve the feasibility of parallelization of block processing, for example, in the judgment condition of deblocking filtering, the motion vector after applying DMVR is not used, but the motion vector before applying DMVR is used.

[0272] The same is true for half-pixel indexes. In non-patent document 1, the motion vector generated by the prediction signal for the object block is the motion vector after DMVR is applied, but for example, it is configured as follows: the motion vector used in the surrounding subsequent blocks through the merging process is the motion vector used before DMVR was applied.

[0273] Here, Fig. 22 and Fig.23 The solutions shown in are all to correct hpelIfIdx or mv after confirming the reference position of mv after applying DMVR.

[0274] However, in this solution, it can be considered that the correction processing must be performed after the DMVR is applied, resulting in a decrease in the feasibility of parallelization of block processing.

[0275] Therefore, the following three options can be adopted as alternatives.

[0276] The first method is as follows: when DMVR is effective in the object block, as described above, mv is modified only in the object block, and the mv used by subsequent blocks that refer to the object block is the mv before DMVR is applied.

[0277] The second method is as follows: when it is determined that DMVR may be effective in the object block, hpelIfIdx is always set to "0". The timing when it can be determined that DMVR may be effective in the object block is when the control data after CABAC (Context Adaptive Binary Arithmetic Coding) is decoded. In this modification example, a known method can be adopted for this timing, so its description is omitted.

[0278] The third method is as follows: for the object block whose hpelIfIdx is "1", DMVR is invalidated.

[0279] By adopting any of the three alternative methods described above, it is expected that the reduction in the feasibility of parallelization of the block processing listed above can be avoided.

[0280] [Change example: Elimination of defects when pair averaging and MMVD / DMVR are connected in series]

[0281] Below, reference Fig.26 , a modified example of the case where the motion vector derived by pairwise average merging of the present embodiment is corrected due to MMVD or DMVR is described.

[0282] Fig.26 This is a diagram showing a modified example of a case where the motion vector derived by pairwise average merging is corrected by MMVD or DMVR.

[0283] In the above example, the following solution is described: when the half-pixel index is "1" and the motion vector deviates from the 1 / 2 pixel precision position through pairwise average merging, the half-pixel index is corrected to "0".

[0284] However, it is considered that after the half-pixel index is corrected to "0", the motion vector that deviates from the 1 / 2 pixel precision position may be restored to the 1 / 2 pixel precision position again through MMVD or DMVR.

[0285] In this case, it is ideal to correct the half-pixel index that has been corrected from "1" to "0" back to "1" again, using Fig.26 An example of this method will be described.

[0286] In addition, the objects of MMVD in non-patent document 1 are limited to the motion vector with a merge index of "0" and the first motion vector. At this time, the motion vector derived by pairwise average merging will not become the object of MMVD. Assuming that the valid objects of MMVD in the future will also be expanded to motion vectors with a merge index of "2" or above, an example of this method is described below.

[0287] First, step S26-4 and the previous steps are combined with the above-mentioned pairwise average merging flowchart, i.e. Fig.19 The flowchart shown is the same, but the processing after step S26-4 is changed in consideration of the above situation.

[0288] Specifically, when the motion vector deviates from the 1 / 2 pixel precision position in step S26-4, hpelIfIdxAvg is corrected to "0" and hpelIfIdxAvg' is set to "1".

[0289] Here, hpelIfIdxAvg' is an index for maintaining the value of the half-pixel index before correction for pairwise average merging.

[0290] In addition, when the motion vector does not deviate from the 1 / 2 pixel precision position in step S26-4, hpelIfIdx is not corrected but remains at "1", and hpelIfIdxAvg' is set to "1".

[0291] Next, in step S26-7, a judgment is made about specific condition 3. When it is judged that specific condition 3 is met, the process proceeds to step S26-8. When it is judged that specific condition 3 is not met, the process ends.

[0292] Here, in specific condition 3, the following judgment may be added: whether the motion vector after applying MMVD or DMVR refers to the 1 / 2 pixel precision position. Also, the following judgment may be added: whether the mvd added to the motion vector by MMVD or DMVR is 1 / 2 pixel precision.

[0293] Next, in step S26-8, specific condition 4 is judged. When it is judged that specific condition 4 is met, the process proceeds to step S26-9. When it is judged that specific condition 4 is not met, the process ends.

[0294] Here, the specific condition 4 may also include the following judgment: whether hpelIfIdxAvg and hpelIfIdxAvg' are both "1".

[0295] In step S26-9, the following structure is adopted: regarding hpelIfIdx which has been corrected from "1" to "0" in the previous step, since it is determined that mv refers to the 1 / 2 pixel precision position after MMVD or DMVR, hpelIfIdx is corrected to "1" again.

[0296] (Scaling of motion vectors in pairwise average merging)

[0297] Below, reference Fig. 27 and Fig.28 , the scaling process of the motion vector in the pairwise average merging of this embodiment is explained. Fig. 27 This is a diagram showing an example of deriving motion vectors by pairwise average merging according to the present embodiment.

[0298] like Fig. 27 As shown, the pairwise average merge is derived using motion vectors and reference picture indices associated with two different merge indices registered in the merge list.

[0299] Here, in Non-Patent Document 1, the merge indexes used for pairwise average merge are designated as 0th and 1st. In addition, when only one of the two merge indexes used in pairwise average merge is available, the available one is registered in the merge list as a merge index associated with the pairwise average merge index.

[0300] That is, the available set of motion vectors and reference picture indices are directly registered as the motion vectors and reference picture indices associated with the pairwise average merge index.

[0301] Also, when two different merge indexes are not available, the pairwise average merge is not registered in the merge list.

[0302] For example, when there are motion vectors mvL0P0 / mvL1P0 and mvL0P1 / mvL1P1 associated with two sets of different merge indexes for derivation, vectors mvL0Avg and mvL1Avg newly derived by pairwise average merging are derived in the following manner.

[0303] mvL0Avg=(mvL0P0+mvL0P1) / 2

[0304] mvL1Avg=(mvL1P0+mvL1P1) / 2

[0305] Here, when one of mvL0P0 / mvL1P0 or mvL0P1 / mvL1P1 does not exist, the above calculation is performed excluding the non-existent motion vector.

[0306] At this time, Non-Patent Document 1 stipulates that the reference image indices RefIdxL0P0 and RefIdxL1P0 associated with the merge index P0 are always used as reference image indices associated with the pairwise average merge index.

[0307] However, the reference frames of mvL0P1 and mvL1P1 averaged in the above equations should correspond to the reference picture indices RefIdxL0P1 and RefIdxL1P1 associated with the merge index "1".

[0308] Therefore, in this embodiment, new motion vectors are derived by pairwise averaging and merging to replace mvL0P1′ and mvL0P1′ obtained by scaling mvL0P1 and mvL1P1 according to RefIdxL0P0, RefIdxL1P0, RefIdxL0P1, and RefIdxL1P1.

[0309] Here, use Fig.28 The scaling process performed on the above-mentioned motion vector will be described below.

[0310] For example, Fig.28 As shown, there are motion vectors and reference picture indices corresponding to two different merge indices derived for pairwise average merging.

[0311] At this time, the mvL0P1′ and mvL1P1′ after the scaling process means that correction is performed based on the distance between the target frame and the reference frame indicated by the reference image indices RefIdxL0P0, RefIdxL1P0, RefIdxL0P1, and RefIdxL1P1 of each merge index.

[0312] Specifically, if Fig.28 As shown in FIG. 1 , when the distances between the target frame and each reference frame are represented by tL0P0, tL0P1, tL1P0, and tL1P1, mvL0P1′ and mvL1P1′ are calculated as follows.

[0313] mvL0P1'=(tL0P1 / tL0P0)×mvL0P1

[0314] mvL1P1'=(tL1P1 / tL1P0)×mvL1P1

[0315] When merge index 1 for pairwise average merging corresponds to a motion vector and reference image index associated with a temporal merge (i.e., when the reference frame is different from the object frame as in a spatial merge or a historical merge), during the temporal merge, the distance is calculated and scaled starting from a reference frame containing a reference block at a common position in a frame different from the object block.

[0316] As an effect of the introduction of the above-mentioned scaling processing, when the paired average merge is calculated based on the motion vectors associated with two different merge indices, by scaling a unidirectional motion vector to another reference image index, the two sets of motion vectors corresponding to the reference frame represented by the common reference image index can be averaged, thereby improving the accuracy of the motion vector and, as a result, improving the encoding performance can be expected.

[0317] [Change example: Change of scaling object]

[0318] In the above example, the scaling object is uniformly set to the motion vector associated with the merge index of P1, but the motion vector of a reference frame that is farther away from the object frame can also be used as the scaling object. In this way, the reference frame for the pairwise average merge can be made close to the object frame, so the effect of improving the prediction accuracy can be expected.

[0319] (Control of half-pixel index according to block size)

[0320] Next, the control of the half-pixel index according to the target block size according to the present embodiment will be described.

[0321] In the above example, it is explained that the smoothing filter is applied when the half-pixel index is valid and the motion vector is at the 1 / 2 pixel position.

[0322] On the other hand, when the block size of the target block is small, it is better not to apply the smoothing filter to the target block.

[0323] This is because it is generally known that the block size has a correlation with image characteristics. When the block size is small, the image often has edges or complex patterns, while when the block size is large, the image often is flat and simple.

[0324] Furthermore, since it is assumed that the reference block of the target block also has the same characteristic, considering this characteristic, it is considered that when the block size of the target block is small, the application of the smoothing filter is prohibited, that is, the value of the half-pixel index is always made "0".

[0325] For example, the threshold of the block size at which the half-pixel index is always 0 may be set to 32 pixels or less.

[0326] There are two expected effects.

[0327] First, since the smoothing filter is prohibited from being applied in a specific block size where the half-pixel index is always "0", RDO (Remote Data Object) processing is not required on the side of the image coding device 100. Therefore, the speed-up effect of the image coding device 100 can be expected.

[0328] Second, when Figure 8When the object block shown in FIG. 1 is AMVP and AMVR, the flag used for determining the half-pixel index can be shortened, specifically, the bit length of amvr_precision_idx can be shortened.

[0329] exist Figure 8 In the example, since the object block has a deformation representing the position of 1 / 2 pixel precision, the bit length of amnr_precision_idx needs to be 3 bits, but in the above-mentioned specific block size, it can be shortened to 2 bits. As a result, the amount of code required for transmission from the image encoding device 100 to the image decoding device 200 is reduced, thereby expecting an improvement in encoding performance.

[0330] According to the present invention, an image decoding device, an image decoding method and a program product can be provided, which can control the reference pixel precision position of the motion vector derived by paired average merging or MMVD or DMVR according to the half-pixel index, or can control the half-pixel index according to the derived motion vector, so as to eliminate the defects of the effective conditions of the smoothing filter of the object block and further avoid the defects of the effective conditions of the smoothing filter from being transmitted to subsequent blocks.

[0331] The above-described image encoding apparatus 100 and image decoding apparatus 200 may be realized by a program that causes a computer to execute each function (each step).

[0332] In addition, in the above-mentioned embodiments, the application of the present invention to the image encoding device 100 and the image decoding device 200 is described as an example, but the present invention is not limited to this, and can also be applied to an image encoding system and an image decoding system having the functions of the image encoding device 100 and the image decoding device 200.

[0333] [Explanation of symbols]

[0334] 10: image processing system; 100: image encoding device; 111, 241: inter-frame prediction unit; 111A: mv derivation unit; 111A1, 241A1: AMVP unit; 111A2, 241A2: merging unit; 111B: AMVR unit; 111C, 241B: mv refinement unit; 111D, 241C: prediction signal generation unit; 111D1, 241C1: filter determination unit; 111D2, 241C2: filter Application unit; 112, 242: intra-frame prediction unit; 121: subtractor; 122, 230: adder; 131: transformation and quantization unit; 132, 220: inverse transformation and inverse quantization unit; 140: encoding unit; 150, 250: loop filter processing unit; 160, 260: frame buffer; 200: image decoding device; 210: decoding unit; 241A: mv decoding unit; 241B1: MMVD unit; 241B2: DMVR unit.

Claims

1. An image decoding device, characterized in that: have: a merging unit configured to decode the motion vector and the half-pixel index according to the merging index, wherein the half-pixel index indicates a type of interpolation filter used to generate the motion-compensated predicted pixel signal; a filter determination unit configured to determine whether to use an interpolation filter and the type of the interpolation filter based on the motion vector and the half-pixel index; as well as a filter application unit configured to generate a motion compensated predicted pixel signal using the interpolation filter, The merging unit is configured to generate a merge list by a specific merge list construction method, and decode the motion vector and the half-pixel index according to the merge list and the merge index. The object blocks are merged, The merging unit is configured to control the value of the derived half-pixel index according to the reference position of the motion vector derived by pairwise average merging.

2. An image decoding device, characterized in that: have: a merging unit configured to decode the motion vector and the half-pixel index according to the merging index, wherein the half-pixel index indicates a type of interpolation filter used to generate the motion-compensated predicted pixel signal; a filter determination unit configured to determine whether to use an interpolation filter and the type of the interpolation filter based on the motion vector and the half-pixel index; as well as a filter application unit configured to generate a motion compensated predicted pixel signal using the interpolation filter, The merging unit is configured to generate a merge list by a specific merge list construction method, and decode the motion vector and the half-pixel index according to the merge list and the merge index. The object blocks are merged, The merging unit is configured to control a reference position of the derived motion vector based on the half-pixel index derived by pairwise average merging.

3. An image decoding device, characterized in that: have: a merging unit configured to decode the motion vector and the half-pixel index according to the merging index, wherein the half-pixel index indicates a type of interpolation filter used to generate the motion-compensated predicted pixel signal; a motion vector refinement unit configured to refine the motion vector by MMVD; a filter determination unit configured to determine whether to use an interpolation filter and the type of the interpolation filter based on the refined motion vector and the half-pixel index; as well as a filter application unit configured to generate a motion compensated predicted pixel signal using the interpolation filter, The merging unit is configured to generate a merge list by a specific merge list construction method, and decode the motion vector and the half-pixel index according to the merge list and the merge index. The object blocks are merged, The motion vector refinement unit is configured to control a reference position of the motion vector after applying the MMVD according to the half-pixel index when the MMVD is applied.

4. An image decoding device, characterized in that: have: a merging unit configured to decode the motion vector and the half-pixel index according to the merging index, wherein the half-pixel index indicates a type of interpolation filter used to generate the motion-compensated predicted pixel signal; A motion vector refinement unit configured to refine the motion vector by DMVR; a filter determination unit configured to determine whether to use an interpolation filter and the type of the interpolation filter based on the refined motion vector and the half-pixel index; as well as a filter application unit configured to generate a motion compensated predicted pixel signal using the interpolation filter, The merging unit is configured to generate a merge list by a specific merge list construction method, and decode the motion vector and the half-pixel index according to the merge list and the merge index. The object blocks are merged, The motion vector refinement unit is configured to control the half-pixel index according to a reference position of the motion vector after applying DMVR when DMVR is applied.

5. An image decoding device, characterized in that: have: a merging unit configured to decode the motion vector and the half-pixel index according to the merging index, wherein the half-pixel index indicates a type of interpolation filter used to generate the motion-compensated predicted pixel signal; A motion vector refinement unit, which is configured to refine the motion vector through DMVR; a filter determination unit configured to determine whether to use an interpolation filter and the type of the interpolation filter based on the refined motion vector and the half-pixel index; as well as a filter application unit configured to generate a motion compensated predicted pixel signal using the interpolation filter, The merging unit is configured to generate a merge list by a specific merge list construction method, and decode the motion vector and the half-pixel index according to the merge list and the merge index. The object blocks are merged, The motion vector refinement unit is configured to control a reference position of the motion vector after DMVR is applied according to the half-pixel index when DMVR is applied.

6. An image decoding method, characterized in that: have: Step A, decoding the motion vector and the half-pixel index according to the merge index, wherein the half-pixel index indicates the type of interpolation filter used to generate the motion compensated predicted pixel signal; Step B, based on the motion vector and the half-pixel index, determining whether to use an interpolation filter and the type of the interpolation filter; as well as Step C, using the interpolation filter to generate a motion compensated predicted pixel signal, In the step A, a merge list is generated by a specific merge list construction method, and the motion vector and the half-pixel index are decoded according to the merge list and the merge index. The object blocks are merged, In the step A, the value of the derived half-pixel index is controlled according to the reference position of the motion vector derived by pairwise average merging.

7. A program product, characterized in that including a program for causing a computer to function as an image decoding device, The image decoding device comprises: a merging unit configured to decode the motion vector and the half-pixel index according to the merging index, wherein the half-pixel index indicates a type of interpolation filter used to generate the motion-compensated predicted pixel signal; a filter determination unit configured to determine whether to use an interpolation filter and the type of the interpolation filter based on the motion vector and the half-pixel index; as well as a filter application unit configured to generate a motion compensated predicted pixel signal using the interpolation filter, The merging unit is configured to generate a merge list by a specific merge list construction method, and decode the motion vector and the half-pixel index according to the merge list and the merge index. The object blocks are merged, The merging unit is configured to control the value of the derived half-pixel index according to the reference position of the motion vector derived by pairwise average merging.

Images