Image decoding device, image decoding method, and program

Abstract

The invention relates to an image decoding device, an image decoding method, and a program. An image decoding device (200) is provided with: a motion vector decoding unit (241B) configured so as to decode a motion vector from encoded data; and a refinement unit (241C) configured to search the motion vector using the value of the motion vector decoded by the motion vector decoding unit (241B) as an initial value, and determine the decoded motion vector as a final motion vector when the search cost at an initial search point is greater than a predetermined threshold value or the search cost at the initial search point is equal to or greater than the threshold value.

Description

[0001] This application is a divisional application of an application with a filing date of June 11, 2020, application number 202080040389.6, and invention name “Image decoding device, image decoding method and program”. Technical Field

[0002] The present invention relates to an image decoding device, an image decoding method and a program. Background Art

[0003] Conventionally, there is known a technique for applying refinement processing to blocks that satisfy application conditions consisting only of information that can be obtained on the decoding device side, namely, DMVR (Decoder-side motion vector refinement) (eg, see Non-Patent Document 1).

[0004] [Prior technical literature]

[0005] (Non-patent literature)

[0006] Non-patent document 1: Versatile Video Coding (Draft 5), JVET-N1001 Summary of the invention

[0007] [Problems to be solved by the invention]

[0008] However, the technology disclosed in Non-Patent Document 1, for example, has the following problem: since thinning processing must be performed on blocks that satisfy the above-mentioned application conditions, thinning processing is also performed on blocks with low correlation in the time direction, which may reduce the coding efficiency.

[0009] Therefore, the present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image decoding device, an image decoding method, and a program that can prevent a decrease in coding efficiency by not performing thinning processing on blocks with low correlation in the time direction.

[0010] [Technical means to solve the problem]

[0011] The first feature of the present invention is a picture decoding device, comprising: a motion vector decoding unit, configured to decode a motion vector from encoded data; and a refinement unit, configured to search for a motion vector using the value of the motion vector decoded by the motion vector decoding unit as an initial value, and when a search cost at an initial search point is greater than a predetermined threshold or when the search cost at the initial search point is greater than the threshold, determining the decoded motion vector as a final motion vector.

[0012] The second feature of the present invention is mainly about an image decoding device, comprising: a motion vector decoding unit, configured to decode a motion vector from encoded data; and a refinement unit, configured to search for a motion vector using the value of the motion vector decoded by the motion vector decoding unit as an initial value, and when a minimum search cost in a search point is greater than a predetermined threshold or when the minimum search cost is above the threshold, determining the decoded motion vector as a final motion vector.

[0013] The third feature of the present invention is mainly based on an image decoding device, including: a motion vector decoding unit, configured to decode a motion vector from encoded data; and a refinement unit, configured to use the value of the motion vector decoded by the motion vector decoding unit as an initial value to search for the motion vector, and when the difference between the search cost at the initial search point and the minimum search cost among the search points is less than a predetermined threshold or the difference is below the threshold, determine the decoded motion vector as the final motion vector.

[0014] The fourth feature of the present invention is mainly about an image decoding device, comprising: a motion vector decoding unit, configured to decode a motion vector from encoded data; a refinement unit, configured to search for a motion vector using the value of the motion vector decoded by the motion vector decoding unit as an initial value; and a prediction signal generation unit, configured to generate a prediction signal based on the motion vector output from the refinement unit; and the prediction signal generation unit is configured to decide not to perform BDOF (Bi-Directional Optical Flow) processing when an index value representing the similarity between a block on a first reference frame side and a block on a second reference frame side is greater than a predetermined threshold or when the index value is above the threshold.

[0015] The gist of a fifth feature of the present invention is an image decoding device, comprising: a motion vector decoding unit configured to decode a motion vector from encoded data; and a refinement unit configured to search for a motion vector using the value of the motion vector decoded by the motion vector decoding unit as an initial value; and the refinement unit is configured to include a norm of a differential vector between an initial search point and a search point in a search cost.

[0016] The main purpose of the sixth feature of the present invention is to include the following steps: decoding a motion vector from encoded data; using the value of the decoded motion vector as an initial value to search for the motion vector; and when the search cost at the initial search point is greater than a predetermined threshold or the search cost at the initial search point is above the threshold, determining the decoded motion vector as the final motion vector.

[0017] The seventh feature of the present invention is a program used in an image decoding device, which causes a computer to execute the following steps: decoding a motion vector from encoded data; searching for a motion vector using the decoded value of the motion vector as an initial value; and determining the decoded motion vector as a final motion vector when the search cost at the initial search point is greater than a predetermined threshold or when the search cost at the initial search point is above the threshold.

[0018] (Effects of the Invention)

[0019] According to the present invention, it is possible to provide an image decoding device, an image decoding method, and a program that can prevent a decrease in coding efficiency by not performing thinning processing on blocks with low correlation in the time direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a diagram showing an example of the configuration of an image processing system 10 according to one embodiment.

[0021] Figure 2 This is a diagram showing an example of a functional block of the image encoding device 100 according to one embodiment.

[0022] Figure 3 FIG. 1 is a diagram showing an example of a functional block of the inter-frame prediction unit 111 of the image encoding device 100 according to one embodiment.

[0023] Figure 4 This is a flowchart showing an example of a processing procedure of the refinement unit 111C of the inter prediction unit 111 of the image encoding device 100 according to one embodiment.

[0024] Figure 5 This is a flowchart showing an example of a processing procedure of the prediction signal generating unit 111D of the inter-frame prediction unit 111 of the video decoding device 30 according to one embodiment.

[0025] Figure 6 This is a diagram showing an example of a functional block of an image decoding device 200 according to an embodiment.

[0026] Figure 7 This is a diagram showing an example of a functional block of the inter prediction unit 241 of the image decoding device 200 according to one embodiment. DETAILED DESCRIPTION

[0027] The following is an explanation of the embodiments of the present invention with reference to the accompanying drawings. In addition, the constituent elements in the following embodiments may be appropriately replaced with existing constituent elements, etc., and various modifications including combinations with other existing constituent elements may be performed. Therefore, the content of the invention recorded in the claims should not be limited by the description of the following embodiments.

[0028] (First Embodiment)

[0029] Below, reference Figures 1 to 7 An image processing system 10 according to a first embodiment of the present invention will be described. Figure 1 1 is a diagram showing an image processing system 10 according to the present embodiment.

[0030] like Figure 1 As shown, the image processing system 10 includes an image encoding device 100 and an image decoding device 200 .

[0031] The image encoding device 100 is configured to generate encoded data by encoding an input image signal, and the image decoding device 200 is configured to generate an output image signal by decoding the encoded data.

[0032] Here, the coded data may be transmitted from the image coding apparatus 100 to the image decoding apparatus 200 via a transmission path. Alternatively, the coded data may be provided from the image coding apparatus 100 to the image decoding apparatus 200 after being stored in a storage medium.

[0033] (Image Coding Device 100)

[0034] Below, reference Figure 2 The image encoding device 100 according to this embodiment will be described. Figure 2 This is a diagram showing an example of functional blocks of the image encoding device 100 according to the present embodiment.

[0035] like Figure 2 As shown, the image encoding device 100 includes an inter-frame prediction unit 111, an intra-frame prediction unit 112, a subtractor 121, an adder 122, a transform and quantization unit 131, an inverse transform and inverse quantization unit 132, an encoding unit 140, a loop filter processing unit 150 and a frame buffer 160.

[0036] The inter-frame prediction unit 111 is configured to generate a prediction signal by inter-frame prediction.

[0037] Specifically, the inter-frame prediction unit 111 is configured to specify a reference block included in the reference frame by comparing a frame to be encoded (hereinafter referred to as a target frame) with a reference frame stored in the frame buffer 160 , and determine a motion vector of the specified reference block.

[0038] The inter prediction unit 111 is configured to generate a prediction signal included in each prediction block based on the reference block and the motion vector. The inter prediction unit 111 is configured to output the prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

[0039] The intra prediction unit 112 is configured to generate a prediction signal by intraframe prediction.

[0040] Specifically, the intra prediction unit 112 is configured to designate a reference block included in the target frame, and to generate a prediction signal for each prediction block based on the designated reference block. Furthermore, the intra prediction unit 112 is configured to output the prediction signal to the subtractor 121 and the adder 122 .

[0041] Here, the reference block is a block that is referenced by a block to be predicted (hereinafter referred to as a target block). For example, the reference block is a block adjacent to the target block.

[0042] The subtractor 121 is configured to subtract the prediction signal from the input image signal and output a prediction residual signal to the transform and quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal which is a difference between a prediction signal generated by intra prediction or inter prediction and the input image signal.

[0043] The adder 122 is configured to add the prediction signal and the prediction residual signal output from the inverse transform and inverse quantization unit 132 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 112 and the loop filter processing unit 150 .

[0044] Here, the decoded signal before filtering constitutes a reference block used in the intra prediction unit 112 .

[0045] The transform and quantization unit 131 is configured to perform a transform process on the prediction residual signal and obtain a coefficient level value. In addition, the transform and quantization unit 131 may be configured to perform quantization on the coefficient level value.

[0046] Here, the transformation process is a process of transforming the prediction residual signal into a frequency component signal. In this transformation process, a base model (transformation matrix) corresponding to a discrete cosine transform (DCT) or a base model (transformation matrix) corresponding to a discrete sine transform (DST) can be used.

[0047] The inverse transform and inverse quantization unit 132 is configured to perform an inverse transform process on the coefficient level value output from the transform and quantization unit 131. Here, the inverse transform and inverse quantization unit 132 may be configured to perform an inverse quantization on the coefficient level value before the inverse transform process.

[0048] Here, the inverse transform process and the inverse quantization are performed in the reverse order of the transform process and the quantization performed by the transform and quantization unit 131 .

[0049] The encoding unit 140 is configured to encode the coefficient level value output from the transform and quantization unit 131 and output the encoded data.

[0050] Here, for example, the encoding is entropy encoding that assigns codes of different lengths based on the probability of occurrence of coefficient level values.

[0051] Furthermore, the encoding unit 140 is configured to encode not only the coefficient level value but also the control data used in the decoding process.

[0052] Here, the control data may include size data such as the coding block (CU, Coding Unit) size, the prediction block (PU, Prediction Unit) size, and the transform block (TU, Transform Unit) size.

[0053] The loop filter processing unit 150 is configured to perform filtering on the pre-filtering decoded signal output from the adder 122 , and output the post-filtering decoded signal to the frame buffer 160 .

[0054] Here, the filtering process is, for example, a deblocking filtering process that reduces distortion generated at a boundary portion of a block (coding block, prediction block, or transform block).

[0055] The frame buffer 160 is configured to store reference frames used by the inter-frame prediction unit 111 .

[0056] Here, the decoded signal after the filtering process constitutes a reference frame used in the inter-frame prediction unit 111.

[0057] (Inter-frame prediction unit 111)

[0058] Below, reference Figure 3 The inter-frame prediction unit 111 of the image encoding device 100 according to this embodiment will be described. Figure 3 This is a diagram showing an example of a functional block of the inter-frame prediction unit 111 of the image encoding device 100 according to the present embodiment.

[0059] like Figure 3 As shown, the inter prediction section 111 includes a motion vector search section 111A, a motion vector encoding section 111B, a refinement section 111C, and a prediction signal generation section 111D.

[0060] The inter prediction unit 111 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

[0061] The motion vector search unit 111A is configured to specify a reference block included in the reference frame by comparing the target frame with the reference frame, and to search for a motion vector of the specified reference block.

[0062] Furthermore, the above search is performed on multiple candidate reference frames to determine the reference frame and motion vector used for prediction in the prediction block. Up to two reference frames and motion vectors can be used for each block. The case where only one set of reference frames and motion vectors is used for a block is called unidirectional prediction, and the case where two sets of reference frames and motion vectors are used is called bidirectional prediction. In the following, the first set is called L0 and the second set is called L1.

[0063] The motion vector search unit 111A is configured to determine a coding method for a reference frame and a motion vector. As a coding method, there are not only a conventional method of transmitting information of a reference frame and a motion vector separately, but also a merge mode described below.

[0064] In addition, regarding the method of searching for a motion vector, the method of determining a reference frame, and the method of determining the encoding method of a reference frame and a motion vector, known methods can be used, and therefore, detailed descriptions thereof are omitted.

[0065] The motion vector encoding unit 111B is configured to encode the information of the reference frame and the motion vector determined by the motion vector search unit 111A using the encoding method determined by the motion vector search unit 111A.

[0066] When the encoding method of the block is the merge mode, a merge list for the block is first created on the side of the image encoding device 100. Here, the merge list is a list that lists combinations of multiple reference frames and motion vectors.

[0067] An index is assigned to each combination, and the image encoding device 100 encodes only the index instead of encoding the reference frame and motion vector information separately, and transmits it to the image decoding device 200. By sharing the creation method of the merge list in advance on the image encoding device 100 side and the image decoding device 200 side, the image decoding device 200 side can decode the reference frame and motion vector information only from the information related to the index.

[0068] In addition, regarding the method of creating the merge list, a known method can be used, and therefore, a detailed description thereof is omitted.

[0069] Regarding the encoding of motion vector information, first, a predicted motion vector, which is a predicted value of a motion vector to be encoded, is generated, and a differential motion vector, which is a difference value between the predicted motion vector and a motion vector to be actually encoded, is encoded.

[0070] The thinning unit 111C is configured to perform a thinning process (for example, DMVR) for correcting the motion vector encoded by the motion vector encoding unit 111B.

[0071] Specifically, the refinement unit 111C is configured to set a search range based on a reference position specified by a motion vector encoded by the motion vector encoding unit 111B, specify a correction reference position with a minimum predetermined cost from the search range, and perform refinement processing to correct the motion vector based on the correction reference position.

[0072] Figure 4 11 is a flowchart showing an example of the processing procedure of the refinement unit 111C.

[0073] like Figure 4 As shown, in step S41, the refinement unit 111C determines whether the prescribed conditions for applying the refinement process are satisfied. When all the prescribed conditions are satisfied, the process proceeds to step S42. On the other hand, when any of the prescribed conditions is not satisfied, the process proceeds to step S48 to end the refinement process.

[0074] Here, the prescribed condition includes a condition that the block is a bidirectionally predicted block. In addition, the prescribed condition may include a condition that the motion vector is encoded in the merge mode.

[0075] In step S42 , the thinning unit 111C generates a search image based on the motion vector encoded by the motion vector encoding unit 111B and the information of the reference frame.

[0076] Here, when the motion vector points to a non-integer pixel position, the thinning section 111C applies a filter to interpolate the pixel at the non-integer pixel position to the pixel value of the reference frame. At this time, the thinning section 111C can reduce the amount of calculation by using an interpolation filter having a smaller number of taps than the interpolation filter used in the prediction signal generation section 111D described below. For example, the thinning section 111C can interpolate the pixel value at the non-integer pixel position by bilinear interpolation.

[0077] In step S43 , the refinement unit 111C calculates the search cost at the initial position using the search image generated in step S42 .

[0078] Here, the initial position is the position indicated by the motion vector encoded by the motion vector encoding unit 111B. And, the search cost is an indicator value of the similarity between the reference block on the L0 side (first reference frame side) and the reference block on the L1 side (second reference frame side) indicated by the above motion vector, for example, the absolute value error sum or square error sum between pixel values ​​can be used.

[0079] In step S44, the refinement unit 111C uses the search cost at the initial position calculated in step S43 as input to determine whether the end condition for stopping the refinement process of the block is satisfied. When all the end conditions are satisfied, the process proceeds to step S45. On the other hand, when any of the end conditions is not satisfied, the process proceeds to step S48 to terminate the refinement process.

[0080] Here, the end condition (stop condition) in step S44 may include, for example, the condition that the search cost at the initial position is less than a predetermined first threshold value (or the search cost at the initial position is below the predetermined first threshold value). Also, the end condition (stop condition) in step S44 may include, for example, the condition that the search cost at the initial position is greater than a predetermined second threshold value (or the search cost at the initial position is greater than the predetermined first threshold value).

[0081] In step S45 , the refinement unit 111C performs a search with integer pixel accuracy using the search image generated in step S42 .

[0082] Here, integer pixel accuracy means searching only points at integer pixel intervals based on the motion vector encoded by the motion vector encoding unit 111B.

[0083] The thinning section 111C determines the corrected motion vector at the integer pixel interval position by the search in step S45. Here, as the search method, a known method can be used.

[0084] For example, the refinement unit 111C may perform the search by a method of searching for points corresponding to a combination in which the signs of the difference motion vectors on the L0 side and the L1 side are inverted.

[0085] Here, the search point that minimizes the search cost at each search position is the corrected motion vector at the integer pixel interval position. As described above, an index such as the absolute value difference sum or the square error sum can be used as the search cost. At this time, the refinement unit 111C can also add the Lp norm of the difference vector between the corrected motion vector and the motion vector at the initial position to the index for evaluating the similarity between blocks such as the absolute value difference sum as the search cost.

[0086] Specifically, the refinement unit 111C may use the sum of the absolute value difference sum and the L1 norm of the difference vector as the search cost. Furthermore, when the refinement unit 111C adds the L1 norm to the absolute value difference sum, it may weight the sum according to a predetermined ratio before adding the sum. For example, the refinement unit 111C may add twice the value of the L1 norm to the absolute value difference sum. In this case, if the magnification is a power of 2, equivalent processing may be achieved by bit shifting. In addition, the search result in step S45 may also be the same value as the motion vector before the search.

[0087] In step S46, the thinning unit 111C uses the search cost corresponding to the corrected motion vector at the integer pixel interval position determined in step S45 to determine whether the end condition for stopping the thinning process in the block is satisfied. When all the end conditions are satisfied, the process sequence proceeds to step S47. On the other hand, when any of the end conditions is not satisfied, the process sequence proceeds to step S48 to end the thinning process.

[0088] Here, the end condition (stop condition) in step S46 may include, for example, the condition that the search cost is greater than a predetermined third threshold (or the search cost is greater than the predetermined third threshold). At this time, the third threshold may also be set to the same value as the second threshold.

[0089] Furthermore, when it is determined that the end condition is met, the refinement unit 111C may discard the search result in step S45 and use the same motion vector as that before the refinement process (ie, the motion vector encoded by the motion vector encoding unit 111B) as the final motion vector of the block.

[0090] Furthermore, the end condition (stop condition) in step S46 may include, for example, a condition that the difference between the search cost at the initial position calculated in step S43 and the search cost corresponding to the corrected motion vector calculated in step S45 is less than a predetermined fourth threshold (or the difference is less than the predetermined fourth threshold). In this case, the fourth threshold may be set to the same value as the first threshold.

[0091] Furthermore, when it is determined that the end condition is satisfied, the thinning unit 111C may discard the search result in step S45 and use the same motion vector as that before the thinning process as the final motion vector of the block.

[0092] In step S47, the thinning unit 111C searches for a motion vector with non-integer pixel accuracy, using the corrected motion vector with integer pixel accuracy determined in step S43 as an initial value. Here, a known method can be used as a method for searching a motion vector.

[0093] Furthermore, the refinement unit 111C may use the result of step S43 as input and determine a motion vector with non-integer pixel accuracy using a parametric model such as parabola fitting without actually performing a search.

[0094] After determining the corrected motion vector of non-integer pixel accuracy in step S47, the refinement unit 111C moves to step S48 to terminate the refinement process. Here, for convenience, the expression of the corrected motion vector of non-integer pixel accuracy is used, but depending on the search result of step S47, the result may be the same value as the motion vector of integer pixel accuracy obtained in step S45.

[0095] For the sake of convenience, step S43 and step S45 are described as different steps above, but the processing of both can be performed in the same step. For example, the processing sequence moves directly to step S45 after step S42, and the refinement unit 111C can calculate both the search cost at the initial position and the search cost corresponding to the corrected motion vector at the pixel interval position in step S45. Thereafter, the processing sequence moves to step S46, and the refinement unit 111C can determine whether the end condition is satisfied in consideration of at least one of the conditions described as step S44 and the conditions described as step S46.

[0096] Furthermore, for example, in step S48 , the refinement unit 111C may also use the search cost at the initial position or the search cost corresponding to the corrected motion vector at the pixel interval position to determine whether to discard the search result.

[0097] For example, when the search cost at the initial position is greater than the second threshold (or is above the second threshold), the refinement unit 111C may discard the search result and use the same motion vector as that before the refinement process (i.e., the motion vector encoded by the motion vector encoding unit 111B) as the final motion vector of the block.

[0098] Furthermore, for example, when the search cost corresponding to the corrected motion vector at the integer pixel interval position is greater than the third threshold (or is above the third threshold), the refinement unit 111C may discard the search result and use the same motion vector as that before the correction process as the final motion vector of the block.

[0099] In addition, for example, when the difference between the search cost at the initial position and the search cost corresponding to the corrected motion vector at the integer pixel interval position is less than the fourth threshold (or is below the fourth threshold), the refinement unit 111C may discard the search result and use the same motion vector as that before the refinement process as the final motion vector of the block.

[0100] In the above, the configuration including all of steps S41 to S48 has been described, but step S44 and step S46 are not necessarily included in the configuration.

[0101] The thinning unit 111C may also divide a block larger than a predetermined threshold into smaller sub-blocks and perform thinning processing on each sub-block. For example, the thinning unit 111C may set the execution unit of the thinning processing to 16×16 pixels in advance, and when the horizontal or vertical size of the block is larger than 16 pixels, the block is divided into smaller blocks of 16 pixels. In this case, as the motion vector that becomes the reference of the thinning processing, the motion vector of the block encoded by the motion vector encoding unit 111B is used for all sub-blocks in the same block.

[0102] When processing each sub-block, the refinement unit 111C may perform Figure 4 Furthermore, the refinement unit 111C may also perform only the following steps on each sub-block: Figure 4 Specifically, the refinement unit 111C may perform partial processing on each block. Figure 4 The processing of steps S41 and S42 is performed on each sub-block, and the processing of steps S43 to S48 is performed on each sub-block.

[0103] The prediction signal generation unit 111D is configured to generate a prediction signal based on the corrected motion vector output from the refinement unit 111C.

[0104] Here, as described below, the prediction signal generation unit 111D is configured to determine whether or not BDOF (Bi-Directional Optical Flow) processing can be performed on each block based on the information (eg, search cost) calculated in the above-mentioned thinning processing.

[0105] Specifically, when the motion vector is not corrected, the prediction signal generator 111D is configured to generate a prediction signal based on the motion vector encoded by the motion vector encoder 111B. On the other hand, when the motion vector is corrected, the prediction signal generator 111D is configured to generate a prediction signal based on the motion vector corrected by the refinement unit 111C.

[0106] Figure 5 11 is a flowchart showing an example of the processing sequence of the prediction signal generating unit 111D. Here, when the refinement unit 111C performs refinement processing in sub-block units, the processing of the prediction signal generating unit 111D is also performed in sub-block units. In this case, the term "block" in the following description can be appropriately replaced with "sub-block".

[0107] Furthermore, for a block that is not subjected to the thinning process by the thinning unit 111C, when the block size is larger than a predetermined threshold, it may be divided into smaller sub-blocks, and the thinning process may be performed on each sub-block. For example, similarly to the thinning unit 111C, the execution unit of the prediction signal generation process may be set to 16×16 pixels in advance, and when the size of the block in the horizontal direction or the vertical direction is larger than 16 pixels, it may be divided into smaller blocks of 16 pixels. In this case, the term "block" in the following description may be appropriately replaced with "sub-block".

[0108] like Figure 5 As shown, in step S51, the prediction signal generation unit 111D generates a prediction signal.

[0109] Specifically, the prediction signal generation unit 111D receives the motion vector encoded by the motion vector encoding unit 111B or the motion vector encoded by the thinning unit 111C as input, and when the position indicated by the motion vector is a non-integer pixel position, applies a filter to interpolate the pixel at the non-integer pixel position to the pixel value of the reference frame. Here, as for the specific filter, a horizontally and vertically separable filter with a maximum of 8 taps disclosed in Non-Patent Document 1 can be applied.

[0110] When the block is a block subjected to bidirectional prediction, the prediction signal generator 111D generates both a prediction signal using a first set (hereinafter referred to as L0) of reference frames and motion vectors and a prediction signal using a second set (hereinafter referred to as L1) of reference frames and motion vectors.

[0111] In step S52 , the prediction signal generation unit 111D checks whether the application conditions of the BDOF process described below are satisfied.

[0112] As the application condition, the condition described in non-patent document 1 can be applied. The application condition includes at least the condition that the block is a block for bidirectional prediction. In addition, as described in non-patent document 1, the application condition may also include the condition that the motion vector of the block is not encoded in the Symmetric MVD mode.

[0113] If the application condition is not satisfied, the processing sequence moves to step S55 and the processing ends. At this time, the prediction signal generation unit 111D outputs the prediction signal generated in step S51 as the final prediction signal.

[0114] On the other hand, when all the application conditions are satisfied, the present processing sequence moves to step S53. In step S53, for the blocks satisfying the application conditions, the present processing sequence actually determines whether to execute the BDOF process of step S54.

[0115] The following uses the absolute value difference sum as the search cost as an example, but other indicators can also be used as the search cost. For example, any indicator value used to judge the similarity between image signals, such as the absolute value difference sum or the square error sum of the signals after removing the local average, can be used as the search cost.

[0116] For example, the prediction signal generating unit 111D calculates the absolute value difference sum of the prediction signal of L0 and the prediction signal of L1, and if the calculated value is smaller than a predetermined threshold (or if the calculated value is below a predetermined threshold), it determines not to perform BDOF processing.

[0117] Furthermore, for example, the prediction signal generating unit 111D may calculate the sum of the absolute value differences between the prediction signal of L0 and the prediction signal of L1, and if the calculated value is greater than a predetermined threshold (or if the calculated value is above a predetermined threshold), a determination is made not to perform BDOF processing.

[0118] Here, for a block subjected to the thinning process by the thinning unit 111C, the prediction signal generating unit 111D may use the result of the thinning process to determine whether to apply the BDOF process.

[0119] The prediction signal generator 111D may use the search cost calculated in the refinement process (eg, the sum of the absolute value differences between the pixel values ​​of the reference block on the L0 side and the pixel values ​​of the reference block on the L1 side) to determine whether to apply the BDOF process.

[0120] For example, when the absolute value difference sum of the search point with the smallest search cost (absolute value difference sum) in the search performed with integer pixel accuracy in step S45 is less than a predetermined fifth threshold (or below the predetermined fifth threshold), the prediction signal generation unit 111D may make a judgment that the BDOF process is not applied. In this case, the fifth threshold may be set to the same value as the first threshold.

[0121] For example, when the absolute value difference sum of the search point with the smallest search cost (absolute value difference sum) in the search performed with integer pixel precision in step S45 is greater than a predetermined sixth threshold (or above the predetermined sixth threshold), the prediction signal generation unit 111D may make a judgment that the BDOF process is not to be applied. In this case, the sixth threshold may be set to the same value as the second threshold or the third threshold.

[0122] For example, when the search cost at the initial position calculated in step S43 is less than a predetermined fifth threshold (or below the predetermined fifth threshold), the prediction signal generation unit 111D may make a decision not to apply the BDOF process. In this case, the fifth threshold may be set to the same value as the first threshold.

[0123] For example, when the search cost at the initial position calculated in step S43 is greater than a predetermined sixth threshold (or above the predetermined sixth threshold), the prediction signal generation unit 111D may make a judgment not to apply the BDOF process. In this case, the sixth threshold may be set to the same value as the second threshold or the third threshold.

[0124] For example, when the difference between the search cost at the initial position calculated in step S43 and the minimum search cost in the search performed with integer pixel accuracy in step S45 is less than a predetermined seventh threshold (or below the predetermined seventh threshold), the prediction signal generation unit 111D may make a judgment that the BDOF process is not applied. In this case, the seventh threshold may be set to the same value as the first threshold or the fourth threshold.

[0125] Furthermore, the prediction signal generating unit 111D may make a determination by a method using the result of the thinning process in each block on which the thinning process has been performed, and may make a determination by a method using the sum of absolute value differences in each of the other blocks.

[0126] Furthermore, as described above, the prediction signal generation unit 111D may be configured to determine whether the BDOF process is appropriate by using only information obtained from the result of the thinning process without separately performing a process of calculating the sum of absolute value differences between the prediction signal on the L0 side and the prediction signal on the L1 side. In this case, in step S53, the prediction signal generation unit 111D determines to always apply the BDOF process to the block that has not been subjected to the thinning process.

[0127] According to this configuration, in this case, since it is not necessary to perform the calculation process of the absolute value difference sum in the prediction signal generating unit 111D, it is possible to reduce the amount of processing and processing delay from the viewpoint of hardware implementation.

[0128] Furthermore, according to this configuration, from the perspective of software implementation, by using the result of the thinning process, the BDOF process is not performed on blocks where the effect of the BDOF process is expected to be poor, thereby shortening the overall image processing time while maintaining coding efficiency.

[0129] Furthermore, the determination process itself using the result of the above-mentioned thinning process is executed inside the thinning unit 111C, and information indicating the result is transmitted to the prediction signal generating unit 111D, thereby the prediction signal generating unit 111D can also determine whether the BDOF process is suitable.

[0130] For example, as described above, the following flag is prepared in advance: the value of the motion vector or search cost before and after the refinement processing is judged, which is "1" when the condition for not applying the BDOF processing is met, and is "0" when the condition for not applying the BDOF processing is not met and the refinement processing is not applied. The prediction signal generation unit 111D can refer to the value of the flag to determine whether it is suitable for BDOF processing.

[0131] Furthermore, here, for the sake of convenience, step S52 and step S53 are described as different steps, but the determination in step S52 and step S53 may be performed at the same time.

[0132] In the above-mentioned determination, for a block determined not to be applied with the BDOF process, the present process sequence moves to step S55. For other blocks, the present process sequence moves to step S54.

[0133] In step S54, the prediction signal generation unit 111D performs BDOF processing. The BDOF processing itself can use a known method, so the detailed description is omitted. After the BDOF processing is performed, the processing sequence moves to step S55 and the processing ends.

[0134] The first to seventh thresholds may be defined so that their values ​​vary according to the block size (the product of the height and width of the block) or the number of pixels used in calculating the search cost such as SAD (Sum of Absolute Differences) within the block.

[0135] The first to seventh thresholds may also be defined so that their values ​​change according to the quantization parameter (QP) of the block. For example, when the cost value tends to be larger when the quantization parameter is larger, the threshold may be defined to increase as the quantization parameter increases.

[0136] (Image Decoding Device 200)

[0137] Below, reference Figure 6 The image decoding device 200 according to this embodiment is described. Figure 6 This is a diagram showing an example of functional blocks of the image decoding device 200 according to this embodiment.

[0138] like Figure 6 As shown, the image decoding apparatus 200 includes a decoding unit 210 , an inverse transform and inverse quantization unit 220 , an adder 230 , an inter prediction unit 241 , an intra prediction unit 242 , a loop filter processing unit 250 , and a frame buffer 260 .

[0139] The decoding unit 210 is configured to decode the encoded data generated by the image encoding device 100 and decode the coefficient level values.

[0140] Here, for example, decoding is entropy decoding in the reverse order to the entropy encoding performed by the encoding unit 140 .

[0141] Furthermore, the decoding unit 210 may be configured to acquire control data by decoding the encoded data.

[0142] In addition, as described above, the control data may include size data such as coding block size, prediction block size, and transform block size.

[0143] The inverse transform and inverse quantization unit 220 is configured to perform an inverse transform process on the coefficient level values ​​output from the decoding unit 210. Here, the inverse transform and inverse quantization unit 220 may be configured to perform an inverse quantization on the coefficient level values ​​before the inverse transform process.

[0144] Here, the inverse transform process and the inverse quantization are performed in the reverse order of the transform process and the quantization performed by the transform and quantization unit 131 .

[0145] The adder 230 is configured to add the prediction signal and the prediction residual signal output from the inverse transform and inverse quantization unit 220 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 242 and the loop filter processing unit 250 .

[0146] Here, the decoded signal before filtering constitutes a reference block used in the intra prediction unit 242 .

[0147] Similar to the inter-frame prediction unit 111 , the inter-frame prediction unit 241 is configured to generate a prediction signal by inter-frame prediction.

[0148] Specifically, the inter-frame prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from the coded data and a reference signal included in a reference frame. The inter-frame prediction unit 241 is configured to output the prediction signal to the adder 230 .

[0149] Similar to the intra prediction unit 112 , the intra prediction unit 242 is configured to generate a prediction signal by intra prediction (intraframe prediction).

[0150] Specifically, the intra prediction unit 242 is configured to designate a reference block included in the target frame, and to generate a prediction signal for each prediction block based on the designated reference block. The intra prediction unit 242 is configured to output the prediction signal to the adder 230 .

[0151] Similar to the loop filter processing unit 150 , the loop filter processing unit 250 is configured to perform filtering on the pre-filtering decoded signal output from the adder 230 and output the post-filtering decoded signal to the frame buffer 260 .

[0152] Here, the filtering process is, for example, a deblocking filtering process for reducing distortion generated at a boundary portion of a block (a coding block, a prediction block, a transform block, or a sub-block obtained by dividing these blocks).

[0153] Similar to the frame buffer 160 , the frame buffer 260 is configured to store reference frames used by the inter-frame prediction unit 241 .

[0154] Here, the filtered decoded signal constitutes a reference frame used in the inter-frame prediction unit 241 .

[0155] (Inter-frame prediction unit 241)

[0156] Below, reference Figure 7 The inter-frame prediction unit 241 according to this embodiment will be described. Figure 7 This is a diagram showing an example of functional blocks of the inter-frame prediction unit 241 according to the present embodiment.

[0157] like Figure 7 As shown, the inter prediction unit 241 includes a motion vector decoding unit 241B, a refinement unit 241C, and a prediction signal generation unit 241D.

[0158] The inter prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

[0159] The motion vector decoding unit 241B is configured to obtain a motion vector by decoding the control data received from the image encoding device 100 .

[0160] Similar to the thinning unit 111C, the thinning unit 241C is configured to execute thinning processing for correcting the motion vector.

[0161] Similar to the prediction signal generating unit 111D, the prediction signal generating unit 241D is configured to generate a prediction signal based on a motion vector.

[0162] According to the image encoding device 100 and the image decoding device 200 of the present embodiment, the thinning unit 111C, 241C may be configured such that when the search cost is greater than a predetermined threshold value (or when the search cost is greater than a predetermined threshold value), the search result is discarded. Here, when the correlation between the pixel values ​​of the block, the reference block on the L0 side, and the reference block on the L1 side is high, the thinning process can improve the coding efficiency. With the above configuration, it is possible to not perform the thinning process on a block with a small correlation in the time direction without transmitting additional information (information on whether to apply the thinning process) from the image encoding device 100 side, thereby preventing a decrease in coding efficiency.

[0163] According to the image encoding device 100 and the image decoding device 200 of the present embodiment, the refinement unit 111C, 241C may be configured to discard the search result when the difference between the search cost at the initial position and the search cost corresponding to the searched point is smaller than a predetermined threshold (or below the predetermined threshold). This can avoid unnecessary correction of the motion vector at a point where the search cost is slightly smaller than the initial position due to the influence of noise, etc., and thus prevent a decrease in coding efficiency.

[0164] According to the image encoding device 100 and the image decoding device 200 of the present embodiment, the refinement units 111C and 241C may include the norm of the difference vector between the initial position and the search point in the search cost. In this case, when the absolute value error and the index value of the equal evaluation similarity are substantially the same at each search point, regularization using the norm of the difference vector can prevent the correction amount of the motion vector from becoming unnecessarily large, thereby preventing the coding efficiency from being reduced.

[0165] According to the image encoding device 100 and the image decoding device 200 of the present embodiment, in the judgment of whether to perform BDOF processing in the prediction signal generation unit 111D, 241D, the condition that the similarity (for example, the sum of absolute value errors) between the reference block on the L0 side and the reference block on the L1 side is greater than a predetermined threshold (or the similarity is above a predetermined threshold) can be included. Similar to the thinning process, the BDOF process is also an effective process when the correlation between the block, the reference block on the L0 side, and the reference block on the L1 side is high. Therefore, by setting the above-mentioned structure, it is not necessary to transmit additional information (information on whether to apply BDOF processing) from the image encoding device 100 side, so that the BDOF process can be not performed on the block with less correlation in the time direction, thereby preventing the coding efficiency from being reduced. In addition, in the above judgment, the result of the above-mentioned thinning process can be used. As a result, the process of calculating the above-mentioned absolute value difference can be reduced.

[0166] Furthermore, the above-described image encoding apparatus 100 and image decoding apparatus 200 may be realized by a program that causes a computer to execute each function (each step).

[0167] In addition, in the above-mentioned embodiments, the application of the present invention to the image encoding device 100 and the image decoding device 200 is described as an example, but the present invention is not limited to this example, and can also be applied to an image encoding / decoding system having the functions of the image encoding device 100 and the image decoding device 200.

[0168] According to the present invention, it is possible to prevent a decrease in encoding efficiency by not performing thinning processing on blocks with low correlation in the time direction.

[0169] [Explanation of Symbols]

[0170] 10 Image Processing System

[0171] 100 Image encoding device

[0172] 111, 241 Inter-frame prediction unit

[0173] 111A Motion vector search unit

[0174] 111B Motion vector encoding unit

[0175] 111C, 241C refinement

[0176] 111D, 241D prediction signal generation unit

[0177] 112, 242 Intra-frame prediction unit

[0178] 121 Subtractor

[0179] 122, 230 Adder

[0180] 131 Transformation and Quantization Department

[0181] 132, 220 Inverse transform and inverse quantization unit

[0182] 140 Coding Department

[0183] 150, 250 loop filter processing unit

[0184] 160, 260 frame buffer

[0185] 200 Image decoding device

[0186] 210 Decoding Department

[0187] 241B Motion vector decoding unit

Claims

1. An image decoding device, characterized in that: have: a motion vector decoding unit configured to decode the motion vector based on the encoded data; and Refinement Department, Wherein, the refinement unit is configured to perform the following processing: searching for a motion vector by using the value of the motion vector decoded by the motion vector decoding unit as an initial value; When the search cost at the initial search point is less than a predetermined first threshold, ending the search for the motion vector; When the search cost at the initial search point is greater than the first threshold, searching at integer pixel intervals; and When the difference between the search cost at the initial search point and the minimum search cost of the search at the integer pixel interval position is less than a predetermined third threshold, or the difference is below the threshold, the search result at the integer pixel interval position is discarded.

2. An image decoding method, characterized in that: It has the following processes: Decoding the motion vector according to the encoded data; Using the value of the motion vector obtained by decoding as an initial value to search for the motion vector; When the search cost at the initial search point is less than a predetermined first threshold, ending the search for the motion vector; When the search cost at the initial search point is greater than the first threshold, searching at integer pixel intervals; as well as When the difference between the search cost at the initial search point and the minimum search cost of the search at the integer pixel interval position is less than a predetermined third threshold, or the difference is below the threshold, the search result at the integer pixel interval position is discarded.

3. A program used in an image decoding device, the program causing a computer to execute the following steps: Decoding the motion vector according to the encoded data; Using the value of the motion vector obtained by decoding as an initial value to search for the motion vector; When the search cost at the initial search point is less than a predetermined first threshold, ending the search for the motion vector; When the search cost at the initial search point is greater than the first threshold, searching at integer pixel intervals; and When the difference between the search cost at the initial search point and the minimum search cost of the search at the integer pixel interval position is less than a predetermined threshold, or the difference is below the threshold, the search result at the integer pixel interval position is discarded.

Images